Everything not indexing every file, problem seems to be especially with text and spreadsheet files

If you are experiencing problems with "Everything", post here for assistance.
Post Reply
Arilando
Posts: 1
Joined: Wed Nov 08, 2017 3:14 am

Everything not indexing every file, problem seems to be especially with text and spreadsheet files

Post by Arilando »

I have no idea why. Uninstalling and installing again solves the problem at first but shortly afterwards it will stop indexing most spreadsheet and text document files.
void
Developer
Posts: 15096
Joined: Fri Oct 16, 2009 11:31 pm

Re: Everything not indexing every file, problem seems to be especially with text and spreadsheet files

Post by void »

Everything should not be missing changes to NTFS volumes.

Are you using NTFS indexing or Folder indexing ? (Check if you volumes are listed under Tools -> Options -> NTFS or Tools -> Options -> Folders)

Are you using the Everything Service (Tools -> Options -> General -> Everything Service - Must be ticked checked, not square checked) ? If so, please make sure the Everything Service is running.
If the service is not running Everything will not detect changes.

To force Everything to reindex:
  • In Everything, from the Tools menu, click Options.
  • Click the Indexes tab on the left.
  • Click Force Rebuild.
  • Click OK.
rseiler
Posts: 41
Joined: Fri Aug 15, 2014 7:08 pm

Re: Everything not indexing every file, problem seems to be especially with text and spreadsheet files

Post by rseiler »

I'm seeing something similar, though it's not about any particular type of file that I've noticed. What happens is I go to search for something that I absolutely know is there (and is not new), like skype.exe for example, and it doesn't find it. But it will find any number of other things that I throw at it. I then rebuild the index and it does find it.

I am on the 1.4.1.895 release, but it was happening before that. Win10 (Insider).

To answer the above questions: Yes, the service is always running. And it's NTFS Indexing (with "Monitor changes" checked for all volumes).

I should also mention, in case it's relevant, that Everything is always running in the background (i.e. on the tray). What I haven't tried when this happens is exiting Everything and restarting it without rebuilding the index. I doubt that would matter, but it's possible.

And finally, Everything definitely picks up changes live as they're happening. It's great. It's just that once in a while it seems to lose track of what appears to be a tiny minority of things (it would be impossible to know how tiny).
void
Developer
Posts: 15096
Joined: Fri Oct 16, 2009 11:31 pm

Re: Everything not indexing every file, problem seems to be especially with text and spreadsheet files

Post by void »

Do you have any exclude filters set? such as exclude recycle bin or exclude hidden/system files/folders?

If so, folders that have been excluded and should be re-included will not show up in Everything, for example, if you exclude your recycle bin, delete your skype folder, restore your skype folder, skype.exe will not be found in Everything. This is a limitation with Everything.

If you do not have any excluded filters, Everything should always find all your files.

If you do not have any exclude filters, I would need to the see verbose debug logs of the file being removed to find the issue.

It could be an issue with the USN Change Journal using a different hard link name for the file that has changed:
viewtopic.php?t=5198

Everything will only maintain one hard link of a file and I suspect skype.exe might be a hard link.
rseiler
Posts: 41
Joined: Fri Aug 15, 2014 7:08 pm

Re: Everything not indexing every file, problem seems to be especially with text and spreadsheet files

Post by rseiler »

OK, so I do have some excludes (though "Exclude hidden/system files and folders" are not checked), which have fooled me before: cache folders in the data profiles of several browsers, c:\windows.old, $Recycle*, and System Volume Information*.

I rarely use Recycle Bin at all, deleting almost everything when holding down Shift, so none of the things where I've noticed this effect before (even though I can't really remember what the earlier ones were now) would have been deleted/restored.

It's possible that some have been hard links, but skype.exe for one isn't: when I right-click a file and look at its properties, thanks to a tool called Link Shell Extension, there's a Link Properties tab if a given file is a hard link. I'll definitely check each file in question when this happens again though.
TOTAL
Posts: 1
Joined: Sat Jul 04, 2020 10:48 pm

Re: Everything not indexing every file, problem seems to be especially with text and spreadsheet files

Post by TOTAL »

A similar problem here. For the first time ever, I have found Everything to miss a file.

In my case this was an entire drive, it turned out. Lightning fast, this indexing on SSD. Many thanks.
rseiler
Posts: 41
Joined: Fri Aug 15, 2014 7:08 pm

Re: Everything not indexing every file, problem seems to be especially with text and spreadsheet files

Post by rseiler »

void wrote: Wed Nov 08, 2017 4:22 am Everything should not be missing changes to NTFS volumes.
I think I've finally figured out when this happens. Let me run this scenario by you.

What happens when there are a LOT of changes, as in a build-to-build Win10 upgrade, which I do practically every week with Win10 Insider?

Afterwards, and not just immediately afterwards but well afterwards, I've seen Everything miss files. Could it be confused by the massive changes? A re-index always rights the ship.
aoz987
Posts: 10
Joined: Tue Oct 05, 2021 10:03 pm

Re: Everything not indexing every file, problem seems to be especially with text and spreadsheet files

Post by aoz987 »

Version 1.4.1.1009 (x64)
same issue. files are missing. i KNEW i was not going crazy..( maybe)

i've noticed it since i have been using this version, on new machine. I had created a file, then later that day looked for it, could not find it; i stopped everything, rebooted machine, then restarted it (it starts at startup). it is as a service, and it is indexing ntfs.
any suggestions to test, are appreciated.
void
Developer
Posts: 15096
Joined: Fri Oct 16, 2009 11:31 pm

Re: Everything not indexing every file, problem seems to be especially with text and spreadsheet files

Post by void »

Everything will need to be reinstalled after a major Windows update.

Major Windows updates will remove the Everything Service.
I'm seeing the Everything service and other non-Microsoft services being removed from all my test machines with some major Windows updates.


Please make sure Everything is installed correctly:
  • In Everything, from the Tools menu, click Options.
  • Click the General tab on the left.
  • Check Store settings and data in %APPDATA%\Everything
  • Uncheck Run as administrator.
  • Check Everything Service. (Please make sure this is tick-checked and not square-checked)
  • Click OK.
  • Exit Everything (right click the Everything tray icon and click Exit).
  • Restart Everything.
Please try forcing a rebuild:
  • In Everything, from the Tools menu, click Options.
  • Click the Indexes tab on the left.
  • Click Force Rebuild.
  • Click OK.
Please make sure no search options are checked under the Search menu (eg: uncheck match case, uncheck match whole words, uncheck regex)

Please make sure the Everything filter is active under the Search menu.

Please make sure these files are not excluded from Tools -> Options -> Exclude.

Please let me know if the issue persists.



Everything 1.4 does not track changes to hard links.
Tracking hard link changes is in development.
rseiler
Posts: 41
Joined: Fri Aug 15, 2014 7:08 pm

Re: Everything not indexing every file, problem seems to be especially with text and spreadsheet files

Post by rseiler »

I've never seen a major Windows update (and by that I mean a build update as opposed to a CU) remove non-Microsoft services, including yours.

Such a practice would be a catastrophic bug that would never fly given the number of 3rd-party services that essentially every system has.

But as I did speculate in my last post, for whatever reason Everything does need to be reindexed after such an update. I just use:
Everything.exe -reindex -update -close
rseiler
Posts: 41
Joined: Fri Aug 15, 2014 7:08 pm

Re: Everything not indexing every file, problem seems to be especially with text and spreadsheet files

Post by rseiler »

One thing I'd be curious about: when that happens, is it a subset of 3rd-party services or all of them?

If it's a subset, I bet it's a case of Defender being overzealous with some smaller, less familiar services.

If it's not a subset and it takes out even extremely well-known ones like Google Update Service that may be the most common non-Microsoft service in the world, it must be a bug.
void
Developer
Posts: 15096
Joined: Fri Oct 16, 2009 11:31 pm

Re: Everything not indexing every file, problem seems to be especially with text and spreadsheet files

Post by void »

It's not all 3rd party services, just obscure ones.

Windows is most likely just removing not well-known services.
aoz987
Posts: 10
Joined: Tue Oct 05, 2021 10:03 pm

Re: Everything not indexing every file, problem seems to be especially with text and spreadsheet files

Post by aoz987 »

ok, i FOUND one.
copied a file, from my sister's computer, to onedrive.
went to onedrive on my system copied the file to the desktop.
it was an EXE file, i store thiese for system reinstalls if needed.
so i then searched for ddmset...., for the prior versions on my machine ,i n my install folders.
lo and behold, everytyhing did NOT FIND the newer version (first picture, on desktop, 2068 exists, but is NOT in everything).
THEN, i exited everything, and restarted it.
it THEN found the 2068 file.
this has happened when i copied stuff from onedrive to the desktop, etc.
but today i was able to prove it.
Attachments
774.jpg
774.jpg (320.81 KiB) Viewed 23459 times
772.jpg
772.jpg (283.13 KiB) Viewed 23459 times
void
Developer
Posts: 15096
Joined: Fri Oct 16, 2009 11:31 pm

Re: Everything not indexing every file, problem seems to be especially with text and spreadsheet files

Post by void »

THEN, i exited everything, and restarted it.
it THEN found the 2068 file.
This sounds like Everything was disconnected from the Everything Service.
If Everything 1.4 is disconnected, it will not reconnect to the Everything Service until Everything is restarted.


Could you please try running with debug logging:
  • In Everything, type in the following search and press ENTER:
    /debug_log=1
  • If successful, debug_log=1 is shown in the statusbar for a few seconds.
  • Type in the following search and press ENTER:
    /restart
  • ---Everything will restart
  • The next time you notice Everything missing files, could you please:
  • Exit Everything (File -> Exit)
  • Please send your %TEMP%\Everything Debug Log.txt to support@voidtools.com

Debug logs can grow very large.
To disable debug logging:
  • In Everything, type in the following search and press ENTER:
    /debug_log=0
  • If successful, debug_log=1 is shown in the statusbar for a few seconds.
  • Type in the following search and press ENTER:
    /restart
  • ---Everything will restart
Debug logs will determine if the connection between Everything and the Everything Service is dropping.
Look for the following line:
service client disconnected by server


Everything 1.5 will reconnect automatically when disconnected from the Everything Service.
aoz987
Posts: 10
Joined: Tue Oct 05, 2021 10:03 pm

Re: Everything not indexing every file, problem seems to be especially with text and spreadsheet files

Post by aoz987 »

i will test as you indicated; may take me some time to do this, but will do it, thanks
aoz987
Posts: 10
Joined: Tue Oct 05, 2021 10:03 pm

Re: Everything not indexing every file, problem seems to be especially with text and spreadsheet files

Post by aoz987 »

i just had everything not find a file, and also still FOUND a file that i had deleted.
i sentyou the debug log as requested.
void
Developer
Posts: 15096
Joined: Fri Oct 16, 2009 11:31 pm

Re: Everything not indexing every file, problem seems to be especially with text and spreadsheet files

Post by void »

Thank you for the debug logs.

I am investigating the issue..
void
Developer
Posts: 15096
Joined: Fri Oct 16, 2009 11:31 pm

Re: Everything not indexing every file, problem seems to be especially with text and spreadsheet files

Post by void »

This issue is caused by Boxcryptor.

Boxcryptor can cause a Blue Screen of Death (BSOD) when calling FSCTL_READ_USN_JOURNAL (Everything reading changes to the USN Journal).
It also appears to bring down the Everything Service.
I am working with Boxcryptor to fix the issue.


For now, please do not include any Boxcryptor volumes in your index:
  • In Everything, from the Tools menu, click Options.
  • Click the NTFS tab on the left.
  • Uncheck Automatically include new fixed volumes.
  • Uncheck Automatically include new removable volumes.
  • Select any Boxcryptor volumes and Uncheck Include in database.
  • Click OK.
aoz987
Posts: 10
Joined: Tue Oct 05, 2021 10:03 pm

Re: Everything not indexing every file, problem seems to be especially with text and spreadsheet files

Post by aoz987 »

thanks very much, i will await your followup, AND if you have other things to test for this, let me know, i hav no problem working to test these.

re: boxcryptor, i do value it in the EVERYTHING search; i can live without it for a while, and if i NEED some of the files in there, to search for, i can temporarily turn it back on, then off again.

again, thanks

the MAIN thing is it has validated that at least i am not losing my mind ! :-) :)
void
Developer
Posts: 15096
Joined: Fri Oct 16, 2009 11:31 pm

Re: Everything not indexing every file, problem seems to be especially with text and spreadsheet files

Post by void »

Thanks for your help with finding the issue.

You can continue to index your Boxcryptor volume as a folder index under Tools -> Options -> Folders.
aoz987
Posts: 10
Joined: Tue Oct 05, 2021 10:03 pm

Re: Everything not indexing every file, problem seems to be especially with text and spreadsheet files

Post by aoz987 »

i just re-logged on here, to see if any changes. i had NOT seen the part about boxcryptor as folder, so i have just implemented that.
but, just to provide feedback, taking boxcryptor out of NTFS has made a big difference, no errors since doing that.
so i will now see what it does as a folder
thanks
nick
void
Developer
Posts: 15096
Joined: Fri Oct 16, 2009 11:31 pm

Re: Everything not indexing every file, problem seems to be especially with text and spreadsheet files

Post by void »

Thanks for your feedback,

Boxcryptor have also implemented a fix in CBFS 2020.
Boxcryptor wrote:FYI: ETA for Boxcryptor with CBFS 2020 --> within the next 1-2 releases 😀
aoz987
Posts: 10
Joined: Tue Oct 05, 2021 10:03 pm

Re: Everything not indexing every file, problem seems to be especially with text and spreadsheet files

Post by aoz987 »

is boxcryptor fixed in this update?
NOTE, since i changed boxcryptor to FOLDER, i hvae not had EVERYTHING get messed up, i've checked it periodically.
any other feedback apreciated,thanks
nick
Attachments
419.jpg
419.jpg (41.41 KiB) Viewed 16614 times
void
Developer
Posts: 15096
Joined: Fri Oct 16, 2009 11:31 pm

Re: Everything not indexing every file, problem seems to be especially with text and spreadsheet files

Post by void »

I can't answer that.

You will need to ask boxcryptor.
aoz987
Posts: 10
Joined: Tue Oct 05, 2021 10:03 pm

Re: Everything not indexing every file, problem seems to be especially with text and spreadsheet files

Post by aoz987 »

hello again.
question - moving something to recycle, and back again - you said somewhre above that it does not show up in "everything".
how come, and any way to force a re-index to include this?

second, any way to copy preferences from my old machine to new machine? (any direcory, file, etc, to clone)?
thanks
void
Developer
Posts: 15096
Joined: Fri Oct 16, 2009 11:31 pm

Re: Everything not indexing every file, problem seems to be especially with text and spreadsheet files

Post by void »

question - moving something to recycle, and back again - you said somewhre above that it does not show up in "everything".
how come, and any way to force a re-index to include this?
Everything will miss undeleted files if you are excluding the recycle bin.

Please consider including your recycle bin in your index:
  • In Everything, from the Tools menu, click Options.
  • Click the Exclude tab on the left.
  • Uncheck Exclude system files and folders.
  • Uncheck Exclude hidden files and folders.
  • Uncheck Enable exclude list.
  • Click OK.

Consider excluding recycle bin files with your Everything filter instead:
  • In Everything, from the Search menu, click Organize filters....
  • Select Everything and click Edit....
  • Change the search to:
    !?:\$recycle.bin\
  • Click OK.

To force a rebuild:
  • In Everything, from the Tools menu, click Options.
  • Click the Indexes tab on the left.
  • Click Force Rebuild.
  • Click OK.
aoz987
Posts: 10
Joined: Tue Oct 05, 2021 10:03 pm

Re: Everything not indexing every file, problem seems to be especially with text and spreadsheet files

Post by aoz987 »

thanks ! i will do that

SECOND QUESTION, above - how to copy settings/includes/excludes from one machine to another, if i install to new mchine?
thanks
raccoon
Posts: 1015
Joined: Thu Oct 18, 2018 1:24 am

Re: Everything not indexing every file, problem seems to be especially with text and spreadsheet files

Post by raccoon »

how to copy settings/includes/excludes from one machine to another if i install to new machine
There are a few methods for migrating settings.

1) Use the Tools -> Import and Export Settings... menu.
... Export settings to a file on a thumb drive, then Import settings on the target computer.

or 2) Copy and Paste a block of settings from one everything.ini to another everything.ini.
... settings lines pasted to the bottom of the target config file will merge and replace the old settings
a) Do this by typing about:config into the Everything search bar and hit Enter,
or b) Do this from Windows Explorer while both instances of Everything are closed.

or 3) Delete-or-rename and replace the everything.ini from one machine to another.
... Do this from Windows Explorer while both instances of Everything are closed.
void
Developer
Posts: 15096
Joined: Fri Oct 16, 2009 11:31 pm

Re: Everything not indexing every file, problem seems to be especially with text and spreadsheet files

Post by void »

As raccoon mentions, copy your Everything.ini in %APPDATA%\Everything while Everything is not running.


-or-


To save a configuration file:
  • In Everything, type in the following search and press ENTER:
    /config_save
  • Choose a filename and click Save.
To load a configuration file:
  • In Everything, type in the following search and press ENTER:
    /config_load
  • Select your Everything.ini and click Open.
  • Type in the following search and press ENTER:
    /restart

1) Use the Tools -> Import and Export Settings... menu.
... Export settings to a file on a thumb drive, then Import settings on the target computer.
This is supported on Everything 1.5 or later.
aoz987
Posts: 10
Joined: Tue Oct 05, 2021 10:03 pm

Re: Everything not indexing every file, problem seems to be especially with text and spreadsheet files

Post by aoz987 »

thanks very much. i'll just do the everything.ini, easiest way
have a great day
Post Reply