omit results startup behavior

[robertoleonardo]

hi - i have everything set to open to tray upon pc startup. when i have omit results disabled, the first time i open the everything window (from the tray icon that launched automatically at startup) -- it's pretty much immediately ready to go. BUT when i have omit results enabled, because i have quite a number of omits in my omit list, there's a good 10-15 second delay while it loads the omit list. after that first time, i can close the window to tray and re-open it and it's ready to go right away, even with omit results enabled.

i understand that using omit results will necessarily have an impact on startup performance -- but is there a way to configure everything such that that negative impact on startup happens at app startup (even when launched silently to the tray using the /startup command line flag) as opposed to occurring when i open my first search window (which is what i'm seeing now)?

or do results omissions just work in such a way that they require an open search window before the list can be loaded for the first time? if that's teh case, bummer, but not the end of the world -- i can just set "omit results" to default to "disabled" and only toggle it on manually when i'm ready.

thanks!

[void]

Thank you for your issue report robertoleonardo,

While the result omission list is loaded on startup, it is not applied until your first search.

I will make a change here for the next update:
when result omissions is enabled, Everything will apply the result omission list on startup.

[robertoleonardo]

awesome -- i was hoping it was something that would be easy to fix/implement. i look forward to it. you're the best --- thank you!!

[void]

Everything 1.5.0.1397a will now load and apply result omissions on startup.

[robertoleonardo]

amazing - thank you!!

[robertoleonardo]

btw it's working great - just what i needed. thank you so much -- very nice to not have to wait every first time i open it for a search.

[win32]

Hmm, I still(1399a) get delayed ability to query the db on startup. Is it because I use the USN loading feature?
But the statusbar stucks at "Loading result omissions", the "loading recent changes" shows only momentarily...

I do use complex omissions, but isnt the old omissions db stored somewhere ready to get used on startup?

(I always wanted to report this as a wish, ie why E doesn't work on startup with the old db until it builds the new one (with some temporarily increased RAM usage). Then I noticed it was one of v1.5 features so I was always thinking this was a problem in my config.)

[robertoleonardo]

hmm interesting. i also use usn so i don't think it's that. for me, as of .1397a update, my omits load on application startup and don't wait until i actually open the window for the first time.

i also have a bunch of omissions that take a while to load -- so id definitely notice if i still had to wait after opening the window. but, fortunately, i dont. must be some setting you have enabled other than usn.

[win32]

It's not a big deal, I dont really ever close my pc.
It's just that you gave me hope only to take it away!
Here is part of the log in case void is interested:

Code: Select all

sort: items 93047, queues 8, threads 8
sort 93047: 0.002093
loaded 93047 of 1942692 changes in 1.200204 seconds
recent changes array memory usage: 1575792 bytes
loaded run history in 0.081997 seconds
run history data count: 16876, data: 1948070 bytes
run history ptr count: 8694
total run history memory usage: 2735478 bytes
loaded result omissions in 0.000236 seconds
result omissions folder filename count: 105
result omissions file filename count: 18
result omissions folder and file filter count: 8
result omissions folder filter count: 1
result omissions file filter count: 11
heap: 10464 bytes
<--- here is the  point  where it stucks on startup  ---->
found 1290489 result omissions pointers in 18.968366 seconds
result omissions folder pointer count: 120602
result omissions file pointer count: 1169887
result omissions pointer memory usage: 10440472 bytes
loaded 603223 folders, 2186932 files, in 21.162149 seconds
Load complete.

[void]

Hmm, I still(1399a) get delayed ability to query the db on startup. Is it because I use the USN loading feature?

It's not the Load USN Journal in into recent changes option.
It's caused by the complex result omission filters.

I do use complex omissions, but isnt the old omissions db stored somewhere ready to get used on startup?

The result omissions are not stored in your database.
The result omissions filters are stored in your Everything.ini
They need to be re-applied each time you start Everything.
I will consider saving 'applied' result omissions filters to your Everything database.
I could (and should) also make applying the result omissions filters multithreaded..
Thanks for the suggestion.

(I always wanted to report this as a wish, ie why E doesn't work on startup with the old db until it builds the new one (with some temporarily increased RAM usage). Then I noticed it was one of v1.5 features so I was always thinking this was a problem in my config.)

Everything shouldn't be doing a reindex on startup.
Please send your Tools -> Debug -> Statistics -> Build -> Last Rebuild Reason

[win32]

result omissions filters are stored in your Everything.ini

yes of course. Maybe I haven't understood how "omit results" works internally, because my understanding was that there is an omit db somewhere (a db with omissions removed/ or a omissions db which is subtracted on the fly from the main db) and that's why it's so fast. Because before this feature, your to go option was to use a negation filters which was not as performant, whereas omit resutls is transparent performance wise.

Everything shouldn't be doing a reindex on startup.
Please send your Tools -> Debug -> Statistics -> Build -> Last Rebuild Reason

Sorry, I didn't mean an actual reindex, I meant the poor (in my box) UX during the update process at startup. Why E doesnt let me query the old "snapshot" of the db while it's doing its "index update" work? Maybe it's possible to store the changes somewhere (while the user can query the not updated db) and then do the merging in an instant. I assumed this is how works already and it's my config which is the culprit.

[void]

my understanding was that there is an omit db somewhere

There is a database stored in memory.
Never on disk.
It is rebuilt in memory from your filters each time you start Everything.

Technical information:
There's a internal database of pointers to each file/folder that is omitted.
pointer lookup is instant.
Everything maintains this result omission database in realtime. so new files will be checked against your result omissions filters, if it matches, a pointer is added to this result omission database.

It's possible to save this database of pointers to disk, and quickly load it on startup.
I have this on my TODO list for a future version of Everything.
It was never implemented initially because I assumed the result omission lists would be small, and only affect a small number of files/folders.

Sorry, I didn't mean an actual reindex, I meant the poor (in my box) UX during the update process at startup. Why E doesnt let me query the old "snapshot" of the db while it's doing its "index update" work? Maybe it's possible to store the changes somewhere (while the use can query the not updated db) and then do the merging in an instant. I assumed this is how works already and it's my config which is the culprit.

I'm not seeing this with a large complex result omission list.
I can search immediately after the result omission list is loaded. (after Loading result omissions... is shown in the status bar for a few seconds)
Updating... is still shown in the status bar and I can still search instantly.

What is shown in the status bar on the left?

Any action on the database is queued. (like performing a search)
If Everything is updating and an action is queued, the updating will pause and the queued action is executed.
This should happen instantly, maybe something funky is happening for you, I would need to see logs.
Terminate Everything.
Launch Everything.exe with -debug-log
Let Everything load your database, try to perform a search while it is updating.
Wait for Everything to finish updating.
From the Tools menu, under the Debug submenu, click Stop Debug Logging.
This will open your %TEMP%\Everything Debug Log.txt in Notepad.
Please upload this file in a bug report

[void]

Sorry, I didn't mean an actual reindex, I meant the poor (in my box) UX during the update process at startup. Why E doesnt let me query the old "snapshot" of the db while it's doing its "index update" work? Maybe it's possible to store the changes somewhere (while the user can query the not updated db) and then do the merging in an instant. I assumed this is how works already and it's my config which is the culprit.

Thank you for the debug logs.
Did you see this issue when you made these logs?

The logs show:
The results omissions take 20 seconds to load.
Loading the result omissions completes at 54.512
Everything starts processing USN journal changes at 54.518
The loading completes at 54.525
A 'test' search is queued during the load.
Everything pauses the background update at 54.534
The search is executed 54.534
The search completes at 54.876 (342 ms)
The results are shown at 54.877
Everything resumes the background update at 54.877

Everything lets you search as soon as possible.
The background update doesn't appear to be blocking the search.
I am misunderstanding something?

342 ms is a long time for a simple search.
Way to long for only 2 million files.

search_max_threads=3

Is there a reason for setting the search threads to a low value?

[win32]

in the log I sent there is 'gap' of 18+secs where E builds its omissions db.
I have finished typing during in this gap, I don't actually type 'test' at 23:46:54.534 but much earlier, before the "loading completes at 54.525"

During this time the result list is empty and status bar shows "Loading results omissions...", which makes sense to me, since "omit results" is enabled, E has to build its omit DB before displaying the results, and we have established there is no saved omit db (which is what I originally thought this thread was about). Maybe you mean it should display the noomit results during this time?

I removed some really careless filters like (**\folderA\**\folderB\**) I had added to the omission filters list and the omit db build time dropped to 10secs. I'll do a better cleanup in the future. I also reset max_threads option.

(Btw just as notice, every time I edit the omit filter list the main program message loop is blocked, UI freezes for a while, in contrast to the startup case where I can type and it's just the resultlist which is empty)

I also attach an excerpt of a startup log with verbose enabled, covers time during omit filters loading, omit db creation, and completing the query. Again I have finished typing before omit db creation.

[void]

I have finished typing during in this gap, I don't actually type 'test' at 23:46:54.534 but much earlier, before the "loading completes at 54.525"

Yes, this was understood.

During this time the result list is empty and status bar shows "Loading results omissions...", which makes sense to me, since "omit results" is enabled, E has to build its omit DB before displaying the results, and we have established there is no saved omit db (which is what I originally thought this thread was about). Maybe you mean it should display the noomit results during this time?

Yes, there's no omit db.
I will consider an omit db for a future version.
There's many other optimizations I could also make here, such as building the db with multiple threads, not applying all filters, if a file/folder is already omitted.

I will not show the noomit results during this load time.
That is likely to cause trouble.
You will have to wait for the result omissions to load before the search is executed.

I removed some really careless filters like (**\folderA\**\folderB\**) I had added to the omission filters list and the omit db build time dropped to 10secs. I'll do a better cleanup in the future. I also reset max_threads option.

If you are happy to share your

(Btw just as notice, every time I edit the omit filter list the main program message loop is blocked, UI freezes for a while, in contrast to the startup case where I can type and it's just the resultlist which is empty)

The omit db is rebuilt on the UI thread.
I will consider moving this to a worker thread in a future version.

I also attach an excerpt of a startup log with verbose enabled, covers time during omit filters loading, omit db creation, and completing the query. Again I have finished typing before omit db creation.

Thanks for the log.
The omit load only takes 10 seconds now.
The search is a quicker now (0.144947 seconds)



I noticed you are omitting a lot of files (1million+)
Is there one or two filters that omits most of these files?
I would love to see your result omissions.csv, but understand if you don't want to share.

Consider using a filter to block most of your files, this will keep your omit db small.
Create a filter (or modify your Everything filter) to exclude most of your files, for example:

!C:\private\ !D:\junk\

[win32]

Ok I did send you the omit filters, but just for curiosity's shake, the problem will resolve itself whenever - if ever - the saving omitdb to disk feature gets implemented, and that's fine.

Consider using a filter to block most of your files

Negation filters have noticeable performance penalty in my experience (for even less complex filter rules) which also affects every single query(and likely disk changes too, if I recall correctly). I had tried to use them before the arrival of 'omit results' feature, I ended up using multiple instances of E, one with db exclusions instead of filters, and one without.

[void]

Thanks for sharing the result omissions.

Some suggestions to improve loading performance:



Avoid wildcards and regex.
Finding absolute paths is instant.
For example, try to use:

\\server\share\subfolder

c:\folder\subfolder

It's the complex regex filters that are taking 99.99% of the time to lookup.

c:\folder\subfolder\*

=>

c:\folder\subfolder

-the wildcard filter will be a lot slower than the absolute path lookup.



I don't support ext:
But probably should -added to my TODO list..



Apply to folders only where possible.
You will naturally have less folders than files.

regex:\\folder-name\\

=>

folder-name

-No need to use regex here, just do a full folder name match.



If possible omit the whole folders closer to the root, instead of applying a regex filter to files in a certain folder.
For example:

regex:^c:\\folder\\subfolder\\.*\.tmp$

=>

c:\folder\subfolder

-use a folder only filter, not sure if it's practical for your case, but the folders do look temporary or cache-like.

[win32]

"ext:tmp" was some late night addition(as many of that list), removed it.
I finally did some consolidations, still the time remains around 10secs due to the regexes, but all good, it's not really an issue to me.
Thank you for your interest in this, void!