Everything + ES.. how to know when indexing is done?

If you are experiencing problems with "Everything", post here for assistance.
Post Reply
ilgrank
Posts: 17
Joined: Mon Jul 06, 2020 1:49 pm

Everything + ES.. how to know when indexing is done?

Post by ilgrank » Fri Jul 17, 2020 3:17 pm

Hello
I'm using Everything + ES to search for files in a batch files.
I currently have some problems:
1) After launching Everything, it takes some time for it to index all files.
I've inserted a sleep of 400 seconds before invoking ES, but it is not ideal as some computer does not complete the indexing before ES invocation.
Is there a way to know Everything is ready (ie: it has finished indexing all the disks/files)?

2) ES currently seem not to be supporting a file containing list of files to look for, so I have to insert the list as a very long regex, i.e.:
-r "^fileA.txt$|^fileB.txt$|^fileC.txt$" or filelist:"fileA.txt|fileB.txt" and so on. (thanks @notnull for pointing out filelist)
Can a list feature be implemented?

3) wouldn't it be more practical to have a truly command-line everything? (ie, ES with the Everything engine built-in)? Now the two processes are async, and this leads to a lot of problematic scenarios.

Thanks @Void!
Last edited by ilgrank on Fri Jul 17, 2020 7:53 pm, edited 2 times in total.


NotNull
Posts: 2156
Joined: Wed May 24, 2017 9:22 pm

Re: Everything + ES.. how to know when indexing is done?

Post by NotNull » Fri Jul 17, 2020 7:20 pm

ilgrank wrote:
Fri Jul 17, 2020 3:17 pm
2) ES currently seem not to be supporting a list of files to look for, so I have to insert the list as a very long regex, i.e.:
-r "^fileA.txt$|^fileB.txt$|^fileC.txt$"

Code: Select all

ES.exe filelist:"everything.exe|everything.ini|some file.txt"
( See https://www.voidtools.com/support/every ... #functions)

ilgrank wrote:
Fri Jul 17, 2020 3:17 pm
3) wouldn't it be more practical to have a truly command-line everything? (ie, ES with the Everything engine built-in)? Now the two processes are async, and this leads to a lot of problematic scenarios.
No. That would mean 2 separate databases to maintain with (hopefully) the same content. *That* is asking for problematic scenarios.

ilgrank
Posts: 17
Joined: Mon Jul 06, 2020 1:49 pm

Re: Everything + ES.. how to know when indexing is done?

Post by ilgrank » Fri Jul 17, 2020 7:38 pm

Thanks @therube , that's perfect!
Thanks @notnull too, since he's the author of the original post
For my considerations about this workaround being 'practical', see my next reply
Last edited by ilgrank on Fri Jul 17, 2020 8:03 pm, edited 5 times in total.

ilgrank
Posts: 17
Joined: Mon Jul 06, 2020 1:49 pm

Re: Everything + ES.. how to know when indexing is done?

Post by ilgrank » Fri Jul 17, 2020 7:42 pm

NotNull wrote:
Fri Jul 17, 2020 7:20 pm
ilgrank wrote:
Fri Jul 17, 2020 3:17 pm
2) ES currently seem not to be supporting a list of files to look for, so I have to insert the list as a very long regex, i.e.:
-r "^fileA.txt$|^fileB.txt$|^fileC.txt$"

Code: Select all

ES.exe filelist:"everything.exe|everything.ini|some file.txt"
( See https://www.voidtools.com/support/every ... #functions)
My bad for not describing my request properly. I meant an input file containing a list of files. I've edited the original question to clarify.
Surely I had missed filelist, but still is less practical than an actual file, since you incur in command line lenght limits, while providing a file that's not an issue.
Other CLI tools, like 7Z, simply allow you to provide an input file. It's way easier and practical,especially when you have a long list of files to search for. I hope you'll agree ;)
NotNull wrote:
Fri Jul 17, 2020 7:20 pm
ilgrank wrote:
Fri Jul 17, 2020 3:17 pm
3) wouldn't it be more practical to have a truly command-line everything? (ie, ES with the Everything engine built-in)? Now the two processes are async, and this leads to a lot of problematic scenarios.
No. That would mean 2 separate databases to maintain with (hopefully) the same content. *That* is asking for problematic scenarios.
Why 2 databases? I can have as many everything instances running, all using the same database. What I'm proposing is two different 'interfaces' (one graphical, one command line), using the same engine.
And besides, I'm not using the GUI at all in scripts, and as you can see from the workaround above (Which, for me at least doesn't work) you need to do lots of trickeries just to launch a GUI in a script that runs with no GUI at all just to invoke a CLI that interfaces with the nonvisible GUI.. call that simple or convenient... :shock:

NotNull
Posts: 2156
Joined: Wed May 24, 2017 9:22 pm

Re: Everything + ES.. how to know when indexing is done?

Post by NotNull » Fri Jul 17, 2020 8:04 pm

ilgrank wrote:
Fri Jul 17, 2020 7:42 pm
why 2 databases? I can have as many everything instances running, all using the same database.
Actually: that is not possible. Every instance has it's own database in RAM.
(Everything keeps it's database in RAM all the time and will only write it to disk if instructed to do so or on exit)

You can open the same database from disk, but that will in almost all cases already be outdated. Differences will occur if you attach and remove a new removable disk when instance 1 is running and instance 2 is not (simple example).

ilgrank wrote:
Fri Jul 17, 2020 7:42 pm
What I'm proposing is two different 'interfaces' (one graphical, one command line), using the same engine.
That still wouldn't change the issue of the Everything database being "locked".

NotNull
Posts: 2156
Joined: Wed May 24, 2017 9:22 pm

Re: Everything + ES.. how to know when indexing is done?

Post by NotNull » Fri Jul 17, 2020 8:17 pm

To clarify a little bit better (I hope):
ilgrank wrote:
Fri Jul 17, 2020 7:42 pm
you need to do lots of trickeries just to launch a GUI in a script that runs with no GUI at all just to invoke a CLI that interfaces with the nonvisible GUI.. call that simple or convenient... :shock:
Everything.exe has a lot of roles. Among others:
- installer
- Service to access MFT and USN Journal
- Database Manager
- User GUI
- HTTP Server


The "database manager" is not part of the Everything Service (as a lot of people think). It is started with Everything.exe -startup (starts database manager without GUI) or Everything.exe (database manager + GUI).

You need to start the "database manager". ES.exe needs it: ES query => Database Manager => Lookup in database => results to ES.exe.
You don't need to start the GUI.

ilgrank
Posts: 17
Joined: Mon Jul 06, 2020 1:49 pm

Re: Everything + ES.. how to know when indexing is done?

Post by ilgrank » Fri Jul 17, 2020 8:24 pm

Thanks for the detailed explanation @notnull, but being a (very basic) programmer too, I still don't see a reason why a standalone CLI-only version could not exist, using the same code, and just not using a GUI, for people that doesn't need/use the GUI, and just exposing a CLI.
I mainly use it thru PSEXEC or WinRM, and so the GUI (again, to me at least) is's more of an hurdle than a benefit.

NotNull
Posts: 2156
Joined: Wed May 24, 2017 9:22 pm

Re: Everything + ES.. how to know when indexing is done?

Post by NotNull » Fri Jul 17, 2020 8:37 pm

I suspect we are on different wavelengths here, while wanting to say the same thing. Let's dig deeper ...

Who would manage the database with all files and folders in your standalone CLI-only version? ES.exe?


( Bit of background: ES.exe started out as a simple test-program for the developers tha use the SDK (I am not a programmer/developer, btw)).

ilgrank
Posts: 17
Joined: Mon Jul 06, 2020 1:49 pm

Re: Everything + ES.. how to know when indexing is done?

Post by ilgrank » Fri Jul 17, 2020 8:41 pm

NotNull wrote:
Fri Jul 17, 2020 8:37 pm
I suspect we are on different wavelengths here, while wanting to say the same thing. Let's dig deeper ...
Yep, you're right,sorry :)
NotNull wrote:
Fri Jul 17, 2020 8:37 pm
Who would manage the database with all files and folders in your standalone CLI-only version? ES.exe?
let me clarify: I do not propose to allow ES to run standalone, rather to build a (let's call it) everything-cli
being the full everything package, just compiled *without* a GUI, and with a richer command line option set (ie, the everything one plus the one of ES)
In my scenario, everything-cli would start, create the in-memory database, answer the query, and quit outputting the results and (optionally) not even writing the everything.db

Please pardon me if I'm not 100% clear, but english is not my mother tongue :)

void
Site Admin
Posts: 5558
Joined: Fri Oct 16, 2009 11:31 pm

Re: Everything + ES.. how to know when indexing is done?

Post by void » Mon Jul 20, 2020 1:42 am

For the OP:

To wait for the Everything database to load when using ES, use the -timeout command line option:
es.exe -timeout 999999

where 999999 is the number of milliseconds to wait. The default is 0.

Technical details:
To find when Everything indexing is done you will need to poll Everything_IsDBLoaded.

I've added the registered window message "EVERYTHING_DB_LOADED" for Everything 1.5, you can register this message, test Everything_IsDBLoaded and if it fails, wait for the "EVERYTHING_DB_LOADED" event.

You can check Everything is running by grabbing the tray window:
HWND everything_hwnd = FindWindow("EVERYTHING_TASKBAR_NOTIFICATION",NULL);
If the window does not exist, register the window message "EVERYTHING_IPC_CREATED", run Everything.exe and wait for the "EVERYTHING_IPC_CREATED" event.
let me clarify: I do not propose to allow ES to run standalone, rather to build a (let's call it) everything-cli
being the full everything package, just compiled *without* a GUI, and with a richer command line option set (ie, the everything one plus the one of ES)
In my scenario, everything-cli would start, create the in-memory database, answer the query, and quit outputting the results and (optionally) not even writing the everything.db
I'm open to discussion on the topic.
My biggest concern is what is indexed? how is this managed, to you want to specify in each cli call what is indexed, eg:
cli.exe -index "C:";"D:";"\\server\share" -search "abc 123"
Would this store a database for C:, D:? -Indexing "\\server\share" over and over would be slow.
How would you want this updated?
..It gets tricky without a user interface and user interaction.

Are you looking to index only NTFS volumes? would adding a command to Everything to dump the NTFS MFT as UTF-8 CSV be useful?

ilgrank
Posts: 17
Joined: Mon Jul 06, 2020 1:49 pm

Re: Everything + ES.. how to know when indexing is done?

Post by ilgrank » Mon Jul 20, 2020 10:16 am

Hello @Void, thanks for the very informative reply
I've searched for Everything 1.5 but I've been not able to find it, and judging form the latest posts on the matter, a 1.5Alpha is not yet available.

As a stopgap, I had used (successfully) a variation of @NotNull proposal, ie:

Code: Select all

everything -startup & start /w everything -startup -update
es filelist:"a.txt|b.txt|c.txt"
everything -quit
del everything.ini
del everything.db
this allows both not launch the GUI and to wait for the update to finish.
I've tried also your suggestion

Code: Select all

everything -startup 
es -timeout 999999 filelist:"a.txt|b.txt|c.txt"
but it does not return (not after 5 minutes at least) on any of my test machines (task is being run as local system)

I do still think tho that a single-file tool would be easier and more practical to use. Following you example

Code: Select all

cli.exe -index "*:";"\\server\share" -search "abc 123"
would be absolutely perfect
In my case i do *not* need to save the database, as each search would be a one timer, but I do understand that's just my scenario. A more general approach would be

Code: Select all

cli.exe -index "*:";"\\server\share" -search "abc 123" -export-txt results.txt -no_DB_save
giving for granted that, as in the GUI version, saving the DB on quit is the default (that I want to avoid)

NotNull
Posts: 2156
Joined: Wed May 24, 2017 9:22 pm

Re: Everything + ES.. how to know when indexing is done?

Post by NotNull » Mon Jul 20, 2020 3:42 pm

You can use the parameter -nodb with Everything.exe. No database will be saved.

Running without creating an INI file: use parameter -config "nul\dummy.ini"

Combined:

Code: Select all

everything.exe   -nodb   -config "nul\dummy.ini"

ilgrank
Posts: 17
Joined: Mon Jul 06, 2020 1:49 pm

Re: Everything + ES.. how to know when indexing is done?

Post by ilgrank » Mon Jul 20, 2020 5:31 pm

NotNull wrote:
Mon Jul 20, 2020 3:42 pm
You can use the parameter -nodb with Everything.exe. No database will be saved.

Code: Select all

everything.exe   -nodb   -config "nul\dummy.ini"
Thanks @NotNull, yes, that's the idea, but I was meaning it for a so-called "cli.exe" version of Everything. :)

NotNull
Posts: 2156
Joined: Wed May 24, 2017 9:22 pm

Re: Everything + ES.. how to know when indexing is done?

Post by NotNull » Mon Jul 20, 2020 6:09 pm

Ah, OK. Missed that ... Thanks for clarification.

void
Site Admin
Posts: 5558
Joined: Fri Oct 16, 2009 11:31 pm

Re: Everything + ES.. how to know when indexing is done?

Post by void » Wed Jul 22, 2020 7:36 am

I've searched for Everything 1.5 but I've been not able to find it, and judging form the latest posts on the matter, a 1.5Alpha is not yet available.
Everything 1.5 is still in development.

Code: Select all

everything -startup 
es -timeout 999999 filelist:"a.txt|b.txt|c.txt"
but it does not return (not after 5 minutes at least) on any of my test machines (task is being run as local system)
What is shown in the status bar in the Everything GUI?

ilgrank
Posts: 17
Joined: Mon Jul 06, 2020 1:49 pm

Re: Everything + ES.. how to know when indexing is done?

Post by ilgrank » Wed Jul 22, 2020 11:34 am

@Void: Alas, being that is being run by NT\System and that I've invoked it using "-startup", I don't know how to find out what it is displaying in the GUI.
The 'bad' thing is that all invocation methods (es -timeout 999999, start /w everything update) on roughly 1/3 of the computers result in Everything process not giving the prompt back, and thus there I have to kill the everything.exe process in order for the batch file to end.
Is there something I can do to debug why this is happening? currently my batch file is looking like this:

Code: Select all

taskkill /f /im everything.exe
start everything.exe.exe -startup -nodb & start /wait everything.exe -startup -update 
es filelist:"a.txt|b.txt|c.txt" -export-csv c:\temp\out.csv
start everything.exe -quit
taskkill /f /im everything.exe
del everything.*
you may be wondering why all the "start" commands,like in "start everything -qui" I don't know why, but without that, hang rate is slighly higher at 50%.
Last edited by ilgrank on Wed Jul 22, 2020 12:53 pm, edited 1 time in total.

void
Site Admin
Posts: 5558
Joined: Fri Oct 16, 2009 11:31 pm

Re: Everything + ES.. how to know when indexing is done?

Post by void » Wed Jul 22, 2020 11:59 am

Please note when you run Everything as NT Authority\System, Everything will run in session 0.

You will be unable to communicate with Everything running in session 0 from a command prompt running in session 1 due to session 0 isolation.

You can run Everything and ES both in session 0 and they communicate fine..

The following worked for me when running as NT Authority\System:

Code: Select all

start "" "C:\Program Files\Everything\Everything.exe" -startup -nodb
"C:\ES\es.exe" hello -timeout 999999 -export-txt out.txt 
"C:\Program Files\Everything\Everything.exe" -quit
What output are you expecting from:
es filelist:"a.txt|b.txt|c.txt"

-There is no output from ES in session 0. (unless you turn on Interactive Services Detection), you can still export to a file.

ilgrank
Posts: 17
Joined: Mon Jul 06, 2020 1:49 pm

Re: Everything + ES.. how to know when indexing is done?

Post by ilgrank » Wed Jul 22, 2020 12:52 pm

Hello @Void
yes, sorry for the error in the example I provided, I'm using export-csv by the way. I fixed my original message to reflect this.
Using -timeout 99999 most of the times the process just hangs, that's why I switched to not using it, as start gives me success in most (2/3) of cases.
Is there a way do diagnose why everything hangs? (like outputting debug data to a file?)
the batch file is launched by NT Authority\System, and the batch contains calls to both executables.. won't this suffice to have both run in the same session?
I must explicitly specify the -instance flag when running es in the same batch file of everything?

void
Site Admin
Posts: 5558
Joined: Fri Oct 16, 2009 11:31 pm

Re: Everything + ES.. how to know when indexing is done?

Post by void » Fri Jul 24, 2020 8:11 am

I haven't been able to reproduce the issue here..
Is there a way do diagnose why everything hangs? (like outputting debug data to a file?)
Please try enabling debug logging:
Everything -debug-log

This will create a debug log in %TEMP%\Everything Debug Log.txt
%TEMP% will be C:\Windows\Temp for session0.

In the log, you should see the following line:
2020-07-24 17:35:11.485: IPC: execute query: hello

This means Everything has received the ES query.

In the log, you should see the following lines:
2020-07-24 17:35:13.356: IPC: query complete: 4294967295 max results, offset 0, reply hwnd 786512
2020-07-24 17:35:13.407: IPC: watch awake
2020-07-24 17:35:13.458: IPC: Client disconnected.

This means the results have been sent to ES.

In the log, you should see the following line:
2020-07-24 17:35:13.608: cl: "C:\Program Files\Everything\Everything.exe" -quit

This means Everything has received the quit command.

See if adding delays between calls makes any difference.
eg: PING localhost -n 6 >NUL
the batch file is launched by NT Authority\System, and the batch contains calls to both executables.. won't this suffice to have both run in the same session?
Yes, this is fine.
ES can communicate with Everything, and Everything can send a reply to ES.
I must explicitly specify the -instance flag when running es in the same batch file of everything?
If you are also running Everything in session 1, then Yes, this is important.
Please use an instance name with Everything and ES:
Everything -instance session0
ES -instance session0

Without the instance specified, Everything will use the unnamed instance. The ES/Everything in session0 might be trying to communicate with Everything in session1.

ilgrank
Posts: 17
Joined: Mon Jul 06, 2020 1:49 pm

Re: Everything + ES.. how to know when indexing is done?

Post by ilgrank » Fri Jul 24, 2020 7:17 pm

Hello @Void, and thanks for your reply!
I've enabled logging, and *perhaps* I have an hint.
even tho my current batch is this:

Code: Select all

start "" everything.exe -session0 -debug-log -startup -nodb
es -session0 -timeout 999999 filelist:"a.txt|b.txt|c.txt" /a-d -export-txt "c:\temp\files.txt"
everything.exe -session0 -debug-log -quit
I've found that on many computers the everything process was still running even tho it would seems the exit routine is called, given the last lines of the debug log:

Code: Select all

2020-07-24 18:07:47.964: search '<file:><filelist:"a.txt|b.txt|c.txt">' filter '' sort 0
2020-07-24 18:07:47.979: IPC: wait 2
2020-07-24 18:07:47.995: USN 2020-07-24 18:07:48.010: DATA_EXTEND 2020-07-24 18:07:48.026: measure.log
2020-07-24 18:07:48.042: USN 2020-07-24 18:07:48.073: DATA_EXTEND 2020-07-24 18:07:48.089: CLOSE 2020-07-24 18:07:48.151: measure.log
2020-07-24 18:07:48.167: USN 2020-07-24 18:07:48.198: DATA_EXTEND 2020-07-24 18:07:48.214: measure.log
2020-07-24 18:07:48.229: term 00000000006d6ca0, flags: 1150, next: 0000000002ca1ca0, notnext: 0000000000000000
folderop: 1, fileop: 0, term: 
2020-07-24 18:07:48.245: term 0000000002ca1ca0, flags: 3140, next: 0000000000000000, notnext: 0000000000000000
folderop: 384, fileop: 384, term: filelist:a.txt|b.txt|c.txt
2020-07-24 18:07:48.260: USN 2020-07-24 18:07:48.276: DATA_EXTEND 2020-07-24 18:07:48.292: CLOSE 2020-07-24 18:07:48.307: measure.log
2020-07-24 18:07:48.323: USN 2020-07-24 18:07:48.339: DATA_EXTEND 2020-07-24 18:07:48.354: measure.log
2020-07-24 18:07:48.370: SET SORT 0
2020-07-24 18:07:48.385: Failed to notify shell icon 2147500037
2020-07-24 18:07:48.401: USN 2020-07-24 18:07:48.417: DATA_EXTEND 2020-07-24 18:07:48.448: CLOSE 2020-07-24 18:07:48.464: measure.log
2020-07-24 18:07:48.479: new thread (5)
2020-07-24 18:07:48.495: USN 2020-07-24 18:07:48.510: CREATE 2020-07-24 18:07:48.526: everything.exe.tmp
2020-07-24 18:07:48.542: new thread (6)
2020-07-24 18:07:48.557: USN 2020-07-24 18:07:48.573: CREATE 2020-07-24 18:07:48.589: DATA_EXTEND 2020-07-24 18:07:48.604: everything.exe.tmp
2020-07-24 18:07:48.620: new thread (7)
2020-07-24 18:07:48.635: USN 2020-07-24 18:07:48.651: CREATE 2020-07-24 18:07:48.667: DATA_EXTEND 2020-07-24 18:07:48.698: CLOSE 2020-07-24 18:07:48.714: everything.exe.tmp
2020-07-24 18:07:48.729: USN 2020-07-24 18:07:48.745: DELETE 2020-07-24 18:07:48.760: CLOSE 2020-07-24 18:07:48.776: everything.exe.stat
2020-07-24 18:07:48.792: USN 2020-07-24 18:07:48.807: RENAME_OLD_NAME 2020-07-24 18:07:48.823: everything.exe.tmp
2020-07-24 18:07:48.839: USN 2020-07-24 18:07:48.854: RENAME_NEW_NAME 2020-07-24 18:07:48.870: everything.exe.stat
2020-07-24 18:07:48.885: found 0 folders, size 0, db search time taken: 0.251313 seconds
2020-07-24 18:07:48.901: USN 2020-07-24 18:07:48.917: RENAME_NEW_NAME 2020-07-24 18:07:48.948: CLOSE 2020-07-24 18:07:48.964: everything.exe.stat
2020-07-24 18:07:48.979: found 0 files, size 0, db search time taken: 0.005460 seconds
2020-07-24 18:07:48.995: USN 2020-07-24 18:07:49.010: CREATE 2020-07-24 18:07:49.026: es.exe.tmp
2020-07-24 18:07:49.042: set sort 0 1
2020-07-24 18:07:49.057: already sorted
2020-07-24 18:07:49.089: finished sort, time taken 0.100539 seconds
2020-07-24 18:07:49.104: USN 2020-07-24 18:07:49.120: CREATE 2020-07-24 18:07:49.135: DATA_EXTEND 2020-07-24 18:07:49.151: es.exe.tmp
2020-07-24 18:07:49.167: update selection 0.000000 seconds
2020-07-24 18:07:49.198: USN 2020-07-24 18:07:49.214: CREATE 2020-07-24 18:07:49.229: DATA_EXTEND 2020-07-24 18:07:49.245: CLOSE 2020-07-24 18:07:49.260: es.exe.tmp
2020-07-24 18:07:49.276: DB_WAIT: _db_ready_proc waiting...
2020-07-24 18:07:49.292: DB_WAIT: _db_ready_proc waited 0.100180 seconds
2020-07-24 18:07:49.323: USN 2020-07-24 18:07:49.339: DELETE 2020-07-24 18:07:49.354: CLOSE 2020-07-24 18:07:49.370: es.exe.stat
2020-07-24 18:07:49.385: SIZE CHECK 20 20 diff 0 count 0
2020-07-24 18:07:49.401: IPC: query complete: 4294967295 max results, offset 0, reply hwnd 3735598
2020-07-24 18:07:49.432: USN 2020-07-24 18:07:49.448: RENAME_OLD_NAME 2020-07-24 18:07:49.463: es.exe.tmp
2020-07-24 18:07:49.479: IPC: watch awake
2020-07-24 18:07:49.495: USN 2020-07-24 18:07:49.511: RENAME_NEW_NAME 2020-07-24 18:07:49.526: es.exe.stat
2020-07-24 18:07:49.542: IPC: Client disconnected.
2020-07-24 18:07:49.573: USN 2020-07-24 18:07:49.604: RENAME_NEW_NAME 2020-07-24 18:07:49.620: CLOSE 2020-07-24 18:07:49.635: es.exe.stat
2020-07-24 18:07:49.760: EVERYTHING_IPC_COPYDATA_COMMAND_LINE_UTF8
2020-07-24 18:07:49.776: cl: everything.exe  -debug-log -quit
2020-07-24 18:07:49.885: shutdown
2020-07-24 18:07:49.901: monitor thread exit
2020-07-24 18:07:49.917: USN 2020-07-24 18:07:49.932: DATA_TRUNCATION 2020-07-24 18:07:49.963: ES.EXE-8C3EDE77.pf
2020-07-24 18:07:49.979: monitor cleanup
2020-07-24 18:07:49.995: USN 2020-07-24 18:07:50.010: DATA_EXTEND 2020-07-24 18:07:50.026: DATA_TRUNCATION 2020-07-24 18:07:50.042: ES.EXE-8C3EDE77.pf
2020-07-24 18:07:50.057: USN 
Yes, the log seems truncated, but I've another process running after that batch that does a

Code: Select all

ping localhost -n 10
taskkill /f /im everything.exe
taskkill /f /im es.exe
still many computers have everything process running (again, actually two of them:

Code: Select all

Handles  NPM(K)    PM(K)      WS(K)     CPU(s)     Id  SI ProcessName
-------  ------    -----      -----     ------     --  -- -----------
    149      13     2320       9036       0,06    620   0 everything
    145      13     2316       9040       0,09  10092   0 everything
still trying variations of kill to address this issue.. I'll keep this post updated
Again, thanks for your support!

NotNull
Posts: 2156
Joined: Wed May 24, 2017 9:22 pm

Re: Everything + ES.. how to know when indexing is done?

Post by NotNull » Sun Jul 26, 2020 8:08 pm

You should replace
-session0
with
-instance session0


The process you see running is very likely the comaand-line helpscreen (as this is an invalid option).

void
Site Admin
Posts: 5558
Joined: Fri Oct 16, 2009 11:31 pm

Re: Everything + ES.. how to know when indexing is done?

Post by void » Mon Jul 27, 2020 3:54 am

Thanks for the debug log.
Everything.exe -session0
This should not work, it will bring up the command line options help dialog.
As NotNull mentioned, It should be:
Everything.exe -instance session0
2020-07-24 18:07:49.979: monitor cleanup
Everything is most likely processing lots of file system changes.

After this message, there should be the following message before Everything exits:
2020-07-27 13:14:41.588: kill

Please try disabling monitors with the -read-only command line option, they don't need to be running since we just using a snapshot of your file systems:
Everything.exe -no-db -read-only

Code: Select all

start "" everything.exe -instance session0 -debug-log -startup -nodb -read-only
es -instance session0 -timeout 999999 filelist:"a.txt|b.txt|c.txt" /a-d -export-txt "c:\temp\files.txt"
everything.exe -instance session0 -quit

ilgrank
Posts: 17
Joined: Mon Jul 06, 2020 1:49 pm

Re: Everything + ES.. how to know when indexing is done?

Post by ilgrank » Tue Jul 28, 2020 9:42 am

Thanks @Void and @NotNull!
It was my mistake with -session0 was causing the hangs. Running in a console-less thread, I had not spotted what was going on..
The fixed batch does indeed work and no hang detected on 100+ computers.

@Void: using
-read-only
won't conflict with
-nodb
?
reading from the manual:

Code: Select all

-nodb         Do not save to or load from the "Everything" database file.
-read-only    Loads the database in read-only mode.
I had understood that -nodb would not save the db to disk, but -read-only would not update the in-memory database then?

my goal is not save the DB to disk, but disabling filesystem change monitoring too would be sweet, as I just need to search for the files and quit, no monitoring is needed.

To be honest tho, I still think that having a native CLI tool would prove to be more easy and practical..

void
Site Admin
Posts: 5558
Joined: Fri Oct 16, 2009 11:31 pm

Re: Everything + ES.. how to know when indexing is done?

Post by void » Tue Jul 28, 2020 10:19 am

I had understood that -nodb would not save the db to disk, but -read-only would not update the in-memory database then?
Yes, I've updated the documentation.

-read-only will prevent Everything from updating the index.

ilgrank
Posts: 17
Joined: Mon Jul 06, 2020 1:49 pm

Re: Everything + ES.. how to know when indexing is done?

Post by ilgrank » Wed Jul 29, 2020 10:26 am

@void: any chance that you could develop a CLI version?
(not that it matters, but I've made a couple of donations in the past, you're doing a really great work with Everything)
Again, thanks

ilgrank
Posts: 17
Joined: Mon Jul 06, 2020 1:49 pm

Re: Everything + ES.. how to know when indexing is done?

Post by ilgrank » Wed Jul 29, 2020 5:23 pm

Hello @Void
tried today adding -read-only flag:
start "" everything.exe -read-only -instance session0 -startup -nodb

but the hangs are back.
The log follows:

Code: Select all

2020-07-29 19:47:59.227: Everything
2020-07-29 19:47:59.232: Version 1.4.1.969 (x64)
2020-07-29 19:47:59.233: Windows NT 6.2
2020-07-29 19:47:59.234: Processors 8
2020-07-29 19:47:59.235: IsAdmin 1
2020-07-29 19:47:59.236: AppData 0
2020-07-29 19:47:59.237: Service 0
2020-07-29 19:47:59.238: cmdline everything.exe  -debug-log -instance session0 -read-only -startup -nodb
2020-07-29 19:47:59.239: load localization
2020-07-29 19:47:59.240: ole initialize: 00000000
2020-07-29 19:47:59.241: create mutex
2020-07-29 19:47:59.248: sys img list 00000000004ec3a0 00000000004ed9c0 3 0
2020-07-29 19:47:59.255: default font MS Shell Dlg 2 -16
2020-07-29 19:47:59.257: load everything icon
2020-07-29 19:47:59.258: bookmarks
2020-07-29 19:47:59.259: filters
2020-07-29 19:47:59.260: search history
2020-07-29 19:47:59.261: set hook
2020-07-29 19:47:59.262: create tray
2020-07-29 19:47:59.286: no db file
2020-07-29 19:47:59.288: enter main loop
after the cmdline you see in the log, there's the call to es:
es.exe -instance session0 -timeout 99999 filelist:"A.txt|b.txt|c.txt" -export-txt files.txt

but it just hangs there. As soon as I kill es from another prompt:
taskkill /f /im es.exe

the batch file goes to the next command:
everything.exe -instance session0 -quit

and the thread correctly ends:

Code: Select all

2020-07-29 19:48:12.227: EVERYTHING_IPC_COPYDATA_COMMAND_LINE_UTF8
2020-07-29 19:48:12.229: cl: everything.exe  -instance session0 -quit
2020-07-29 19:48:12.235: shutdown
2020-07-29 19:48:12.253: read_directory_changes stop thread
2020-07-29 19:48:12.254: read_directory_changes stop get fd thread
2020-07-29 19:48:12.255: read_directory_changes remove events
2020-07-29 19:48:12.256: read_directory_changes free events
2020-07-29 19:48:12.257: read_directory_changes remove got fd events
2020-07-29 19:48:12.258: read_directory_changes free got fd events
2020-07-29 19:48:12.259: read_directory_changes after update events
2020-07-29 19:48:12.259: read_directory_changes free monitors
2020-07-29 19:48:12.261: kill
2020-07-29 19:48:12.262: event start 0000000000000000
so to make a long story short, ES doesn't seem to play nicely with the
-read-only
flag..

void
Site Admin
Posts: 5558
Joined: Fri Oct 16, 2009 11:31 pm

Re: Everything + ES.. how to know when indexing is done?

Post by void » Thu Jul 30, 2020 3:02 am

Thank you for your support.
@void: any chance that you could develop a CLI version?
Yes, I'll consider adding an option to Everything.exe.
Thank you for your suggestion.
so to make a long story short, ES doesn't seem to play nicely with the -read-only flag..
Ah, sorry, the -read-only command line option also prevents a new database from being created...

Please try without -read-only and with the -monitor-pause command line option:

Code: Select all

start "" everything.exe -monitor-pause -instance session0 -startup -nodb

ilgrank
Posts: 17
Joined: Mon Jul 06, 2020 1:49 pm

Re: Everything + ES.. how to know when indexing is done?

Post by ilgrank » Mon Aug 03, 2020 9:09 am

Thanks @Void!
I can confirm that this final command line works without any hang, tested on 200+ computers with no problems.
Thanks again for your support!
Just one last curiosity.. why
start "" everything.exe
is still needed? (I can confirm it does hang if used without)
I would expect the first part of the command line to just start the thread...

Post Reply