I'm positive about this, after having had some "gaslighting" episode in which I hadn't been sure if perhaps I had done some things wrong. This may (?) occur when I "remove" (=button) some other (sic!) drive from the index, e.g. some "G:" drive, and my T: and U: drives (drive letters being fixed, and they were never "G:", I am sure of that) have lost their respective index, totally, i.e. even a simple t: or t:\ or t:\* will not show anything anymore.
Settings: Include in db yes, Enable USN Journal yes (but that always switches to no when offline, but then back to on when connected again, so this does not seem to be a / the problem?), Load USN Journal into the Recent Changes db no, Monitor changes yes, and (forced by EV): Max size 32768, Allocation delta 4096: to these values it often reverts from my 131072 / 8192, but here again, given on these drives are only about 10,000 files each, this forced change to (much lesser) values should not be a problem? But it seems to indicate problems; also a problem indicator: Often, the "Remove" button is greyed-out, both for offline, and for currently-connected drives.
I use 1.5a 1.5.0.1383a (x64): Is the (main) problem (i.e. the loss of whole drive indexes) known for that? Has it been addresses in a newer release? I would not really like to go back to 1.4 since all new functionality would be lost then, but since I heavily rely upon EV, for numerous checks for the location (or even just the existence!) of files, I would be forced to revert to 1.4 if the frailness of the drive indexes in 1.5 remains.
Again, I have observed missing entries "which should have been there though" for many months now, but I'm really old, have failing memory, so this "gaslighting" "had" me, again and again, but I now am sure, certain, that these (partial, or even whole-drive) losses are caused by an internal EV 1.5a problem, not from my memory fooling me.
Also, I very often "force index rebuild", before switching off drives (and also in order to then being able to switch them off / disconnect them "safely" (and my pc tells me it's safe then), so disconnecting drives "at a bad time" cannot cause my problems; of course, I sometimes also switch the whole pc off, in the evening, when external drives may yet be connected, but I hope that such "controlled, regular" (sic!) shutdown of the pc can not (?) harm the EV index? (I have NO bluescreens, power cuts or the like, fortunately.)
As said, since, in my Seventies, I cannot rely on my memory anymore, I MUST rely on the EV index, and with the above 1.5a version, that seems impossible now! ;-(
1.5 alpha sometimes loses indexes of offline drives
Re: 1.5 alpha sometimes loses indexes of offline drives
Tools | Options | Indexes -> <FAT|NTFS>Re: 1.5 alpha sometimes loses indexes of offline drives
Please check Tools -> Debug -> Statistics -> Build -> Last Rebuild Reason
Re: 1.5 alpha sometimes loses indexes of offline drives
Thank you for your answers.
Yesterday, I hadn't found my post even hours later, and after trying to update 1.5 (I had to google for the download location, hadn't found it on your site), I lost the rest of my indexes, so I de-installed 1.5.
Obviously, I had not set "Automatically remove offline volumes" but then the update to 1.5 latest might have reset that setting to "remove" indeed, by some default?
Anyway, after installing 1.4 again, and doing some checks, I have discovered that in 1.5, I had not only lost whole offline drives, but I had and have to confirm, unfortunately, that 1.5 "gaslighted" me indeed, also - I'm positive about this - losing numerous (but specific) files (perhaps some hundreds, or then perhaps several dozen "only", out of a tiny 4-digit number of files, not in specific sub-folders, but spread "all over the place" of those drives (and neither having special characteristics in their file names, nor being "latest arrivals" or such, but "chosen to be left out" by 1.5 seemingly in a totally aleatoric way) - now with 1.4, they are all present again, and, as said, I had regularly "forced re-index", after adding new files to those drives, in order to be sure that, after de-connecting them, the EV index would (remain to) be complete - obviously it wasn't.
I have to say that on the first (?) page of the 1.5 options (I speak from my memory, after de-install of 1.5), there is / was a setting like "maximum size" with my number 131072, and I had understood this setting NOT as the max size of the whole db (?), but as the max number then allowed for the specific settings "max size" of the different drives, but I may have been mistaken here, and, with a max size of 131072 kb for the whole db, I might have had systematic index data losses, over time?
Since I index (in 1.5 and in 1.4) almost all attributes, etc. (with the only exception for folder sizes, all other check boxes are checked then), and often have quite long file names (often 150-200 characters but path lengths are always beneath 255 characters, so overlong path names could not have caused the problem), my indexes are rather bulky, and, now back in 1.4, and with 2 of my external drives not yet reconnected / indexed yet, my 1.4 EV index db is already about 115,000 kb: This indicates that IF that "131072 kb" setting on the first page of the 1.5 options is / was the total db size, and not a limit for the specific drive index sizes (as I had understood it, as said), it's obvious that that total this, total (?), db size was not sufficient for holding all my data, hence "allocation delta" losses of 8192 kb here and there, but without EV indicating such automatic discards of index data, so if they took place, I never became aware of them, otherwise I would have enlarged the 131072 kb setting; on the other hand, the specific 131072 settings, for each drives, obviously were plenty of room to hold all my index data for every drive, IF there was not a global cut-off by the 131072 setting on that first option page - I could not find an explanation for that value on 1.5's for that, and in the 1.4 options, there is no such "global value" or a corresponding setting, there's just the specific settings for each drive.
To clarify again, if I had wrong "remove offline" settings, I would have regularly lost whole drives (which just occurred yesterday), but over many months, with 1.5, I had just lost specific (sic!) file entries from my drive indexes, and even multiple "force index" did not re-introduce these files into the index*, while now in 1.4, they are all in there again, so "file name specifics" or other aspects on the "file system side" obviously did not cause the problem, the files are "ok", but either (?) were left out from the index by 1.5, or (?) then discarded later on, unfortunately I cannot say which one of these alternatives applied, or even if it might have been a mix of both.
I have another question now. Unfortunately, I had de-installed 1.5, without either looking at its db size before, or using EV to note the location of the relevant files, and now, after de-install, I have a mix of 1.5 residuals, and 1.4 files, and I see that I have TWO ini-files:
Everything.ini in C:\Program Files\Everything, just with:
run_as_admin=0
allow_http_server=1
allow_etp_server=1
(and additions here are refused by my Windows system (W10 Prof engl.) anyway)
And then Everything.ini in C:\Users\MyName\AppData\Roaming\Everything
with lots of (default) entries, among them
statusbar_selected_item_format=
which I (EV not running then, obviously) changed to
statusbar_selected_item_format=$b|$p|$n
but even after an intermediate pc shutdown / restart, these new settings are not considered by 1.4, it stays at its defaults; what do I wrong?
*=I seriously think now that I, personally, should stay at 1.4 since again and again in the past, I had done "Force re-indexes" since I thought, for multiple files, "it MUST be in there, somewhere, am I really so wrong already?", and then, after the re-index, it didn't appear though either, and I resigned myself to a, "again, my memory had me, all the worse then for me!" - and, frankly, a tool gaslighting in such a way, old people who have serious memory problems anyway, is unbearable for them, making them even more ill then they are already... whilst 1.4 works fine, obviously, and very thankfully, and I'm very pleased to discover lots of files again now I had, for months, thought they only existed in my head! (I have backups, but don't index those backup drives, in order to "streamline" my searches; I have never used the Windows index service and have no intention to do so, but rely exclusively on EV.)
Yesterday, I hadn't found my post even hours later, and after trying to update 1.5 (I had to google for the download location, hadn't found it on your site), I lost the rest of my indexes, so I de-installed 1.5.
Obviously, I had not set "Automatically remove offline volumes" but then the update to 1.5 latest might have reset that setting to "remove" indeed, by some default?
Anyway, after installing 1.4 again, and doing some checks, I have discovered that in 1.5, I had not only lost whole offline drives, but I had and have to confirm, unfortunately, that 1.5 "gaslighted" me indeed, also - I'm positive about this - losing numerous (but specific) files (perhaps some hundreds, or then perhaps several dozen "only", out of a tiny 4-digit number of files, not in specific sub-folders, but spread "all over the place" of those drives (and neither having special characteristics in their file names, nor being "latest arrivals" or such, but "chosen to be left out" by 1.5 seemingly in a totally aleatoric way) - now with 1.4, they are all present again, and, as said, I had regularly "forced re-index", after adding new files to those drives, in order to be sure that, after de-connecting them, the EV index would (remain to) be complete - obviously it wasn't.
I have to say that on the first (?) page of the 1.5 options (I speak from my memory, after de-install of 1.5), there is / was a setting like "maximum size" with my number 131072, and I had understood this setting NOT as the max size of the whole db (?), but as the max number then allowed for the specific settings "max size" of the different drives, but I may have been mistaken here, and, with a max size of 131072 kb for the whole db, I might have had systematic index data losses, over time?
Since I index (in 1.5 and in 1.4) almost all attributes, etc. (with the only exception for folder sizes, all other check boxes are checked then), and often have quite long file names (often 150-200 characters but path lengths are always beneath 255 characters, so overlong path names could not have caused the problem), my indexes are rather bulky, and, now back in 1.4, and with 2 of my external drives not yet reconnected / indexed yet, my 1.4 EV index db is already about 115,000 kb: This indicates that IF that "131072 kb" setting on the first page of the 1.5 options is / was the total db size, and not a limit for the specific drive index sizes (as I had understood it, as said), it's obvious that that total this, total (?), db size was not sufficient for holding all my data, hence "allocation delta" losses of 8192 kb here and there, but without EV indicating such automatic discards of index data, so if they took place, I never became aware of them, otherwise I would have enlarged the 131072 kb setting; on the other hand, the specific 131072 settings, for each drives, obviously were plenty of room to hold all my index data for every drive, IF there was not a global cut-off by the 131072 setting on that first option page - I could not find an explanation for that value on 1.5's for that, and in the 1.4 options, there is no such "global value" or a corresponding setting, there's just the specific settings for each drive.
To clarify again, if I had wrong "remove offline" settings, I would have regularly lost whole drives (which just occurred yesterday), but over many months, with 1.5, I had just lost specific (sic!) file entries from my drive indexes, and even multiple "force index" did not re-introduce these files into the index*, while now in 1.4, they are all in there again, so "file name specifics" or other aspects on the "file system side" obviously did not cause the problem, the files are "ok", but either (?) were left out from the index by 1.5, or (?) then discarded later on, unfortunately I cannot say which one of these alternatives applied, or even if it might have been a mix of both.
I have another question now. Unfortunately, I had de-installed 1.5, without either looking at its db size before, or using EV to note the location of the relevant files, and now, after de-install, I have a mix of 1.5 residuals, and 1.4 files, and I see that I have TWO ini-files:
Everything.ini in C:\Program Files\Everything, just with:
run_as_admin=0
allow_http_server=1
allow_etp_server=1
(and additions here are refused by my Windows system (W10 Prof engl.) anyway)
And then Everything.ini in C:\Users\MyName\AppData\Roaming\Everything
with lots of (default) entries, among them
statusbar_selected_item_format=
which I (EV not running then, obviously) changed to
statusbar_selected_item_format=$b|$p|$n
but even after an intermediate pc shutdown / restart, these new settings are not considered by 1.4, it stays at its defaults; what do I wrong?
*=I seriously think now that I, personally, should stay at 1.4 since again and again in the past, I had done "Force re-indexes" since I thought, for multiple files, "it MUST be in there, somewhere, am I really so wrong already?", and then, after the re-index, it didn't appear though either, and I resigned myself to a, "again, my memory had me, all the worse then for me!" - and, frankly, a tool gaslighting in such a way, old people who have serious memory problems anyway, is unbearable for them, making them even more ill then they are already... whilst 1.4 works fine, obviously, and very thankfully, and I'm very pleased to discover lots of files again now I had, for months, thought they only existed in my head! (I have backups, but don't index those backup drives, in order to "streamline" my searches; I have never used the Windows index service and have no intention to do so, but rely exclusively on EV.)
Re: 1.5 alpha sometimes loses indexes of offline drives
Tools -> Options -> Indexes -> Enable Journal / Maximum SizeI have to say that on the first (?) page of the 1.5 options
The maximum size settings controls the maximum size of the Journal. (not the index)
There's no size limit for the index.
Missing a few files is odd.I had just lost specific (sic!) file entries from my drive indexes
Usually the whole drive index will be lost.
The files could have been omitted or there might have been an active filter. (active search filters are shown in the status bar on the right)
Maybe Everything rescanned the wrong drive..
I would need more information about your setup to investigate further.
monitor_file_operation might have caused the issue. (it's currently changing things for offline volumes)
Did you only close Everything (not exit Everything?)but even after an intermediate pc shutdown / restart, these new settings are not considered by 1.4, it stays at its defaults; what do I wrong?
which I (EV not running then, obviously) changed to
Please exit Everything before changing your Everything.ini
Everything -> File -> Exit
-or-
Copy and paste the following into your Everything search box:
/statusbar_selected_item_format=$b|$p|$nPress ENTER in your Everything search box.
If successful, statusbar_selected_item_format=$b|$p|$n is shown in the status bar for a few seconds.
Re: 1.5 alpha sometimes loses indexes of offline drives
Thank you very much for your detailed answers!
Inifile: my (stupid) mistake: I had closed EV, not exited it, hadn't been aware of there being any difference: I had right-clicked on its symbol, then "close window", then my inifile edit seemed to be successful (was saved with the new setting), but upon your hint, I checked the inifile again now, and my edit wasn't there. Now, following your instructions (in EV: File - Exit), everything works as expected. Sorry!
Journal size vs. Index sizes: Yesterday, I wrote, and today, I write, having deleted 1.5 (more or less), thus writing "from memory", thus my mix-up: sorry again! Hence, we can say, even 1.5 does NOT have a global max index size setting, overriding the specific (i.e. drive) max index sizes, but see below.
The 1.5 monitor_file_operation setting: Obviously, I can't tell about that setting anymore now, but I'm perfectly certain I never hampered with the 1.5 ini-files for anything else, except that I changed the statusbar setting (which worked fine in 1.5, and works fine now again in 1.4), so the setting you mention should have been at its default.
Obviously, 1.4 does not have such a setting, but just (in my case, all defaults):
filelists=
filelist_monitor_changes=
folders=
folder_monitor_changes=
folder_buffer_size_list=
folder_rescan_if_full_list=
folder_update_types=
folder_update_days=
folder_update_ats=
folder_update_intervals=
folder_update_interval_types=
while the info in your link,
monitor_file_operation
Set to 1 to immediately update FAT, network drive and folder indexes when successfully renaming, moving or deleting a file. (default)
Set to 0 to update through ReadDirectoryChanges or SHChangeNotify events.
is way too technical for me to understand what it really would do in both cases, and anyway, all those settings left to their respective defaults, even their interaction should not cause such problems.
So, unfortunately, there is some real problem here, with 1.5.
I'm certain I neither try to "force rebuild" ("rebuild" always meaning "force rebuild" here now) the wrong drives, neither do I then try to search the wrong drives: I have not only fixed the drive letters, all within the second half of the ABC, so that temporary, Windows-driven drive-letter assignments will never interfere, and whenever I do rebuilds, even though then the rebuild is done for all drives then connected (incl. c:, d:, and so on), I always check the drive letters / names, and thankfully, EV also displays the name, e.g.
T1 (T:)
meaning drive t:, with name T1 (names T2 and T3 being my backup drives for this example, not with fixed drive letters, so when they are connected, Windows assigns them letters like H: or J: or such, within the the first half of the ABC, so I'm really sure which drives are connected / updated / searched at any given moment).
I'm also certain no filters are intervening at any time since I don't use filters, I just use (sometimes elaborate) search string, but when I'm "Desperately Seeking Susan", i.e. some file I couldn't find, I just search "susan" (without the quotes) in the end, not even specifying the drive anymore, so filter, or, more generally, search syntax, problems definitely are not the problem (I use the present tense, but always mean the past, with 1.5, since no problems with 1.4, thankfully).
Also, when I don't find some file which "should be there", I do that rebuild, and at least immediately after the rebuild, the missing files should show up, but they don't, and sometimes, I do even do this with two external drives, not being sure on which one the missing files would have meed placed... and none of the two, with fresh rebuilds, then shows them! (Now, back to 1.4, I have the same problem with not always knowing on which drive I had stored my file, but I search for key words of the file name, and then EV shows me the path, so I know which drive to connect to access it; some people use RAIDS but in them, all drives run all the time which multiplies the wear, compared to drives which you only connect / run one day a week or so.)
Thus, at least immediately after such a rebuild, the files should be retrieved, but definitely, they are not; yesterday, I hadn't remembered yet that very often, I had done such, then systematically unsuccessful, rebuilds (incl. for alternative drives where the missing files in question also could have been, and are in fact, as my no-problem searches with 1.4 now prove), and since we have out-ruled that some "global" max size could interfere with the individual, drive, max sizes, which, at 131,072 kb each, are multiples of the max size provision really needed (which for these drives might be in the range of 20k or 30k kb (we're speaking of thousand kb's here, not of kb's, 131,072 kb is something near / about 130 mb, that's plenty of room for a file names / attributes index), certainly not more),
it unfortunately becomes evident that those rebuilds, for external drives at least (I have not yet encountered such "files missing from the drive" problems with c: or d:), and with 1.5's default settings insofar, do not process all files present correctly (while 1.4, for the same drives, does).
Btw., if I understand the design correctly, the (also drive-specific) "allocation delta" is the size of the ("earlier", "first-in") file / folder index entries which EV discards when, in "normal run", the addition of new entries ("last-in") would otherwise cause an "overflow" of the set index max size for that drive; whilst obviously, it would be / have been preferable that in such cases, EV notified the situation to the user, inviting them to resize the max size setting in case, instead of just deleting thousands of index entries "just like that", behind the scene, without the user even knowing, it should be evident here that this "allocation delta problem", as I might call it, should not cause the bigger problem described here, since within or directly after a fresh rebuild, with rebuild data to be processed just weighting a fraction of the index' current "max size" setting (just some mb vs. about 130 mb), the "allocation delta" function should never be triggered, thus should discard nothing, so it seems (i.e. every forced, or automated, full or partial) data retrieval (= data writing to the index) misses some data.
The missing files do not (sic!) share some specific characteristic, e.g. they don't contain special characters, are "system" or "hidden" or the like, but as said (I think), except for "folder sizes", I had checked all check boxes to include my files' characteristics, etc., into the index, so - just an idea - some of those checks = some retrieval command might cause skipping of entries of the MTF table? But in some aleatoric way then, since, as said, it's not some file attribute or combination of attributes which causes the leaving-out, and my MTF seems to be ok since otherwise I would encounter the same problems with 1.4, logically.
Did I say that for all of my (EV-indexed) drives, I have Include in db yes, enable USN journal yes, monitor changes yes, but Load recent changes from USN journal NO? Since leaving out the latter, without me really understanding "what it does", is recommended by you, on your site.
Obviously, I don't have the 1.5 settings at hand anymore, but I'm certain I hadn't fiddled around with some, by inifile edits or otherwise, except for the usual Search settings in the Options, e.g. Fast ASCII yes, Match path yes, Match whole filename, etc., those usual, "benign" settings which shouldn't cause any specific problems - and a reset the normal-font setting to "default but 12 point" indeed.
(Btw, it just occurs to me that I have never tried to then search those files, missing from the "normal" search results (i.e. in the EV window), by es.exe, had used es.exe only to create export lists for Android...)
On the other hand, other users should have encountered similar problems with 1.5? I think possible fails in my ssd (my EV index is on c:, obviously, the external drives being hdd's) could not be the culprit here since the ramifications of physical damages would not be limited to specific files, just discarding them from the result list, but would show "all over the place" of such a result list, or rather crash the whole index, and similar if my EV installation (i.e. not the index but the program: exe, dll, etc.) was harmed? Obviously, these latter considerations will become highly pertinent whenever my EV 1.4 installation / index will start to lose data / files from the expected search results: I would then notify here that yes, my system is at fault, but in the meanwhile, I'm well-advised to trust 1.4, while suggesting that in 1.5 there might be glitch, currently?
You, on the other hand, might rather wait for other users to relate similar problems, before investigating further? (The specific function I really relied upon in 1.5 was that placeholders * and ? work in there for path sub-strings, which is really brilliant for elaborate searches, but even my very simple searches, just for key words in the file names, showing results I can rely upon, obviously is even more important to me!
Inifile: my (stupid) mistake: I had closed EV, not exited it, hadn't been aware of there being any difference: I had right-clicked on its symbol, then "close window", then my inifile edit seemed to be successful (was saved with the new setting), but upon your hint, I checked the inifile again now, and my edit wasn't there. Now, following your instructions (in EV: File - Exit), everything works as expected. Sorry!
Journal size vs. Index sizes: Yesterday, I wrote, and today, I write, having deleted 1.5 (more or less), thus writing "from memory", thus my mix-up: sorry again! Hence, we can say, even 1.5 does NOT have a global max index size setting, overriding the specific (i.e. drive) max index sizes, but see below.
The 1.5 monitor_file_operation setting: Obviously, I can't tell about that setting anymore now, but I'm perfectly certain I never hampered with the 1.5 ini-files for anything else, except that I changed the statusbar setting (which worked fine in 1.5, and works fine now again in 1.4), so the setting you mention should have been at its default.
Obviously, 1.4 does not have such a setting, but just (in my case, all defaults):
filelists=
filelist_monitor_changes=
folders=
folder_monitor_changes=
folder_buffer_size_list=
folder_rescan_if_full_list=
folder_update_types=
folder_update_days=
folder_update_ats=
folder_update_intervals=
folder_update_interval_types=
while the info in your link,
monitor_file_operation
Set to 1 to immediately update FAT, network drive and folder indexes when successfully renaming, moving or deleting a file. (default)
Set to 0 to update through ReadDirectoryChanges or SHChangeNotify events.
is way too technical for me to understand what it really would do in both cases, and anyway, all those settings left to their respective defaults, even their interaction should not cause such problems.
So, unfortunately, there is some real problem here, with 1.5.
I'm certain I neither try to "force rebuild" ("rebuild" always meaning "force rebuild" here now) the wrong drives, neither do I then try to search the wrong drives: I have not only fixed the drive letters, all within the second half of the ABC, so that temporary, Windows-driven drive-letter assignments will never interfere, and whenever I do rebuilds, even though then the rebuild is done for all drives then connected (incl. c:, d:, and so on), I always check the drive letters / names, and thankfully, EV also displays the name, e.g.
T1 (T:)
meaning drive t:, with name T1 (names T2 and T3 being my backup drives for this example, not with fixed drive letters, so when they are connected, Windows assigns them letters like H: or J: or such, within the the first half of the ABC, so I'm really sure which drives are connected / updated / searched at any given moment).
I'm also certain no filters are intervening at any time since I don't use filters, I just use (sometimes elaborate) search string, but when I'm "Desperately Seeking Susan", i.e. some file I couldn't find, I just search "susan" (without the quotes) in the end, not even specifying the drive anymore, so filter, or, more generally, search syntax, problems definitely are not the problem (I use the present tense, but always mean the past, with 1.5, since no problems with 1.4, thankfully).
Also, when I don't find some file which "should be there", I do that rebuild, and at least immediately after the rebuild, the missing files should show up, but they don't, and sometimes, I do even do this with two external drives, not being sure on which one the missing files would have meed placed... and none of the two, with fresh rebuilds, then shows them! (Now, back to 1.4, I have the same problem with not always knowing on which drive I had stored my file, but I search for key words of the file name, and then EV shows me the path, so I know which drive to connect to access it; some people use RAIDS but in them, all drives run all the time which multiplies the wear, compared to drives which you only connect / run one day a week or so.)
Thus, at least immediately after such a rebuild, the files should be retrieved, but definitely, they are not; yesterday, I hadn't remembered yet that very often, I had done such, then systematically unsuccessful, rebuilds (incl. for alternative drives where the missing files in question also could have been, and are in fact, as my no-problem searches with 1.4 now prove), and since we have out-ruled that some "global" max size could interfere with the individual, drive, max sizes, which, at 131,072 kb each, are multiples of the max size provision really needed (which for these drives might be in the range of 20k or 30k kb (we're speaking of thousand kb's here, not of kb's, 131,072 kb is something near / about 130 mb, that's plenty of room for a file names / attributes index), certainly not more),
it unfortunately becomes evident that those rebuilds, for external drives at least (I have not yet encountered such "files missing from the drive" problems with c: or d:), and with 1.5's default settings insofar, do not process all files present correctly (while 1.4, for the same drives, does).
Btw., if I understand the design correctly, the (also drive-specific) "allocation delta" is the size of the ("earlier", "first-in") file / folder index entries which EV discards when, in "normal run", the addition of new entries ("last-in") would otherwise cause an "overflow" of the set index max size for that drive; whilst obviously, it would be / have been preferable that in such cases, EV notified the situation to the user, inviting them to resize the max size setting in case, instead of just deleting thousands of index entries "just like that", behind the scene, without the user even knowing, it should be evident here that this "allocation delta problem", as I might call it, should not cause the bigger problem described here, since within or directly after a fresh rebuild, with rebuild data to be processed just weighting a fraction of the index' current "max size" setting (just some mb vs. about 130 mb), the "allocation delta" function should never be triggered, thus should discard nothing, so it seems (i.e. every forced, or automated, full or partial) data retrieval (= data writing to the index) misses some data.
The missing files do not (sic!) share some specific characteristic, e.g. they don't contain special characters, are "system" or "hidden" or the like, but as said (I think), except for "folder sizes", I had checked all check boxes to include my files' characteristics, etc., into the index, so - just an idea - some of those checks = some retrieval command might cause skipping of entries of the MTF table? But in some aleatoric way then, since, as said, it's not some file attribute or combination of attributes which causes the leaving-out, and my MTF seems to be ok since otherwise I would encounter the same problems with 1.4, logically.
Did I say that for all of my (EV-indexed) drives, I have Include in db yes, enable USN journal yes, monitor changes yes, but Load recent changes from USN journal NO? Since leaving out the latter, without me really understanding "what it does", is recommended by you, on your site.
Obviously, I don't have the 1.5 settings at hand anymore, but I'm certain I hadn't fiddled around with some, by inifile edits or otherwise, except for the usual Search settings in the Options, e.g. Fast ASCII yes, Match path yes, Match whole filename, etc., those usual, "benign" settings which shouldn't cause any specific problems - and a reset the normal-font setting to "default but 12 point" indeed.
(Btw, it just occurs to me that I have never tried to then search those files, missing from the "normal" search results (i.e. in the EV window), by es.exe, had used es.exe only to create export lists for Android...)
On the other hand, other users should have encountered similar problems with 1.5? I think possible fails in my ssd (my EV index is on c:, obviously, the external drives being hdd's) could not be the culprit here since the ramifications of physical damages would not be limited to specific files, just discarding them from the result list, but would show "all over the place" of such a result list, or rather crash the whole index, and similar if my EV installation (i.e. not the index but the program: exe, dll, etc.) was harmed? Obviously, these latter considerations will become highly pertinent whenever my EV 1.4 installation / index will start to lose data / files from the expected search results: I would then notify here that yes, my system is at fault, but in the meanwhile, I'm well-advised to trust 1.4, while suggesting that in 1.5 there might be glitch, currently?
You, on the other hand, might rather wait for other users to relate similar problems, before investigating further? (The specific function I really relied upon in 1.5 was that placeholders * and ? work in there for path sub-strings, which is really brilliant for elaborate searches, but even my very simple searches, just for key words in the file names, showing results I can rely upon, obviously is even more important to me!
Re: 1.5 alpha sometimes loses indexes of offline drives
Everything might be rescanning the wrong drive (not you)I'm certain I neither try to "force rebuild" ("rebuild" always meaning "force rebuild" here now) the wrong drives
You said Everything found some files, but others were missing.
I wonder if Everything had scanned the wrong drive..
There's a few issues with the current version (1390) when there are volumes with the same drive letters in your index.
Allocation delta is for the USN Journal.Btw., if I understand the design correctly, the (also drive-specific) "allocation delta" is the size of the ("earlier", "first-in") file / folder index entries which EV discards
The USN Journal is a log of changes to your file system.
The USN Journal is maintained by the NTFS driver (not Everything)
Everything uses the Master File Table (MFT) to index files/folders.
Everything uses the USN Journal to keep your index up to date.
No, enabling the index basic file property options (Tools -> Options -> Indexes) will not cause Everything to skip files.some of those checks = some retrieval command might cause skipping of entries of the MTF table?
All basic file property information is gathered from the MFT.
Please leave it disabled.Load recent changes from USN journal NO?
It's only really useful if you use recent changes and wish to view the USN Journal in Everything.
es.exe calls the Everything GUI, so it would have had the same issue.
I would love for you to try Everything 1.5 alpha again, see if we can find the issue..
Everything 1.4 and the Everything 1.5 alpha can run at the same time.
Re: 1.5 alpha sometimes loses indexes of offline drives
Thank you very much!
As for rebuilding wrong indexes, it's true that in the past, I had some "doubles", re the drive-letter only (but not for derive-names), and, as (already partly) said, both 1.4 and 1.5 sometimes refuse to "Remove" unwanted drive entries from the NTFS list (button is greyed-out); in he last days with 1.5 though, I had been able to purge all unwanted entries, and then rebuilds did not include some files anyway; in EV 1.5 (I don't find it in 1.4 now) I had found some function which displayed (or even allowed to edit) the "name" (I don't remember what it was called) of the drive in question, and that dialogue then showed cryptic, very long "names" (which I refrained from trying to edit, obviously), and which for the regular user therefore don't seem to be of much use (but e.g. the "T1 (T:)" for the "T1" drive, as said, already is displayed in the list, so obviously, 1.5 tries to identify each drive unequivoqually (whilst in 1.4, you display the name but don't amend the code, understandably), so EV updating the wrong index should soon be impossible; btw, I clearly remember that in one case, that could have been done indeed, with a re-assigned letter i: (different names though, and where the ("offline", obviously) "old" i: could not be "removed" then), but also, in other cases, some files were left out, even after rebuild, for drive letters (t:, u:, etc) for where there had not been any other physical drive for many months (and then, I had used 1.4, not 1.5 already, so in these cases such interference was even technically impossible).
As for the "Allocation delta", without really grasping all that, so it's not that allocation delta which is cut off from the drive's index, when the drive's max size "overflows", but obviously, EV cuts off at least some data from the index when the respective drive's index' "Maximum size" is reached, without notifying the user, and that seems to be an inherent problem, all the more so since there is no indicator (e.g. the % when it's over 80 %, as an invitation to the user to resize that max); perhaps a max size setting of 0 could be used for "no max", since EV purges old entries, not only in forced rebuilds (forced by user), but also in its regular run, here and there, so at the end of the day, that max size is just as threat to the user who has to hope that it will never be reached, unbeknownst to them - but as said before, a max size of 131,072 kb, for a 4- or 5-digit number of files, even with lots of indexed meta data, should be sufficient (albeit the same size might become insufficient for a c: or d: drive index, for more than 1 million of files?!).
If it's a "full-metadata" index, and with file names up to 255 characters, does the index probably need up to 400 bytes per file and folder? Thus, a number of 100,000 files-and-folders (on some drive) would need up to 400,000,000 bytes (400 million) = some 400,000 kb = 400 mb (not counting some minus 10 p.c. (in reality) for the 1000-vs.-1024 problem (since in reality it's much more than just about 2.5 % since e.g. 6 tb HDDs only come with about 5.600 mb available, which makes almost 10 %, so 10 % seems to be a reasonable margin here)). Thus, 1 million files and folders (on d: in my case) (I never use Options - Indexes - Compress db", so it's un-checked) would need an index size of 4 million kb - compared to the recommended max size of 131,072 kb, this arises the question if / how my calculation is totally wrong?
I, personally, can't rely upon current 1.5, must rely on 1.4 (that "gaslighting" has just had an effect too awful on me!), but I think having read in your site / forum that it's possible to run both versions side-by-side, without them to interfere (?) with each other? How would I do that in practice, what would I have to observe in order for this to process smoothly? If that's really possible though, I could use 1.4, but would have run 1.5 in the background, and would also "mirror" any rebuilds I do in 1.4, in 1.5 - but significant results (if the 1.5 problem prevails and (!) is caused by EV and not by other factors in my system) would probably only occur after some weeks, after some new file moves, additions, etc., on these drives, especially since my 1.4 installation is (and my 1.5 will be) brand-new, and without drive-letter re-assignments... but, after some real "action", after some weeks, the file counts should be identical indeed, and if they are not, I could compare export-lists (I have been using Beyond Compare for years and also own their newest "Pro" version, so compares, when needed, are not the problem; unfortunately, I had de-installed 1.5 without first creating lists; I regret this now that it's too late). Before re-installing 1.5 (which I'll do), and making it index my (internal, and external) drives: What index sizes would really be sufficient? As said, with all meta data EV can handle, 400 bytes per file / folder seems more or less necessary when I want EV never cut off data from the index except when the files / folders have been deleted before? (And those sizes should be identical in 1.4 and 1.5?)
(Up to 255 characters, plus all the meta data EV can handle probably makes up to 400 bytes per file / folder, on the condition that every character just needs 1 byte = 8 bits, so UTF-8, every character there being 2 bytes, not 1, we're rather at 600 bytes per file? So with "Compress db: NO", I might have been overly optimistic, with my systematic "131,072" settings? No data loss I'd be aware of so far in 1.4 though...)
EDIT
I just think of it: Before deleting 1.5, I had (as said) updated to (then-) latest 1.5, but this made me lose some more (or all of my external) drive indexes, so it would be preferable to be able to update 1.5 while preserving the indexes, if necessary (really) exiting 1.5, then renaming / moving the (compound) index, installing the 1.5 update, (really) exit 1.5 again, in case deleting the index (of the internal drives) it will have automatically created in its some-minutes run, replacing (or overwriting) it with the previously saved one, start 1.5 again, trigger "Force Rebuild" (which would update the indexes of internal drives c: and d: so there are no inconsistencies for these either), thus preserving the parts of the index mirroring the external drives (which are not connected during this procedure). If that's possible, (even multiple) updates of 1.5 would be realistic, i.e. without them causing lots of fuss, but all to the contrary, simulating a "normal use case" where the external drives are only connected / re-indexed in case, in case of file moves / additions / deletions on them.
As for rebuilding wrong indexes, it's true that in the past, I had some "doubles", re the drive-letter only (but not for derive-names), and, as (already partly) said, both 1.4 and 1.5 sometimes refuse to "Remove" unwanted drive entries from the NTFS list (button is greyed-out); in he last days with 1.5 though, I had been able to purge all unwanted entries, and then rebuilds did not include some files anyway; in EV 1.5 (I don't find it in 1.4 now) I had found some function which displayed (or even allowed to edit) the "name" (I don't remember what it was called) of the drive in question, and that dialogue then showed cryptic, very long "names" (which I refrained from trying to edit, obviously), and which for the regular user therefore don't seem to be of much use (but e.g. the "T1 (T:)" for the "T1" drive, as said, already is displayed in the list, so obviously, 1.5 tries to identify each drive unequivoqually (whilst in 1.4, you display the name but don't amend the code, understandably), so EV updating the wrong index should soon be impossible; btw, I clearly remember that in one case, that could have been done indeed, with a re-assigned letter i: (different names though, and where the ("offline", obviously) "old" i: could not be "removed" then), but also, in other cases, some files were left out, even after rebuild, for drive letters (t:, u:, etc) for where there had not been any other physical drive for many months (and then, I had used 1.4, not 1.5 already, so in these cases such interference was even technically impossible).
As for the "Allocation delta", without really grasping all that, so it's not that allocation delta which is cut off from the drive's index, when the drive's max size "overflows", but obviously, EV cuts off at least some data from the index when the respective drive's index' "Maximum size" is reached, without notifying the user, and that seems to be an inherent problem, all the more so since there is no indicator (e.g. the % when it's over 80 %, as an invitation to the user to resize that max); perhaps a max size setting of 0 could be used for "no max", since EV purges old entries, not only in forced rebuilds (forced by user), but also in its regular run, here and there, so at the end of the day, that max size is just as threat to the user who has to hope that it will never be reached, unbeknownst to them - but as said before, a max size of 131,072 kb, for a 4- or 5-digit number of files, even with lots of indexed meta data, should be sufficient (albeit the same size might become insufficient for a c: or d: drive index, for more than 1 million of files?!).
If it's a "full-metadata" index, and with file names up to 255 characters, does the index probably need up to 400 bytes per file and folder? Thus, a number of 100,000 files-and-folders (on some drive) would need up to 400,000,000 bytes (400 million) = some 400,000 kb = 400 mb (not counting some minus 10 p.c. (in reality) for the 1000-vs.-1024 problem (since in reality it's much more than just about 2.5 % since e.g. 6 tb HDDs only come with about 5.600 mb available, which makes almost 10 %, so 10 % seems to be a reasonable margin here)). Thus, 1 million files and folders (on d: in my case) (I never use Options - Indexes - Compress db", so it's un-checked) would need an index size of 4 million kb - compared to the recommended max size of 131,072 kb, this arises the question if / how my calculation is totally wrong?
I, personally, can't rely upon current 1.5, must rely on 1.4 (that "gaslighting" has just had an effect too awful on me!), but I think having read in your site / forum that it's possible to run both versions side-by-side, without them to interfere (?) with each other? How would I do that in practice, what would I have to observe in order for this to process smoothly? If that's really possible though, I could use 1.4, but would have run 1.5 in the background, and would also "mirror" any rebuilds I do in 1.4, in 1.5 - but significant results (if the 1.5 problem prevails and (!) is caused by EV and not by other factors in my system) would probably only occur after some weeks, after some new file moves, additions, etc., on these drives, especially since my 1.4 installation is (and my 1.5 will be) brand-new, and without drive-letter re-assignments... but, after some real "action", after some weeks, the file counts should be identical indeed, and if they are not, I could compare export-lists (I have been using Beyond Compare for years and also own their newest "Pro" version, so compares, when needed, are not the problem; unfortunately, I had de-installed 1.5 without first creating lists; I regret this now that it's too late). Before re-installing 1.5 (which I'll do), and making it index my (internal, and external) drives: What index sizes would really be sufficient? As said, with all meta data EV can handle, 400 bytes per file / folder seems more or less necessary when I want EV never cut off data from the index except when the files / folders have been deleted before? (And those sizes should be identical in 1.4 and 1.5?)
(Up to 255 characters, plus all the meta data EV can handle probably makes up to 400 bytes per file / folder, on the condition that every character just needs 1 byte = 8 bits, so UTF-8, every character there being 2 bytes, not 1, we're rather at 600 bytes per file? So with "Compress db: NO", I might have been overly optimistic, with my systematic "131,072" settings? No data loss I'd be aware of so far in 1.4 though...)
EDIT
I just think of it: Before deleting 1.5, I had (as said) updated to (then-) latest 1.5, but this made me lose some more (or all of my external) drive indexes, so it would be preferable to be able to update 1.5 while preserving the indexes, if necessary (really) exiting 1.5, then renaming / moving the (compound) index, installing the 1.5 update, (really) exit 1.5 again, in case deleting the index (of the internal drives) it will have automatically created in its some-minutes run, replacing (or overwriting) it with the previously saved one, start 1.5 again, trigger "Force Rebuild" (which would update the indexes of internal drives c: and d: so there are no inconsistencies for these either), thus preserving the parts of the index mirroring the external drives (which are not connected during this procedure). If that's possible, (even multiple) updates of 1.5 would be realistic, i.e. without them causing lots of fuss, but all to the contrary, simulating a "normal use case" where the external drives are only connected / re-indexed in case, in case of file moves / additions / deletions on them.
Re: 1.5 alpha sometimes loses indexes of offline drives
Allocation delta is for the USN Journal (not the MFT or the index)As for the "Allocation delta", without really grasping all that, so it's not that allocation delta which is cut off from the drive's index, when the drive's max size "overflows", but obviously, EV cuts off at least some data from the index when the respective drive's index' "Maximum size" is reached, without notifying the user, and that seems to be an inherent problem, all the more so since there is no indicator (e.g. the % when it's over 80 %, as an invitation to the user to resize that max); perhaps a max size setting of 0 could be used for "no max", since EV purges old entries, not only in forced rebuilds (forced by user), but also in its regular run, here and there, so at the end of the day, that max size is just as threat to the user who has to hope that it will never be reached, unbeknownst to them - but as said before, a max size of 131,072 kb, for a 4- or 5-digit number of files, even with lots of indexed meta data, should be sufficient (albeit the same size might become insufficient for a c: or d: drive index, for more than 1 million of files?!).
When the USN Journal is full (Maximum size is reached), allocation delta is trimmed from the start of the USN Journal.
The NTFS driver maintains the USN Journal (not Everything)
The USN Journal is not required for indexing your volumes.
The number of files doesn't matter.
The USN Journal is a log of changes.
131072 KB will store a few million file changes.
Everything uses the USN Journal to monitor changes to your index.
Everything will keep track of the current USN Journal Entry.
If you don't run Everything for a week, the next USN Journal entry could be deleted (USN Journal trimmed)
If a USN Journal Entry is missing, Everything will automatically reindex.
You are confusing the USN Journal with the MFT.If it's a "full-metadata" index, and with file names up to 255 characters, does the index probably need up to 400 bytes per file and folder?
The USN Journal is not an index, it's a log of changes.
A USN Journal entry is dynamic in size.
A filename with 255 characters will consume 574 bytes.
There's no option to set a maximum index size.What index sizes would really be sufficient?
There is no index size limit.
I recommend a USN Journal size of 131072 KB on Windows 10/11.
This is very roughly a week of changes.