[BUGS] 2 strange massive bugs of Everything indexing...!

[nikov]

Hi,
I think these 2 bugs can ruin Everything experience:

The following video is demonstrating these 2 bugs:
Bug #1: Every indexing processing is not silent/background (Everything does not allow any search if an indexing processing is ongoing).
Bug #2: Though, I only changed the H drive's mode from read-only to writable, Everything starts indexing H, U, and W drive!

video.gif (489.9 KiB) Viewed 2406 times

Explanation:

The BUG #1 (Every indexing processing is not silent/background):
When Everything starts processing index (by building, scanning, sorting, etc.), it does not allow the user to search for files from the old database. For search to work, users have to wait until the index is built + sorting is finished + every other processing is finished.

I am aware that I may not get my file before the index is fully processed; I am ready to wait only if my file is not found. But if my file can be found in the old database, why should I wait?
I don't mind if Everything takes hours or even days to process a very big index; this bug is not about any performance improvement, it is about an option that will let the user to search for the existing files while it is building + sorting + processing the index (maybe based on the old database).

Shouldn't every indexing related processing be silent/background, even it is fully rebuilding the index?


The BUG #2 (Unnecessary re-indexing):
Suppose, I have 3 read-only NTFS drives with drive letters "H", "U" and "W". (H, U and W drive contains 320160, 320160 and 2101050 objects respectively.)
Some time later, I just only changed the H drive's mode from read-only to writable but Everything starts re-indexing all read-only drives (H, U and W drive)!
(This is strange; a single drive is affecting all drives! This unnecessary re-indexing can take a lot of time.)

Best expected behavior:
As I have only changed the mode from read-only to writable, Everything should not start any re-indexing at all.

Decent expected behavior:
Everything should re-index only that particular drive whose mode was changed from read-only to writable (here, only the H drive).
(Manual cancellation of ongoing indexing can also be considered.)

Worst expected behavior (the current behavior):
Everything starts re-indexing all read-only drives just for changing only a single drive's (here, only the H drive's) mode from read-only to writable.
And there is no way to cancel or ignore this re-indexing.

Note: This does not happen if "Monitor changes" is disabled on the mode changing drive (here, H drive). But the issue is, why it starts re-indexing even when no changes were done in the H drive except writability?
Moreover, as this problem does not occur when mode is changed from writable to read-only and writable to offline to writable, so it is expected read-only to writable should also be OK.
Also, disabling "Monitor changes" prevents automatic index update, thus, it is not a good way either.

Also note that, these problems happen for both removable and fixed/non-removable NTFS disks.


How to reproduce these bugs:
Well, reproducing these bugs are easy:

• Install ImDisk
• Right click any useless file
• Create and mount that file as a disk
• Format that disk to NTFS
• Put 2 million+ objects on that disk (you can generate these objects programmatically (i.e. using Python, you may try this Python script:

  objgnrtr.zip

  (654 Bytes) Downloaded 68 times

  ))
• As usual, dismount/remount such drives as read-only/writable/offline/removable/fixed media by right clicking; to reproduce these bugs.

Many thanks for your time and consideration.

[void]

Thank you for your feedback nikov,

The cause of the reindex is Everything automatically adding or removing an NTFS volume.
The reason for the reindex is to improve search and sort performance.

You cannot search the existing index while Everything is performing a reindex.

An option to avoid the reindex process is on my TODO list.
However, don't expect good performance when adding or removing a volume.

I did experiment with adding volumes without a rebuild for Everything 1.5. (temporary indexes)
Unfortunately, it needs a major rewrite.

Closely related, there is an ini setting to index folders in the background:
folder_background_index
However, this still requires a quick rebuild when adding/removing the folder index.



This reindex you are seeing is a "quick" reindex.
Everything will not go to disk to rescan your existing indexes.
The existing indexes are rescanned from memory.
The reindex should only take a few seconds when adding or removing a volume from your index.

Note: This does not happen if "Monitor changes" is disabled on the mode changing drive (here, H drive). But the issue is, why it starts re-indexing even when no changes were done in the H drive except writability?

Windows dismounts the drive, then remounts the drive.
Everything will see this volume as being removed, then re-added.



I recommend disabling "Automatically include new volumes" and disabling "Automatically remove offline volumes":

• In Everything, from the Tools menu, click Options.
• Click the FAT tab.
• Uncheck Automatically include new fixed volumes.
• Uncheck Automatically include new removable volumes.
• Uncheck Automatically remove offline volumes.
• Click the NTFS tab.
• Uncheck Automatically include new fixed volumes.
• Uncheck Automatically include new removable volumes.
• Uncheck Automatically remove offline volumes.
• Click the ReFS tab.
• Uncheck Automatically include new fixed volumes.
• Uncheck Automatically include new removable volumes.
• Uncheck Automatically remove offline volumes.
• Click the Network Drives tab.
• Uncheck Automatically include new mapped network drives.
• Uncheck Automatically remove offline mapped network drives.
• Click OK.

Everything should no longer reindex after a volume is remounted.

Unfortunately, you'll now need to manually include any new volumes.
Consider leaving "include new fixed volumes" checked for NTFS volumes.

[ChrisGreaves]

I did experiment with adding volumes without a rebuild for Everything 1.5. (temporary indexes)
Unfortunately, it needs a major rewrite before a release.

@Void: Does "rewrite" refer to 1.6?
Thanks, Chris

[void]

Yes, temporary indexes will not be ready for the 1.5 release.
It needs to be redesigned and rewritten for a future release.

[ChrisGreaves]

Yes, temporary indexes will not be ready for the 1.5 release.
It needs to be redesigned and rewritten for a future release.

Thanks Void.

[nikov]

Thank you void,

You cannot search the existing index while Everything is performing a reindex. [...]

It is not always about the existing index.
Why not use the old database while Everything is building the index?
Old database is already there and it is completely functional, why not utilize the old database before building a new one?
Let the user use the old database and meanwhile let the Everything process the index.
Then users will not have to wait for minutes.
It sounds and seems very simple; I don't understand why it needs a major rewrite before a release.

[...] The reindex should only take a few seconds when adding or removing a volume from your index.

You can see from the video that it is not taking a few seconds; instead it is taking almost a minute just to index about 2.5 million objects. And on top of it, it is mostly an unnecessary indexing as only the writability mode was changed (no changes were done to objects)!

Windows dismounts the drive, then remounts the drive.
Everything will see this volume as being removed, then re-added. [...]

Note: As mentioned earlier, this problem does not occur when mode is changed from [writable to offline to read-only] and [writable to offline to writable], so it is expected and it should work for [read-only to offline to writable] as well; the same way. But it is not working.

I recommend disabling "Automatically include new volumes" and disabling "Automatically remove offline volumes": [...]

I tried all of your instructions but it behaves exactly the same way as before.
I also tried folder_background_index but still it behaves the same.

Thanks.

[void]

Everything should only index once.

What is shown under Tools -> Debug -> Statistics -> Build -> Last rebuild reason?

[nikov]

Everything should only index once.

What is shown under Tools -> Debug -> Statistics -> Build -> Last rebuild reason?

"Out of date NTFS volume H: is available"

[ChrisGreaves]

... as only the writability mode was changed (no changes were done to objects)

Hi nikov.
On a small point of logic: YOU know that only the writability mode has changed (presumably because you changed it), but how is Everything expected to know that?
This logic question could be asked of anyone in a similar situation using computers.
I might swear that only such-and-such was changed on my removable backup drive, but how is Everything expected to know that?

I pose the question in a logical sense only; I'm not suggesting you have done anything wrong, and indeed your suggestions for changes to Everything is probably quite valid.

But a computer program often has to satisfy itself, by inspection, to discover that only one small thing (out of millions) is changed.
In the Good Old Days, adding one employee to the 250,000 already hired by the firm could require half an hour while a reel of magnetic tape was read and copied to a second tape, all for the purpose of inserting a single record in the correct sequence. The managers would cry out "But we only added one employee this week!"

With best wishes
Chris

[nikov]

... but how is Everything expected to know that?

Hi ChrisGreaves,
Nice question, I also thought about this.
Then I thought 3 ways Everything is expected to know that:

Way 1:
Everything is already OK in all other circumstances:
i.e. this problem does not happen when mode is changed from [writable to offline to read-only], [read-only to offline to read-only], [writable to offline to writable].
So, it is expected this problem should not happen for [read-only to offline to writable] too.

Way 2:
Look, Everything is already running in the background, and an operating system may have many event-driven triggers. And Everything maybe listening/waiting to these event-driven triggers. That's how it can take necessary steps and can decide not to re-index.

Way 3:
Even if Everything is not running, Everything can still perform complex time saving task, it can locate newly created files fast. I guess, the author(s) of Everything is very expert in these things, Everything does some magical stuff that I don't know.
So, the 3rd point is basically believing based on Everything's reputation.

Thanks, with best wishes too.

[ChrisGreaves]

Then I thought 3 ways Everything is expected to know that:

Good thinking. I am outclassed!
Cheers, Chris

[NotNull]

When I used imdisk in the past -- before switching to Dataram RAMdisk -- I noticed that imdisk is not fully compatible with NTFS.
And that caused issues in Everything.

@void was able to provide a workaround for that, but the message here is that imdisk "does not follow the rules"

(also in linked thread: re-scanning other drives)

[therube]

(I'll just note that I happened to run into this this morning.
I opened an instance I hadn't opened in some time, & instead of "just showing results", it was re-indexing.
So I had to wait. For me, it wasn't really a big deal. Nonetheless, it just happened.)

[NotNull]

Your old database was probably in pre-1344a state then?
(database structure changed with 1344 and that requires a rebuild)

[therube]

(Yes, I know that. What I don't know, or particularly pay attention to, is what version of Everything I may have in a particular instance.

As in, I might copy the current version into various instances, & not run them, for a time, or I might not be able to update a particular instance, because I'm not at a point where I can Quit... So very possible I went from a pre- to a later, & just hadn't happened to run into the re-index scenario - yet.)

[nikov]

Hi NotNull,
Thanks, I have read the thread, that thread's issue basically arrived for not running Everything as an administrator.
But I am running Everything as an administrator. Moreover, after void's workaround, most probably they will be treated as normal NTFS disks as well.
And also, can you be sure that this problem will only be specific to ImDisk and not for others? Maybe this will happen for all NTFS disks as well.

Thanks.

[NotNull]

Thanks, I have read the thread, that thread's issue basically arrived for not running Everything as an administrator.

Then you missed the point: win32 file namespace vs win32 device namespace.

And I am not concluding anything; just pointing out things.

[nikov]

I tried to mean maybe not running as an administrator was the cause for that issue, I am not sure either.

[void]

Everything will flag a volume as out-of-date when any of the following of the errors are encountered:

• ERROR_JOURNAL_ENTRY_DELETED (1181): too many changes occurred while Everything wasn't running.
• ERROR_JOURNAL_NOT_ACTIVE (1179): USN Journal was deleted.
• ERROR_JOURNAL_DELETE_IN_PROGRESS (1178): USN Journal is currently being deleted.

Everything will also flag a volume as out-of-date when a new USN Journal ID is detected.
The USN Journal ID will change when the USN Journal is deleted and recreated.

An out-of-date volume means there is not enough information in the USN Journal to update your Everything database.
Everything must rescan the entire volume when out-of-date.

[nikov]

So, maybe when a disk is changed to read-only; its USN journal is changed, and thus no guidance from the operating system, something like that?
But what about, Everything can run in the background and keep record of read-only to writable mode change to prevent rescanning/rebuilding the index?
I don't mind if Everything is always running in the background.
Also, maybe another way to mitigate this, is that, let the user use the old database while Everything is rescanning/processing the index.
Otherwise users may have to wait for a long time.

Thank you void for your consideration.