Disable re-scan when changing certain settings

Discussion related to "Everything" 1.5 Alpha.
Post Reply
David.P
Posts: 75
Joined: Fri May 29, 2020 3:22 pm

Disable re-scan when changing certain settings

Post by David.P » Wed May 11, 2022 8:51 am

Hello forum,

I am scanning various network folders, some of which are located (via VPN) on other continents...

It seems that Everything re-scans everything every time a file exclusion is added, for example. This makes sense, and is probably needed to apply the respective change in the settings.

However, this can easily take a couple of hours in my case.

Is there any way to disable the instant re-scan, or postpone it to overnight?

Also, what would be the best settings (in particular for "Attempt to monitor changes" and "Fast rescan") for such a slow network drive, that is being scanned nightly, or even only weekly?
Image
Last edited by David.P on Thu May 12, 2022 4:04 pm, edited 1 time in total.

void
David Carpenter (Developer)
Posts: 9367
Joined: Fri Oct 16, 2009 11:31 pm

Re: Disable re-scan when changing certain settings

Post by void » Wed May 11, 2022 9:24 am

It seems that Everything re-scans everything every time a file exclusion is added, for example. This makes sense, and is probably needed to apply the respective change in the settings.
Everything should only perform a rescan when you remove an exclude.
Adding an exclude should perform a quick reindex without the rescan.


Is there any way to disable the instant re-scan, or postpone it to overnight?
No, Everything must rescan the entire network share.


Also, what would be the best settings (in particular for "Attempt to monitor changes" and "Fast rescan") for such a slow network drive, that is being scanned nightly, or even only weekly?
Only use Fast Rescan if the remote drive is NTFS.
Leave Attempt to monitor changes enabled.



I recommend removing all your excludes, indexing all files and omitting any unwanted results:
  • In Everything, from the Tools menu, click Options.
  • Click the Exclude tab on the left.
  • Uncheck Exclude hidden files and folders.
  • Uncheck Exclude system files and folders.
  • Uncheck Enable exclude list.
  • Click OK.
  • From the Index menu, check Enable Omit Results.
  • From the Index menu, check Organize Result Omissions.
  • Add the files and folders you wish to exclude from your results to this list.
  • Click OK.
Toggling Omit Results from the Index menu is very fast and does not require a rescan.



If possible, consider running an Everything Server remotely and including the remote index in your local index.

David.P
Posts: 75
Joined: Fri May 29, 2020 3:22 pm

Re: Disable re-scan when changing certain settings

Post by David.P » Thu May 12, 2022 4:10 pm

void wrote:
Wed May 11, 2022 9:24 am
Everything should only perform a rescan when you remove an exclude.
Adding an exclude should perform a quick reindex without the rescan.
Thank you, got it.
void wrote:
Wed May 11, 2022 9:24 am
Only use Fast Rescan if the remote drive is NTFS.
Leave Attempt to monitor changes enabled.
Ok. I suppose that it is, but is there any way I can tell whether the remote disk/server actually is NTFS? Or would Everything behave in a certain way if "Fast Rescan" is enabled and the remote disk is not NTFS?
void wrote:
Wed May 11, 2022 9:24 am
I recommend removing all your excludes, indexing all files and omitting any unwanted results.
[...]
Toggling Omit Results from the Index menu is very fast and does not require a rescan.
Thanks, this is of course the best way.

Image
Would there be a way to scan the entire file server directly, for example above the server with the address \\fileserver1?

Just to give an example, this is possible with Archivarius 3000:
Image

void
David Carpenter (Developer)
Posts: 9367
Joined: Fri Oct 16, 2009 11:31 pm

Re: Disable re-scan when changing certain settings

Post by void » Fri May 13, 2022 12:12 am

Ok. I suppose that it is, but is there any way I can tell whether the remote disk/server actually is NTFS? Or would Everything behave in a certain way if "Fast Rescan" is enabled and the remote disk is not NTFS?
Do you have access to \\fileserver\c$ ?
The properties should show the file system.

Otherwise, the file system is unknown.

The file system does not have to be NTFS.
Fast rescan is supported if making a change to a directory causes the parent modified timestamp to update.
FAT, exFAT, FAT32 are not supported.


One way to test if you have write access:
Create a new file in a folder.
Does the folders date modified update?
-If so, use fast rescan.

One way to test if you have read-only access:
Check for a recently created file.
Make sure the date modified for the parent folder is later than or equal to the file creation date.
-If it is, use fast rescan.

If you find Everything missing changes, disable fast rescan.


Would there be a way to scan the entire file server directly, for example above the server with the address \\fileserver1?
To add a root server to your index:
  • In Everything, from the Tools menu, click Options.
  • Click the Folders tab on the left.
  • Click Add Custom....
  • Type in:
    \\fileserver1
  • Click OK.
  • Click OK.

David.P
Posts: 75
Joined: Fri May 29, 2020 3:22 pm

Re: Disable re-scan when changing certain settings

Post by David.P » Fri May 13, 2022 9:39 pm

void wrote:
Fri May 13, 2022 12:12 am
Do you have access to \\fileserver\c$ ?
The properties should show the file system.
Nope, access denied.

void wrote:
Fri May 13, 2022 12:12 am
Fast rescan is supported if making a change to a directory causes the parent modified timestamp to update.
FAT, exFAT, FAT32 is not supported.

One way to test if you have write access:
Create a new file in a folder.
Does the folders date modified update?
That worked! So I will switch on fast rescan.

void wrote:
Fri May 13, 2022 12:12 am
To add a root server to your index:
  • In Everything, from the Tools menu, click Options.
  • Click the Folders tab on the left.
  • Click Add Custom....
  • Type in:
    \\fileserver1
  • Click OK.
  • Click OK.
Awesome! This worked as well. Took an entire day to index that server.

However, then I tried to search for stuff -- first I had set Everything to "Show folders".

I then switched to "Show everything", but that made Everything instantly start "Rescanning: \\fileserver1"

These are the current settings for that server:
Image

Can it be said why Everything would re-start scanning here?

void
David Carpenter (Developer)
Posts: 9367
Joined: Fri Oct 16, 2009 11:31 pm

Re: Disable re-scan when changing certain settings

Post by void » Sat May 14, 2022 12:39 am

Changing the filter should not cause a rescan.

What is the rebuild reason shown under Tools -> Debug -> Statistics -> Build -> Last Rebuild Reason?

David.P
Posts: 75
Joined: Fri May 29, 2020 3:22 pm

Re: Disable re-scan when changing certain settings

Post by David.P » Sat May 14, 2022 10:45 am

Here is the statistics log with the entry "Last rebuild reason: Add folder"

Code: Select all

Build
Count:	19
Total duration:	1:13:29:54
Minimum duration:	0,639486 seconds
Maximum duration:	41.627,028390 seconds
Average duration:	7.104,985268 seconds
Last duration:	41.627,028390 seconds
Last build date:	13.05.2022 22:26
Last rebuild reason:	Add folder.

Folder Index
Path:	\\fileserver1
Drive type:	Remote
Label:	
Index number:	4
Out of date:	No
Online:	
Disk device index:	
Multithreaded:	Separate device thread
Folder count:	122.279
File count:	881.108
Last rescan date:	13.05.2022 22:26
Last rescan successful:	Yes
Last successful rescan date:	13.05.2022 22:26
Last successful rescan duration:	13:04:21.853
Next scheduled rescan date:	15.05.2022 03:00

Folder Index
Path:	C:
Drive type:	Fixed
Label:	
Index number:	5
Out of date:	No
Online:	Yes
Disk device index:	0
Multithreaded:	Separate device thread
Folder count:	183.764
File count:	729.809
Last rescan date:	14.05.2022 11:30
Last rescan successful:	Yes
Last successful rescan date:	14.05.2022 11:30
Last successful rescan duration:	00:01:04.260
Next scheduled rescan date:	15.05.2022 05:00
Thus, a folder had probably been added to the server in the meantime.

And I didn't have "Fast rescan" enabled at that time, so the entire index was rebuilt -- is that assumption correct?
Last edited by void on Sat May 14, 2022 10:49 am, edited 1 time in total.

void
David Carpenter (Developer)
Posts: 9367
Joined: Fri Oct 16, 2009 11:31 pm

Re: Disable re-scan when changing certain settings

Post by void » Sat May 14, 2022 11:06 am

Here is the statistics log with the entry "Last rebuild reason: Add folder"
Thank you for the stats.

This means a folder was added to Tools -> Options -> Folders.
It is possible this rescan was not a rebuild.

Maybe a rescan was accidentally triggered from Index -> Recsan?

Did you recently add the C: drive to Tools -> Options -> Folders?
Your C: drive should be indexed as an NTFS volume under Tools -> Options -> NTFS.

What settings are shown for your C: drive under Tools -> Options -> Folders?
What settings are shown for your C: drive under Tools -> Options -> NTFS?


Thus, a folder had probably been added to the server in the meantime.
Adding a share to \\fileserver will not cause a full rescan.


And I didn't have "Fast rescan" enabled at that time, so the entire index was rebuilt -- that correct?
Changing Fast rescan should not trigger a rebuild.



I recommend removing \\fileserver from Tools -> Options -> Folders and adding each individual \\fileserver\share.
Detecting changes to \\fileserver is limited. (ShellChangeNotify)
Detecting changes to \\fileserver\share is more robust. (ReadDirectoryChanges)

David.P
Posts: 75
Joined: Fri May 29, 2020 3:22 pm

Re: Disable re-scan when changing certain settings

Post by David.P » Sat May 14, 2022 11:21 am

Thanks very much, will do!

Here are the screenshots of the C: drive settings:
void wrote:
Sat May 14, 2022 11:06 am
What settings are shown for your C: drive under Tools -> Options -> Folders?
What settings are shown for your C: drive under Tools -> Options -> NTFS?
Image
Image

void
David Carpenter (Developer)
Posts: 9367
Joined: Fri Oct 16, 2009 11:31 pm

Re: Disable re-scan when changing certain settings

Post by void » Mon May 16, 2022 8:47 am

Thank you for the screenshots.

Your C: drive is indexed as a "folder index" only.
This is fine.

For the best performance, please consider indexing your C: drive as an NTFS volume:
  • In Everything, from the Tools menu, click Options.
  • Click the General tab on the left.
  • Check Everything Service. (Please make sure this is tick-checked and not square-checked)
  • Click the Folders tab on the left.
  • Select C: and click Remove.
  • Click the NTFS tab on the left.
  • Select C: and check Include in database.
  • Click OK.

David.P
Posts: 75
Joined: Fri May 29, 2020 3:22 pm

Re: Disable re-scan when changing certain settings

Post by David.P » Mon May 16, 2022 11:08 am

O.k. thanks very much for the information.

I believe that I can't add my C: disk as NTFS volume, because I don't have administrator rights on this machine.

Please one more question about the best indexing settings for the remote server \\fileserver1 (over the slow VPN connection):
Image

If both "Attempt to monitor changes" and "Fast rescan" are enabled, is a scheduled rescan (currently every night) useful, or even necessary?

void
David Carpenter (Developer)
Posts: 9367
Joined: Fri Oct 16, 2009 11:31 pm

Re: Disable re-scan when changing certain settings

Post by void » Mon May 16, 2022 11:12 am

Are you keeping Everything running 24/7?

If so, you can disable the daily rescan and just rescan once a week/month.
Attempt to monitor changes is quite reliable and will pickup all changes.


Please note:
Indexing your fileserver1 as \\fileserver1 is not as reliable as
indexing your fileserver1 as \\fileserver1\share1 and \\fileserver1\share2 etc..



Everything will miss changes to \\fileserver1 when Everything is not running.
A rescan is necessary to pickup any changes while Everything was not running.

David.P
Posts: 75
Joined: Fri May 29, 2020 3:22 pm

Re: Disable re-scan when changing certain settings

Post by David.P » Mon May 16, 2022 11:21 am

Thanks! Yes, the laptop with Everything runs 24/7.

David.P
Posts: 75
Joined: Fri May 29, 2020 3:22 pm

Re: Disable re-scan when changing certain settings

Post by David.P » Mon May 16, 2022 11:26 am

...another small issue:

It seems that in the Network Drives tab, I can no longer delete the shares that I have added there -- the Remove button is grayed out?
Image

Edit: updated Screenshot
Last edited by David.P on Mon May 16, 2022 11:36 am, edited 1 time in total.

void
David Carpenter (Developer)
Posts: 9367
Joined: Fri Oct 16, 2009 11:31 pm

Re: Disable re-scan when changing certain settings

Post by void » Mon May 16, 2022 11:27 am

The Network Drives list shows all online network drives and offline network drives in your index.

Online drives cannot be removed.

Instead of removing these drives, please exclude them from your index:
  • In Everything, from the Tools menu, click Options.
  • Click the Network Drives tab on the left.
  • For each Network drive:
    • Uncheck Include in database.
  • Click OK.

David.P
Posts: 75
Joined: Fri May 29, 2020 3:22 pm

Re: Disable re-scan when changing certain settings

Post by David.P » Mon May 16, 2022 11:40 am

🤭 I see, thanks!

Post Reply