too many remote folders - all night scan still not done - quality of life help needed

[FrozenGamer]

Big fan of the program, but i have 2 servers i scan folders on.. If there is any hang up, it will scan all drives which in this case i went to bed 10 hours ago and its not done with the first server yet. lots of questions. very grateful for the program and have contributed a few times.

1. In the case of a folder, not ntfs, can i have it remain even if offline (i would like to leave one server off and only turn it on when i want to get a file)
2. How to stop these giant timely rescans
3. Any options other than folder scans for unraid
4.. Pretty sure i need to turn off attempt to monitor for changes on all of the folders.
5. Switch the the Alpha? - it looks like a lot of features including mapped network drives vs folders.
6. settings for quality of life that i may not have asked about?

[void]

Thank you for your feedback FrozenGamer,

What version of Everything are you using?

1. In the case of a folder, not ntfs, can i have it remain even if offline (i would like to leave one server off and only turn it on when i want to get a file)

Everything will keep your folder indexes.
Even when the source is offline.

2. How to stop these giant timely rescans

Everything uses the FindFirstFile API to index and rescan your server.
Check for any bottlenecks on the server side.
Check for any network issues, try to use 10g if possible.
Windows Server should index 1million files per minute.

Everything 1.5 has an option to improve indexing time and rescanning time:
Add multiple folder under Tools -> Options -> Folders
For example: \\server\share\folder1, \\server\share\folder2, \\server\share\folder3, etc...
Right click each folder and under Advanced -> Threads -> Check Multiple threads.
Each folder will then use a separate thread to index.

Everything 1.5 has an option to improve rescanning time:
For each folder index under Tools -> Options -> Folders, check fast rescan.
Note: this will only work if the network share is an NTFS compatible volume.
When enabled, Everything will only rescan folders that have been modified.

3. Any options other than folder scans for unraid

Maybe hosting an Everything Server closer to your hardware might help.
Currently, the Everything Server only supports Windows.

4.. Pretty sure i need to turn off attempt to monitor for changes on all of the folders.

Attempt to monitor changes can be left on.
Everything monitors changes with ReadDirectoryChanges.

5. Switch the the Alpha? - it looks like a lot of features including mapped network drives vs folders.

Yes, switch to the alpha.
Network drives and folders are the same thing internally.
It's just a UI feature to make it easier to include a network drive.

6. settings for quality of life that i may not have asked about?

There's a few settings for Everything 1.5:
read_directory_changes
sh_change_notify
folder_update_rescan_asap

These can be changed under Everything 1.5 -> Tools -> Options -> Advanced

[heyheyhey]

Hi FrozenGamer,

2. How to stop these giant timely rescans

It may help to have multiple separate folders for Everything to scan.

Example of only 1 folder to scan: Everything will scan Parent Folder as one location.

1. Parent Folder
   • Archive
   • Client Uploads
   • Invoices

Example of 3 folders to scan: If interrupted or a rescan is needed, Everything will only need to rescan the folder affected.

1. Archive Folder (initially many files to scan, rarely changed, possibly a candidate for fast rescans)
2. Client Uploads Folder (files come in often enough that a full rescan instead of fast rescan may be needed)
3. Invoices Folder

You've mentioned that your folder is not NTFS, but the filesystem itself does not strictly need to be NTFS, so it would be worth testing if fast rescan is available to you.

Although it was a while ago and an older Everything build, I recall that pausing or interrupting the initial scan on a large drive/share/location would cause the scan to take much longer; the scan seemed to restart from the beginning instead of where it left off. So I'd let the initial scan complete before turning off the server.

6. settings for quality of life that i may not have asked about?

Here are a couple commands I liked to use overnight for speed, though I've never benchmarked them.
content_multithreaded_max_memory_percent - Set to the maximum memory as a percentage to use when reading content.
memcpy_max_threads - The maximum number of threads to use when copying memory. Set to 0 to use the maximum available.

This is off by default, perhaps void can chime in on its utility?
db_save_on_rebuild - Set to 1 to save the database to disk after a rebuild.


void's helped me with slow scanning over remote in the past. Of their recommendations, adding multiple folders instead of the whole drive while enabling multiple threading for them helped the most. I was able to get scanning time down a considerable degree but it turned out the other bottlenecks was aging local office networking and a misconfigured DNS on the cloud server. Something to consider if all else fails.

[void]

This is off by default, perhaps void can chime in on its utility?
db_save_on_rebuild - Set to 1 to save the database to disk after a rebuild.

This setting, when enabled, saves your index to disk after the initial build.
Everything should only index once.
Everything 1.5 tries to avoid writing to disk.
It only really exists to replicate the same behavior in Everything 1.4.