(1.4/1.5a) Moved folders' contents missing from index, evade 'monitor changes', 'fast rescan', etc.
-
- Posts: 23
- Joined: Tue Apr 28, 2020 12:32 am
(1.4/1.5a) Moved folders' contents missing from index, evade 'monitor changes', 'fast rescan', etc.
Recently did some sorting/cleanup of a large network share and found folder sizes to be misreported as smaller than expected. Digging deeper, I noted that when moving a folder within the share, Everything (both 1.4 and 1.5a) will lose track of the contents of that moved folder. I believe this is due to how a local folder move is handled by the share (a single operation updates the inode of the folder(s), so the contents are not changed). Everything is seeing the old folder disappear (so its children are removed from the index), and the 'new' (moved) folder appears in the new location immediately afterward, but the only 'new' item is the folder (not its contents). End result is that after a folder move, Everything's index is no longer in sync with the contents of the network share. I've confirmed that the buffer is not overfilling (rescan on full buffer has no effect).
The only way to resync is to manually do a full rescan of the share (not a great workaround, especially for larger shares), but when reproducing with 1.5a I found another issue: 'Fast rescan' fails to scan the moved folder contents due to the way that the last modified dates are changed when a folder is moved. The moved folder itself retains its original date, and the folder it is moved into is the one updated to the current date. Everything sees the moved folder as having a date older than the last rescan, so it doesn't scan those branch contents, and so the moved folder contents continue to remain out of sync with the index even after the manual rescan.
Two different fixes are required to address both 'monitor changes' and 'fast rescan' features:
- For 'Attempt to monitor changes', Everything should do a partial branch scan of every newly appearing folder. Especially those appearing a short time after another folder was removed.
- For 'Fast rescan' to catch moved folder contents, the entire branch (all subdirs) of a folder with a newer modified date needs to be scanned. This is because the moved folder itself will retain the old modified date.
Hope this info helps to improve this most excellent search tool!
Allyn Malventano
Storage Technical Analyst, Intel
The only way to resync is to manually do a full rescan of the share (not a great workaround, especially for larger shares), but when reproducing with 1.5a I found another issue: 'Fast rescan' fails to scan the moved folder contents due to the way that the last modified dates are changed when a folder is moved. The moved folder itself retains its original date, and the folder it is moved into is the one updated to the current date. Everything sees the moved folder as having a date older than the last rescan, so it doesn't scan those branch contents, and so the moved folder contents continue to remain out of sync with the index even after the manual rescan.
Two different fixes are required to address both 'monitor changes' and 'fast rescan' features:
- For 'Attempt to monitor changes', Everything should do a partial branch scan of every newly appearing folder. Especially those appearing a short time after another folder was removed.
- For 'Fast rescan' to catch moved folder contents, the entire branch (all subdirs) of a folder with a newer modified date needs to be scanned. This is because the moved folder itself will retain the old modified date.
Hope this info helps to improve this most excellent search tool!
Allyn Malventano
Storage Technical Analyst, Intel
Re: (1.4/1.5a) Moved folders' contents missing from index, evade 'monitor changes', 'fast rescan', etc.
Thank you for the issue report Allyn Malventano,
When you move a network folder, Everything will see the following events:
ADD new-location\folder-name
DEL old-location\folder-name
Everything 1.4 will:
Miss any subfolders and files from the move.
Pickup any missed folders and files at the next scheduled rescan. (rescans occur daily)
Everything 1.5 will:
Add new-location\folder-name to a rescan queue and should pickup the moved folder, subfolder and files quickly.
It's interesting to hear Everything 1.5 is not picking up the moved folder contents for you.
What type of hardware is used to host the network share?
I am looking into an issue with fast rescan missing moved folders.
Thank you for the bug report.
When you move a network folder, Everything will see the following events:
ADD new-location\folder-name
DEL old-location\folder-name
Everything 1.4 will:
Miss any subfolders and files from the move.
Pickup any missed folders and files at the next scheduled rescan. (rescans occur daily)
Everything 1.5 will:
Add new-location\folder-name to a rescan queue and should pickup the moved folder, subfolder and files quickly.
It's interesting to hear Everything 1.5 is not picking up the moved folder contents for you.
What type of hardware is used to host the network share?
I am looking into an issue with fast rescan missing moved folders.
Thank you for the bug report.
-
- Posts: 23
- Joined: Tue Apr 28, 2020 12:32 am
Re: (1.4/1.5a) Moved folders' contents missing from index, evade 'monitor changes', 'fast rescan', etc.
It's a debian linux smbd share. Fairly standard/simple config, but to be safe, I have reproduced on an NTFS share with identical results.What type of hardware is used to host the network share?
Watching 1.4 and 1.5a side by side, 1.5a does appear to be scanning some depth of the moved folder, but not all depth.
Experiment initial state (same on both 1.4 and 1.5a, performed on an NTFS share):
1.4 behavior when moving 'folder' from 'location 1' to 'location 2': (only the moved folder is indexed and no depth is scanned, as expected)
1.5 result on same move operation: (scan depth appears to be stopping prematurely, only catching contents of the first layer subdirs of the moved folder)
Separately (re: 'fast rescan'), I have reproduced on NTFS shares, and in the above example, a fast rescan with 1.5a does not correct the issue (result after the move was still unchanged and the additional layers were not scanned).
All of the above produces an identical result on both NTFS and Linux-SMBD shares.
Is there perhaps some optimization that can be done, where if such a pair is received together, indexed children of the del folder can initially have their paths switched over to the add folder? This would minimize Everything's re-sorting load on larger operations and make the initial index update much faster after a move, and the add folder can still be rescanned to double-check for a coincidental add/del that wasn't actually a move.ADD new-location\folder-name
DEL old-location\folder-name
Re: (1.4/1.5a) Moved folders' contents missing from index, evade 'monitor changes', 'fast rescan', etc.
Thank you for the hardware information malventano,
Thank you for the NTFS test information.
Everything 1.5.0.1312a fixes the issue with only partial folders being added after a move.
Everything was not building the full path correctly when rescanning.
Fast Rescan will also work as expected now.
Please force a rebuild of your network drive to pickup any partially rescanned folders in previous versions:
In Everything, type in the following search and press ENTER:
/reindex T:
where T: is your network drive.
Thank you for the suggestion.
Thank you for the NTFS test information.
Everything 1.5.0.1312a fixes the issue with only partial folders being added after a move.
Everything was not building the full path correctly when rescanning.
Fast Rescan will also work as expected now.
Please force a rebuild of your network drive to pickup any partially rescanned folders in previous versions:
In Everything, type in the following search and press ENTER:
/reindex T:
where T: is your network drive.
I have put this on my TODO list.Is there perhaps some optimization that can be done, where if such a pair is received together, indexed children of the del folder can initially have their paths switched over to the add folder?
Thank you for the suggestion.
-
- Posts: 23
- Joined: Tue Apr 28, 2020 12:32 am
Re: (1.4/1.5a) Moved folders' contents missing from index, evade 'monitor changes', 'fast rescan', etc.
I attempted to confirm this by allowing the index to become out of sync with the test folders and then performing a fast rescan, but I noted an odd configuration limit: With 'Attempt to monitor changes' disabled, shouldn't 'Fast rescan' still be allowed?
Also note there are still rescan issues carried forward into Fast rescan, as indicated below.
Understood. The partial rescan did work correctly with my test folder example from before, but when I retried the original move in the live environment, the behavior remained the same as before (rescan is very fast but only goes two layers deep). I've again duplicated the live environment files on both NTFS and Linux SMBD with the same results. With further investigation, it appears the rescan fails its depth when there is more than one folder in any subdir down the rescan branch. For example, the rescan will only progress one layer past the dir with two folders present, and this issue again will evade a 'Fast rescan':void wrote: ↑Tue May 03, 2022 6:00 am Everything 1.5.0.1312a fixes the issue with only partial folders being added after a move.
Everything was not building the full path correctly when rescanning.
Before: (note that I added another folder 'layer 2(2)' alongside folder 'layer 2')
After 'folder' moved from 'location 1' to 'location 2': (note anything after layer 3 is now gone, and the now mismatched index will also evade a 'Fast rescan')
If you would like additional testing of this optimization once implemented, please ping me back on this thread and I can test performance on a few very large file/folder count shares.
Re: (1.4/1.5a) Moved folders' contents missing from index, evade 'monitor changes', 'fast rescan', etc.
Thank you for the feedback malventano,
I will allow Fast Rescan to be toggled even when 'Attempt to monitor changes' is disabled in the next alpha update.
Everything sees a new folder is created (folder in location 2) from ReadDirectoryChanges.
FindFirstFile on "location 2\folder" succeeds
FindFirstFile on "location 2\folder\*" fails (either 2: path not found or 123: invalid filename)
If I wait for 5 seconds, FindFirstFile will succeed on "location 2\folder\*"
Setting HKLM\System\CurrentControlSet\Services\LanmanWorkstation\Parameters\FileNotFoundCacheLifetime to 0 fixes the issue.
Create a new DWORD value if FileNotFoundCacheLifetime does not exist.
FileNotFoundCacheLifetime
In the next alpha update Everything will wait 30 seconds until FindFirstFile on "location 2\folder\*" succeeds (if FindFirstFile on "location 2\folder" succeeds)
Yes, this is an oversight.With 'Attempt to monitor changes' disabled, shouldn't 'Fast rescan' still be allowed?
I will allow Fast Rescan to be toggled even when 'Attempt to monitor changes' is disabled in the next alpha update.
From what I am seeing, this appears to be an issue with SMB cache.(note anything after layer 3 is now gone, and the now mismatched index will also evade a 'Fast rescan')
Everything sees a new folder is created (folder in location 2) from ReadDirectoryChanges.
FindFirstFile on "location 2\folder" succeeds
FindFirstFile on "location 2\folder\*" fails (either 2: path not found or 123: invalid filename)
If I wait for 5 seconds, FindFirstFile will succeed on "location 2\folder\*"
Setting HKLM\System\CurrentControlSet\Services\LanmanWorkstation\Parameters\FileNotFoundCacheLifetime to 0 fixes the issue.
Create a new DWORD value if FileNotFoundCacheLifetime does not exist.
FileNotFoundCacheLifetime
In the next alpha update Everything will wait 30 seconds until FindFirstFile on "location 2\folder\*" succeeds (if FindFirstFile on "location 2\folder" succeeds)
Thank you for your offer to help with testing.If you would like additional testing of this optimization once implemented, please ping me back on this thread and I can test performance on a few very large file/folder count shares.
-
- Posts: 23
- Joined: Tue Apr 28, 2020 12:32 am
Re: (1.4/1.5a) Moved folders' contents missing from index, evade 'monitor changes', 'fast rescan', etc.
My question about this interaction would be: If 'Fast rescan' was running into the same issue, then why did subsequent manual (fast) rescans still fail to pick up the missing folder? Those rescans were done >5s after the move, yet they still missed the changes.From what I am seeing, this appears to be an issue with SMB cache.
Everything sees a new folder is created (folder in location 2) from ReadDirectoryChanges.
FindFirstFile on "location 2\folder" succeeds
FindFirstFile on "location 2\folder\*" fails (either 2: path not found or 123: invalid filename)
If I wait for 5 seconds, FindFirstFile will succeed on "location 2\folder\*"
Setting HKLM\System\CurrentControlSet\Services\LanmanWorkstation\Parameters\FileNotFoundCacheLifetime to 0 fixes the issue.
Create a new DWORD value if FileNotFoundCacheLifetime does not exist.
FileNotFoundCacheLifetime
In the next alpha update Everything will wait 30 seconds until FindFirstFile on "location 2\folder\*" succeeds (if FindFirstFile on "location 2\folder" succeeds)
Re: (1.4/1.5a) Moved folders' contents missing from index, evade 'monitor changes', 'fast rescan', etc.
Fast rescan would currently miss the change because:
FindFirstFile on "location 2\folder" succeeds
"location 2\folder" is added to the index with the correct date modified.
FindFirstFile on "location 2\folder\*" currently fails (hits FileNotFoundCache)
Everything will think "location 2\folder" is empty.
On the next fast rescan update:
Everything will see the date modified has not changed for "location 2\folder"
Everything will assume "location 2\folder" is still empty.
The next update will ensure "location 2\folder\*" succeeds so Everything will never get in this out-of-sync state.
FindFirstFile on "location 2\folder" succeeds
"location 2\folder" is added to the index with the correct date modified.
FindFirstFile on "location 2\folder\*" currently fails (hits FileNotFoundCache)
Everything will think "location 2\folder" is empty.
On the next fast rescan update:
Everything will see the date modified has not changed for "location 2\folder"
Everything will assume "location 2\folder" is still empty.
The next update will ensure "location 2\folder\*" succeeds so Everything will never get in this out-of-sync state.
Re: (1.4/1.5a) Moved folders' contents missing from index, evade 'monitor changes', 'fast rescan', etc.
Everything 1.5.0.1313a fixes an issue with re-adding moved folders.
Everything will now wait for the SMB cache to update when re-adding moved folders.
There's still a sync issue where a parent folder has a newer cache than a child.
Changes to children could be missed if they are made between the two parent/child cache snapshots.
This should be rare.
I have put on my TODO list to bypass the SMB cache.
Everything will now wait for the SMB cache to update when re-adding moved folders.
There's still a sync issue where a parent folder has a newer cache than a child.
Changes to children could be missed if they are made between the two parent/child cache snapshots.
This should be rare.
I have put on my TODO list to bypass the SMB cache.
-
- Posts: 23
- Joined: Tue Apr 28, 2020 12:32 am
Re: (1.4/1.5a) Moved folders' contents missing from index, evade 'monitor changes', 'fast rescan', etc.
Retried the above example with .1313a - the issue remains as it was before (not fixed). Tried waiting several minutes for SMB caches - no effect.void wrote: ↑Thu May 12, 2022 6:47 am Everything 1.5.0.1313a fixes an issue with re-adding moved folders.
I've included a zip of the example folder structure for reproduction. Extract to a share and then move the 'folder' from 'location 1' to 'location 2' and note that Everything will miss anything below layer 3 (and will evade a Fast rescan).
- Attachments
-
- move-test.zip
- (2.96 KiB) Downloaded 523 times
Last edited by malventano on Mon May 16, 2022 4:09 am, edited 1 time in total.
Re: (1.4/1.5a) Moved folders' contents missing from index, evade 'monitor changes', 'fast rescan', etc.
Thank you for testing 1313a malventano,
Thank you for the sample folder structure.
I was able to produce the issue my end.
Everything was not processing the list of folders to rescan correctly.
This issue will be fixed in the next alpha update.
Thank you for the sample folder structure.
I was able to produce the issue my end.
Everything was not processing the list of folders to rescan correctly.
This issue will be fixed in the next alpha update.
-
- Posts: 23
- Joined: Tue Apr 28, 2020 12:32 am
Re: (1.4/1.5a) Moved folders' contents missing from index, evade 'monitor changes', 'fast rescan', etc.
Glad to help. Does this mean the SMB cache was not actually an issue? Or perhaps the SMB cache fix will still be useful for other related cases?
I'll keep a lookout and retest once available. If successful on the example (for both NTFS and Linux+SMBD), I will re-perform the original folder moves on some much larger production shares to torture test the fixes .
Re: (1.4/1.5a) Moved folders' contents missing from index, evade 'monitor changes', 'fast rescan', etc.
SMB cache syncronization is still an issue for versions prior to 1313a.Glad to help. Does this mean the SMB cache was not actually an issue? Or perhaps the SMB cache fix will still be useful for other related cases?
I just did a test with your sample folder structure and the folder named "folder" was missing when this SMB cache fix is temporarily disabled.
It's only noticeable if you move a folder, then move it again within 5 seconds.
There's still possible sync issues when the SMB cache for a subfolder is older than a parent folder.
Thank you for your help with testing.
Re: (1.4/1.5a) Moved folders' contents missing from index, evade 'monitor changes', 'fast rescan', etc.
Everything 1.5.0.1314a fixes an issue with rescanning moved folders.
Everything should now correctly rescan the full sub folder structure after a folder is moved.
Everything should now correctly rescan the full sub folder structure after a folder is moved.