Folder Scan incomplete
Folder Scan incomplete
Hi
still the scanning behavior looks pretty implausible to me. On large volumes I scanned subfolders separately and quite often the size of data found in some subfolders is larger than the top folder. Some subfolders may have access restrictions and I have the feeling this affects the overall result. I did a complete rescan with the most current version because the scanning behavior has been changed. Still it is the same.
Unfortunately, I cannot undisclose folder scans structure. But top level is like see attachment. I will detail the subfolders a bit more for a next test to see if a certain subfolder is causing it...
Best,
Fred
still the scanning behavior looks pretty implausible to me. On large volumes I scanned subfolders separately and quite often the size of data found in some subfolders is larger than the top folder. Some subfolders may have access restrictions and I have the feeling this affects the overall result. I did a complete rescan with the most current version because the scanning behavior has been changed. Still it is the same.
Unfortunately, I cannot undisclose folder scans structure. But top level is like see attachment. I will detail the subfolders a bit more for a next test to see if a certain subfolder is causing it...
Best,
Fred
- Attachments
-
- Everything Subfolders smaller than top level.jpg (40.23 KiB) Viewed 26327 times
Re: Folder Scan incomplete
If you have an installation using the service there should be no access restriction.
If you can't do this, user folder indexing with folders you definetely have read access.
If you can't do this, user folder indexing with folders you definetely have read access.
Re: Folder Scan incomplete
Please check the debug log.
Scanning errors are reported in red in the debug console.
The debug console can be shown from Tools -> Debug -> Console.
Please try rescanning when debug logging started:
Scanning errors are reported in red in the debug console.
The debug console can be shown from Tools -> Debug -> Console.
Please try rescanning when debug logging started:
- In Everything, from the Tools menu, under the Debug submenu, click Start Debug Logging.
- From the Tools menu, click Options.
- Click the Folders tab on the left.
- Select your \\server\share
- Please make sure Fast Rescan is unchecked.
- Click Rescan Now.
- After rescanning completes (current folder being rescanned is shown at the top right):
- In Everything, from the Tools menu, under the Debug submenu, click Stop Debug Logging.
---This will open your Everything Debug Log.txt---
Please check this log for any FindFirstFileEx errors - What error codes are reported? (if any)
- If unsure, please send this file to support@voidtools.com
Re: Folder Scan incomplete
Dear void team,
I did the following test and found some FindFirstFileEx error codes 1908.
Reason seems to be: the top folder takes 1-2 days to scan. In the afternoon I checked the progress and the top folder was about 50% done. Then I switched the notebook in power saving/hyberenate.
In the morning the rest of the top folder was imediatelly skipped. From the logs I can see that all the rest of the subfolders got FindFirstFileEx error codes 1908. => scan completed
I suspect that always after waking up the notebook in the morning the rest of the current rescan activity is ended the same way and contained files are gone from the database.
Best regards,
Martin
I did the following test and found some FindFirstFileEx error codes 1908.
Reason seems to be: the top folder takes 1-2 days to scan. In the afternoon I checked the progress and the top folder was about 50% done. Then I switched the notebook in power saving/hyberenate.
In the morning the rest of the top folder was imediatelly skipped. From the logs I can see that all the rest of the subfolders got FindFirstFileEx error codes 1908. => scan completed
I suspect that always after waking up the notebook in the morning the rest of the current rescan activity is ended the same way and contained files are gone from the database.
Best regards,
Martin
Re: Folder Scan incomplete
Thank you for your feedback Martin,
Error 1908 is ERROR_DOMAIN_CONTROLLER_NOT_FOUND
The next alpha update will wait for 1 minute for the network to reconnect when rescanning your network share.
I'll make another post here once this is ready for testing.
Everything should take roughly 1 minute to index 1 million files on a network drive.
Ideally, Everything should be indexing your network drive within 60 minutes.
Everything is not really designed to index for longer than 60 minutes.
How many files are you indexing?
Error 1908 is ERROR_DOMAIN_CONTROLLER_NOT_FOUND
The next alpha update will wait for 1 minute for the network to reconnect when rescanning your network share.
I'll make another post here once this is ready for testing.
Everything is taking too long to index your network drive.the top folder takes 1-2 days to scan
Everything should take roughly 1 minute to index 1 million files on a network drive.
Ideally, Everything should be indexing your network drive within 60 minutes.
Everything is not really designed to index for longer than 60 minutes.
How many files are you indexing?
Re: Folder Scan incomplete
Hi,
when I do DIR /S on a folder structure it takes roughly 1.5sec for a folder. There are >100.000 folders and 650k files. (at least to the point Everything did the indexing). So also Dir /s would take similarly long...
I assume there might be rate limiter on server side or active directory or whatever else...
Have a good day,
Martin
when I do DIR /S on a folder structure it takes roughly 1.5sec for a folder. There are >100.000 folders and 650k files. (at least to the point Everything did the indexing). So also Dir /s would take similarly long...
I assume there might be rate limiter on server side or active directory or whatever else...
Have a good day,
Martin
Re: Folder Scan incomplete
Everything 1.5.0.1334a will now wait for one minute for the network to reconnect after waking from sleep.
Please check your network, hardware and domain settings.
Something is not right here, indexing ~1 million files should only take 1 minute, not 1-2days.when I do DIR /S on a folder structure it takes roughly 1.5sec for a folder
Please check your network, hardware and domain settings.
Re: Folder Scan incomplete
Hello, thank you for the improvement.
It took quite a while, but the folder sizes look plausible now!
The slower the browsing is the more useful is a tool like Everything
On my end I cannot do too much to improve the infrastructure. Maybe the limitation is intended.
Asking friends working for other companies the scan rate is 5-20 folders/second.
My 650k folders would still take 9h.. multiple cross domain scans seem to be the worst case...
Anyway, good that it continues scanning after waking from PC sleep mode now!
It took quite a while, but the folder sizes look plausible now!
The slower the browsing is the more useful is a tool like Everything
On my end I cannot do too much to improve the infrastructure. Maybe the limitation is intended.
Asking friends working for other companies the scan rate is 5-20 folders/second.
My 650k folders would still take 9h.. multiple cross domain scans seem to be the worst case...
Anyway, good that it continues scanning after waking from PC sleep mode now!
Last edited by Fred0815 on Tue Jan 31, 2023 10:38 am, edited 1 time in total.
Re: Folder Scan incomplete
I forgot to highlight my conclusion: It is really important to keep scanned network data as long as possible.
If a scan still fails for reasons not known so far, the files/folders should be kept for a certain time.
Maybe a property "last seen" help or marking them "obsolete".
If large amount of data will be deleted (because Everything thinks the files are gone) users should acknowledge.
I can also image a "last seen" property may bring new fields of application to Everything because it included the history of a folder structure.
So, an option could be "show history" or not and never (or manually) delete valuable information!
If a scan still fails for reasons not known so far, the files/folders should be kept for a certain time.
Maybe a property "last seen" help or marking them "obsolete".
If large amount of data will be deleted (because Everything thinks the files are gone) users should acknowledge.
I can also image a "last seen" property may bring new fields of application to Everything because it included the history of a folder structure.
So, an option could be "show history" or not and never (or manually) delete valuable information!
Re: Folder Scan incomplete
Hi again,
unfortunately, there's still some issues with losing data from scanned folders and debug was not running.
A larger folder was scanning for almost two days and during that scan I moved from homeoffice to office. The notebook was in hybernate mode.
After logging in the scan was ended and a new one was running. All information from the previous scan is lost.
It may be of interest that we use Zscaler to connect to our drives.
During another event I noticed that Everything took its minute to let the network recover. Now it looks like Everything did not wait.
I am also wondering why ALL data of the scan is lost. Wouldn't it be possible to add new files/folder more frequently? e.g. every 1000 items?
In that case losses would be limited.
Re-scans should scan "unknown" folders first to continue after a partial scan is re-scanned.
Will post again as soon I can reproduce it with Debug ON.
Cheers,
Martin
unfortunately, there's still some issues with losing data from scanned folders and debug was not running.
A larger folder was scanning for almost two days and during that scan I moved from homeoffice to office. The notebook was in hybernate mode.
After logging in the scan was ended and a new one was running. All information from the previous scan is lost.
It may be of interest that we use Zscaler to connect to our drives.
During another event I noticed that Everything took its minute to let the network recover. Now it looks like Everything did not wait.
I am also wondering why ALL data of the scan is lost. Wouldn't it be possible to add new files/folder more frequently? e.g. every 1000 items?
In that case losses would be limited.
Re-scans should scan "unknown" folders first to continue after a partial scan is re-scanned.
Will post again as soon I can reproduce it with Debug ON.
Cheers,
Martin
Re: Folder Scan incomplete
Thank you for the issue report.
The wake from hibernate is most likely killing the rescan.
It will be interesting to see the error code in the debug logs.
Some hard errors will force Everything to abandon the rescan immediately.
Debug logs will show the scanning error code.
I'll consider an option to add on the fly.
Removing deleted files/folders no.
Thank you for enabling logging.
The wake from hibernate is most likely killing the rescan.
It will be interesting to see the error code in the debug logs.
It will depend on the error code.During another event I noticed that Everything took its minute to let the network recover. Now it looks like Everything did not wait.
Some hard errors will force Everything to abandon the rescan immediately.
Debug logs will show the scanning error code.
Adding files/folders maybe...I am also wondering why ALL data of the scan is lost. Wouldn't it be possible to add new files/folder more frequently? e.g. every 1000 items?
I'll consider an option to add on the fly.
Removing deleted files/folders no.
Thank you for enabling logging.
Re: Folder Scan incomplete
Good morning, I am happy that I captured the issue (hopefully).
Normally I have the Notebook in Hybernate, but now I found the problem occurs after a normal restart.
In our environment it may take several minutes ... 30min until all network drives are accessible. Some shares require signing in with different methods, others just take that long.
If the drives are accessed before they are properly working we get Error Code 0x80070035 (Network path not found).
I added the Windows error message as well the part of the log file.
In my opinion scanned files and folders must not easily be deleted! Imagine being on a business trip without logging in the proper way would cause all items deleted (in case a re-scan is running on that folder).
The folder it scanned this time took almost a week and at 90% done this one happened. I have six larger folders that never finished within several month now.
Normally I have the Notebook in Hybernate, but now I found the problem occurs after a normal restart.
In our environment it may take several minutes ... 30min until all network drives are accessible. Some shares require signing in with different methods, others just take that long.
If the drives are accessed before they are properly working we get Error Code 0x80070035 (Network path not found).
I added the Windows error message as well the part of the log file.
In my opinion scanned files and folders must not easily be deleted! Imagine being on a business trip without logging in the proper way would cause all items deleted (in case a re-scan is running on that folder).
The folder it scanned this time took almost a week and at 90% done this one happened. I have six larger folders that never finished within several month now.
- Attachments
-
- 2023-02-27 09_46_05-Everything Voidtools V01.docx - Word.jpg (115.29 KiB) Viewed 24821 times
Re: Folder Scan incomplete
Hi, this one really drives me crazy! I had one folder scan running for 6days and to prevent the error above I put indexing to PAUSE > shutdown notebook completely.
Next day (in office environment) I continued indexing. It worked well for 2h but then Everything switched to the next folder and did not store any file or folder from the previous scan.
Debug ON has not been remembered and was off, so I cannot (again) identify the reason.
=> would it be possible to keep Debug On by default for the alpha versions? It's a pain in the a.. to dig into this error, but I believe the scanning mode needs to modified, chunks should be stored every 1k...100k items to not loose it + rescan shall scan "unknown" folders first, otherwise there will never be progress...
Disappointed Greetings,
Martin
Next day (in office environment) I continued indexing. It worked well for 2h but then Everything switched to the next folder and did not store any file or folder from the previous scan.
Debug ON has not been remembered and was off, so I cannot (again) identify the reason.
=> would it be possible to keep Debug On by default for the alpha versions? It's a pain in the a.. to dig into this error, but I believe the scanning mode needs to modified, chunks should be stored every 1k...100k items to not loose it + rescan shall scan "unknown" folders first, otherwise there will never be progress...
Disappointed Greetings,
Martin
Re: Folder Scan incomplete
To keep debug logs on:=> would it be possible to keep Debug On by default for the alpha versions?
- Copy and paste the following into your Everything search box:
/debug_log=1 - Press ENTER in your Everything search box.
- If successful, debug_log=1 is shown in the status bar for a few seconds.
--- - Copy and paste the following into your Everything search box:
/restart - Press ENTER in your Everything search box.
FindNextFile <error-code>
FindFirstFileEx <error-code> <path>
Please make sure you are using version 1339 or later for this detailed debug information.
Please let me know if you find any errors reported.
Please note, debug logs can grow very large.
To turn off debug logs:
- Copy and paste the following into your Everything search box:
/debug_log=0 - Press ENTER in your Everything search box.
- If successful, debug_log=0 is shown in the status bar for a few seconds.
--- - Copy and paste the following into your Everything search box:
/restart - Press ENTER in your Everything search box.
Error 0x80070035 is ERROR_BAD_NETPATH
Everything expects the path to always be online during the scan.
You can do partial updates with a /rescan search command.
For example:
/rescan \\server\share\a folder\that needs\updating
If you can, instead of indexing \\server\share, please try indexing \\server\share\folder1, \\server\share\folder2, \\server\share\folder3 etc...
This will help split the updating done over multiple indexes.
Re: Folder Scan incomplete
Hi, I will mitigate and split the task as much as possible. Although there are some "flat" large folders with some hundred subfolders. These cannot be split...
Folder/Sub1
Folder/...
Folder/SubNx100
Folder/Sub1
Folder/...
Folder/SubNx100
Re: Folder Scan incomplete
Hi
this time with debug on I caught the issue.
I did a normal restart and the folder that has been scanned before ended and approx. every 1minute another folder was scanned but also ended every 1min. After some minutes all re-scanning/scanning was finished without results.
Looking at the log I cannot see why it ended, so I add it here. Folders/server needed to be renamed, but structure in general is visible.
It started with P-Order and then changed as described above.
Best,
Martin
this time with debug on I caught the issue.
I did a normal restart and the folder that has been scanned before ended and approx. every 1minute another folder was scanned but also ended every 1min. After some minutes all re-scanning/scanning was finished without results.
Looking at the log I cannot see why it ended, so I add it here. Folders/server needed to be renamed, but structure in general is visible.
It started with P-Order and then changed as described above.
Best,
Martin
Last edited by void on Tue Mar 14, 2023 11:09 am, edited 1 time in total.
Reason: removed logs
Reason: removed logs
Re: Folder Scan incomplete
Thanks for the logs Martin,
I'm looking into the issue..
I'm looking into the issue..
Re: Folder Scan incomplete
Everything is failing to access the root network share.
I'm unsure why at this stage.
I will add more debug information in the next alpha update.
I'll report back here again soon.
I'm unsure why at this stage.
I will add more debug information in the next alpha update.
I'll report back here again soon.
Re: Folder Scan incomplete
Everything 1.5.0.1340a adds more debug information when trying to access root folders.
Could you please repeat the same debug log capture with this version:
Please try rescanning when debug logging started:
Could you please repeat the same debug log capture with this version:
Please try rescanning when debug logging started:
- In Everything, from the Tools menu, under the Debug submenu, click Start Debug Logging.
- From the Tools menu, click Options.
- Click the Folders tab on the left.
- Select your \\server\share
- Click Rescan Now.
- After rescanning completes (current folder being rescanned is shown at the top right):
- In Everything, from the Tools menu, under the Debug submenu, click Stop Debug Logging.
---This will open your Everything Debug Log.txt--- - Please check this log for the following error:
failed to get find data for <filename>: <error-code> - What error codes are reported? (if any)
- If unsure, please send this file to support@voidtools.com
Re: Folder Scan incomplete
Good morning,
I got error code 1908. Note that the PC was restored from power saving/sleep at 9:06:00.
After only 19s Everything stopped scanning without waiting/retry. Shouldn't it wait/retry at least for 1min?
As I said, sometimes it takes up to 15min until all network connections work well. I guess active directory and other services take that time.
You might think "what a crappy network", but the company I am working with has >200.000 employees, so it is not like a garage-startup not able to handle networks... I asked many other friends to benchmark their network in other companies and the speed to scan a network is similar.
So, I guess in order to use Everything, there needs to be some modification to account for very long scan times.
The easiest workaround could be:
- write scan results after an error and do not ignore them
- do not delete scan results if the folder is not available
- continue scanning: skip known folders and concentrate on unknown folders. Hopefully, that avoids re-scanning from start again and again and never reach the end
Best regards,
2023-03-27 09:06:19.781: failed to get find data for \\Server1.net\a\a\a\a\20_CSY: 1908
2023-03-27 09:06:19.781: scan completed in 315845.332347 seconds
2023-03-27 09:06:19.781: rescan update db
2023-03-27 09:06:19.781: rescan update db in 0.000316 seconds
2023-03-27 09:06:19.781: next update at 8/05 0:06
2023-03-27 09:06:19.862: leave folder update
2023-03-27 09:06:19.864: check for folder updates 0000000000000000
2023-03-27 09:06:19.864: update folder start \\Server1.net\a\a\a\a\30_ES
PS: error 53 often occurs on folders with missing access permission. In that case Everything retries 10x? every 1min. I guess that's not necessary. Such folders may be skipped instead.
I got error code 1908. Note that the PC was restored from power saving/sleep at 9:06:00.
After only 19s Everything stopped scanning without waiting/retry. Shouldn't it wait/retry at least for 1min?
As I said, sometimes it takes up to 15min until all network connections work well. I guess active directory and other services take that time.
You might think "what a crappy network", but the company I am working with has >200.000 employees, so it is not like a garage-startup not able to handle networks... I asked many other friends to benchmark their network in other companies and the speed to scan a network is similar.
So, I guess in order to use Everything, there needs to be some modification to account for very long scan times.
The easiest workaround could be:
- write scan results after an error and do not ignore them
- do not delete scan results if the folder is not available
- continue scanning: skip known folders and concentrate on unknown folders. Hopefully, that avoids re-scanning from start again and again and never reach the end
Best regards,
2023-03-27 09:06:19.781: failed to get find data for \\Server1.net\a\a\a\a\20_CSY: 1908
2023-03-27 09:06:19.781: scan completed in 315845.332347 seconds
2023-03-27 09:06:19.781: rescan update db
2023-03-27 09:06:19.781: rescan update db in 0.000316 seconds
2023-03-27 09:06:19.781: next update at 8/05 0:06
2023-03-27 09:06:19.862: leave folder update
2023-03-27 09:06:19.864: check for folder updates 0000000000000000
2023-03-27 09:06:19.864: update folder start \\Server1.net\a\a\a\a\30_ES
PS: error 53 often occurs on folders with missing access permission. In that case Everything retries 10x? every 1min. I guess that's not necessary. Such folders may be skipped instead.
Re: Folder Scan incomplete
Thank you for your reply Fred0815,
Error 1908 is ERROR_DOMAIN_CONTROLLER_NOT_FOUND.
Everything 1.5.0.1341a will now retry scanning the folder after receiving this error.
Everything will retry every minute for up-to 5 minutes.
Error 53 is ERROR_BAD_NETPATH.
This error is common when the network is down or a server is offline.
It's important for Everything to wait and retry on this error.
Please try excluding folder paths where you don't have access under Tools -> Options -> Exclude.
Error 1908 is ERROR_DOMAIN_CONTROLLER_NOT_FOUND.
Everything 1.5.0.1341a will now retry scanning the folder after receiving this error.
Everything will retry every minute for up-to 5 minutes.
Error 53 is ERROR_BAD_NETPATH.
This error is common when the network is down or a server is offline.
It's important for Everything to wait and retry on this error.
Please try excluding folder paths where you don't have access under Tools -> Options -> Exclude.
Re: Folder Scan incomplete
Thank you, will try.
The folder structures are too deeply nested to individually exclude folders with no authorization.
I found this option that was ON so far. I am wondering whether that helps to keep "offline" data: Network drives/Automatically remove offline mapped network drives => now OFF
BTW: I modified Indexes options to add "date created" (without "force rebuild"). That caused ALL 7mio files to be deleted from the database... OK, starting from scratch again.
=> such huge delete operations must be avoided (some ideas in my previous mails). Minimum would be to have a confirmation dialog. + maybe a backup of the db to allow to restore it.
The folder structures are too deeply nested to individually exclude folders with no authorization.
I found this option that was ON so far. I am wondering whether that helps to keep "offline" data: Network drives/Automatically remove offline mapped network drives => now OFF
BTW: I modified Indexes options to add "date created" (without "force rebuild"). That caused ALL 7mio files to be deleted from the database... OK, starting from scratch again.
=> such huge delete operations must be avoided (some ideas in my previous mails). Minimum would be to have a confirmation dialog. + maybe a backup of the db to allow to restore it.
Re: Folder Scan incomplete
Version 1341a now makes a backup of the database file.Fred0815 wrote: ↑Wed Apr 05, 2023 11:21 am Thank you, will try.
The folder structures are too deeply nested to individually exclude folders with no authorization.
I found this option that was ON so far. I am wondering whether that helps to keep "offline" data: Network drives/Automatically remove offline mapped network drives => now OFF
BTW: I modified Indexes options to add "date created" (without "force rebuild"). That caused ALL 7mio files to be deleted from the database... OK, starting from scratch again.
=> such huge delete operations must be avoided (some ideas in my previous mails). Minimum would be to have a confirmation dialog. + maybe a backup of the db to allow to restore it.
viewtopic.php?p=55420#p55420
Re: Folder Scan incomplete
great, thank you
Re: Folder Scan incomplete
Hello again,
Everything scanned
folder A: \\srv1.net\a\b\c\d\gr\Projekte
after a restart of the PC it looks like without any error it ended folder A, did not write anything to db and continued with
folder B \\srv1.net\a\b\c\d\gr\Standardprojekt\Standards
Actually, Everything scans currently folder A and shows progress in the tooltip text. But the current log does not show scanning activities in folder A as far as I can see it.
Strange things happen.... any idea?
Thanks,
Martin
2023-04-13 13:03:21.805: scan completed in 1433.184708 seconds
2023-04-13 13:03:21.806: rescan update db
2023-04-13 13:03:21.912: rescan update db in 0.107055 seconds
2023-04-13 13:03:21.912: next update at 25/05 4:03
2023-04-13 13:03:21.913: leave folder update
2023-04-13 13:03:21.913: check for folder updates 0000000000000000
2023-04-13 13:03:21.913: update folder start \\srv1.net\a\b\c\d\gr\Projekte
2023-04-13 13:03:21.913: scanning \\srv1.net\a\b\c\d\gr\Projekte...
2023-04-13 13:03:21.914: RESCAN CREATE \\srv1.net\a\b\c\d\gr\Standardprojekt\Standards
Everything scanned
folder A: \\srv1.net\a\b\c\d\gr\Projekte
after a restart of the PC it looks like without any error it ended folder A, did not write anything to db and continued with
folder B \\srv1.net\a\b\c\d\gr\Standardprojekt\Standards
Actually, Everything scans currently folder A and shows progress in the tooltip text. But the current log does not show scanning activities in folder A as far as I can see it.
Strange things happen.... any idea?
Thanks,
Martin
2023-04-13 13:03:21.805: scan completed in 1433.184708 seconds
2023-04-13 13:03:21.806: rescan update db
2023-04-13 13:03:21.912: rescan update db in 0.107055 seconds
2023-04-13 13:03:21.912: next update at 25/05 4:03
2023-04-13 13:03:21.913: leave folder update
2023-04-13 13:03:21.913: check for folder updates 0000000000000000
2023-04-13 13:03:21.913: update folder start \\srv1.net\a\b\c\d\gr\Projekte
2023-04-13 13:03:21.913: scanning \\srv1.net\a\b\c\d\gr\Projekte...
2023-04-13 13:03:21.914: RESCAN CREATE \\srv1.net\a\b\c\d\gr\Standardprojekt\Standards
Re: Folder Scan incomplete
Thank you for your reply Martin,
To enable verbose logging:
Everything will abandon a rescan if you restart your PC.
Everything will only show the current folder being scanned in the console/log if verbose logging is enabled.But the current log does not show scanning activities in folder A as far as I can see it.
To enable verbose logging:
- In Everything, from the Tools menu, under the Debug submenu, check Verbose.
Everything will abandon a rescan if you restart your PC.
Re: Folder Scan incomplete
Hi again, ok I switched verbose ON.
looks like a few network error 64 showed up and after 1 minute showing the error scanning ended + all results are lost.
Again: I don't know how many other errors will show up causing to loose all the information from the current scan.
I my opinion only a different approach to have higher priority to keep data will help...
BR,
Martin
2023-04-18 12:28:52.376: FindFirstFileEx 64 srv1.net\a\b\c\d\sysadmi\Software\Remote\CiscoVPN\*
2023-04-18 12:28:52.538: MSG: 00000000000305a2 0113 0000000000000002 0000000000000000
2023-04-18 12:28:52.540: TRAY 00000113 0000000000000002 0000000000000000
2023-04-18 12:28:52.540: _db_query_results_changed_event_proc 0000000000000000
2023-04-18 12:28:52.540: _db_query_results_changed_delay_timer_create
2023-04-18 12:28:52.540: new results 2788969
2023-04-18 12:28:52.540: listview message 0000004e 0000000000002719 0000000000fbdbf0
2023-04-18 12:28:52.540: listview message 0000004e 0000000000002719 0000000000fbdbf0
2023-04-18 12:28:52.540: get_total_result_size 2346847 3627586310821
2023-04-18 12:28:52.541: MSG OK
2023-04-18 12:28:52.550: FindFirstFileEx 64 srv1.net\a\b\c\d\sysadmi\Software\Remote\CiscoVPN\*
2023-04-18 12:28:52.568: MSG: 00000000000305a2 0113 0000000000000003 0000000000000000
2023-04-18 12:28:52.570: TRAY 00000113 0000000000000003 0000000000000000
2023-04-18 12:28:52.570: MSG OK
2023-04-18 12:28:52.608: failed to get find data for srv1.net\a\b\c\d\sysadmi: 64
2023-04-18 12:28:52.708: MSG: 00000000000305a2 0113 0000000000000000 0000000000000000
2023-04-18 12:28:52.708: TRAY 00000113 0000000000000000 0000000000000000
2023-04-18 12:28:52.708: MSG OK
2023-04-18 12:28:52.709: MSG: 00000000000305a2 0113 0000000000000001 0000000000000000
2023-04-18 12:28:52.709: TRAY 00000113 0000000000000001 0000000000000000
2023-04-18 12:28:52.709: check for folder updates 000000001e0dab10
2023-04-18 12:28:52.709: MSG OK
looks like a few network error 64 showed up and after 1 minute showing the error scanning ended + all results are lost.
Again: I don't know how many other errors will show up causing to loose all the information from the current scan.
I my opinion only a different approach to have higher priority to keep data will help...
BR,
Martin
2023-04-18 12:28:52.376: FindFirstFileEx 64 srv1.net\a\b\c\d\sysadmi\Software\Remote\CiscoVPN\*
2023-04-18 12:28:52.538: MSG: 00000000000305a2 0113 0000000000000002 0000000000000000
2023-04-18 12:28:52.540: TRAY 00000113 0000000000000002 0000000000000000
2023-04-18 12:28:52.540: _db_query_results_changed_event_proc 0000000000000000
2023-04-18 12:28:52.540: _db_query_results_changed_delay_timer_create
2023-04-18 12:28:52.540: new results 2788969
2023-04-18 12:28:52.540: listview message 0000004e 0000000000002719 0000000000fbdbf0
2023-04-18 12:28:52.540: listview message 0000004e 0000000000002719 0000000000fbdbf0
2023-04-18 12:28:52.540: get_total_result_size 2346847 3627586310821
2023-04-18 12:28:52.541: MSG OK
2023-04-18 12:28:52.550: FindFirstFileEx 64 srv1.net\a\b\c\d\sysadmi\Software\Remote\CiscoVPN\*
2023-04-18 12:28:52.568: MSG: 00000000000305a2 0113 0000000000000003 0000000000000000
2023-04-18 12:28:52.570: TRAY 00000113 0000000000000003 0000000000000000
2023-04-18 12:28:52.570: MSG OK
2023-04-18 12:28:52.608: failed to get find data for srv1.net\a\b\c\d\sysadmi: 64
2023-04-18 12:28:52.708: MSG: 00000000000305a2 0113 0000000000000000 0000000000000000
2023-04-18 12:28:52.708: TRAY 00000113 0000000000000000 0000000000000000
2023-04-18 12:28:52.708: MSG OK
2023-04-18 12:28:52.709: MSG: 00000000000305a2 0113 0000000000000001 0000000000000000
2023-04-18 12:28:52.709: TRAY 00000113 0000000000000001 0000000000000000
2023-04-18 12:28:52.709: check for folder updates 000000001e0dab10
2023-04-18 12:28:52.709: MSG OK
Re: Folder Scan incomplete
Thank you for the logs Martin,
Everything 1.5.0.1344a simplifies rescanning folders.
If FindFirstFile/FindNextFile fails for any reason, Everything will check if the root volume is still online.
If the root volume is still online, the folder is treated as empty and Everything will continue.
If the root volume is offline, Everything will wait for up-to 5 minutes until the volume comes back online and try again.
If the root volume does not come back online after 5 minutes, Everything will abort the rescan process and reschedule a rescan for another time.
Please let me know if this helps with rescanning the folder.
Everything 1.5.0.1344a simplifies rescanning folders.
If FindFirstFile/FindNextFile fails for any reason, Everything will check if the root volume is still online.
If the root volume is still online, the folder is treated as empty and Everything will continue.
If the root volume is offline, Everything will wait for up-to 5 minutes until the volume comes back online and try again.
If the root volume does not come back online after 5 minutes, Everything will abort the rescan process and reschedule a rescan for another time.
Please let me know if this helps with rescanning the folder.
Re: Folder Scan incomplete
Thank you, will try...
Just noticed that .1344 deletes all previous scan results...
will try to use db backup.... of course it does not help.. still empty...
ok, I am used to start from scratch... so let's do it again...
;-|
Just noticed that .1344 deletes all previous scan results...
will try to use db backup.... of course it does not help.. still empty...
ok, I am used to start from scratch... so let's do it again...
;-|
Re: Folder Scan incomplete
As noted in the 1344a announcement the database format has changed.
You can't use an old database, it will be automatically rebuild.
Re: Folder Scan incomplete
after reboot network takes up to 30min to establish. Everything 1350 tries several times (total a few minutes) and then skips this folder.
All scan results done so far are still lost. I thought the database changes were meant to address it?
2023-06-20 09:21:12.036: failed to get find data for \\aserver.net\101\File\E\ORDERs\Transfer: 50
=> there is so much going on in Everything development, but the real basics are not addressed sufficiently. Am I really the only one suffering from slow network? We are >200.000 employees over here...
All scan results done so far are still lost. I thought the database changes were meant to address it?
2023-06-20 09:21:12.036: failed to get find data for \\aserver.net\101\File\E\ORDERs\Transfer: 50
=> there is so much going on in Everything development, but the real basics are not addressed sufficiently. Am I really the only one suffering from slow network? We are >200.000 employees over here...
Re: Folder Scan incomplete
For such environment, there is the Everything server.Fred0815 wrote: ↑Tue Jun 20, 2023 8:57 am after reboot network takes up to 30min to establish. Everything 1350 tries several times (total a few minutes) and then skips this folder.
All scan results done so far are still lost. I thought the database changes were meant to address it?
2023-06-20 09:21:12.036: failed to get find data for \\aserver.net\101\File\E\ORDERs\Transfer: 50
=> there is so much going on in Everything development, but the real basics are not addressed sufficiently. Am I really the only one suffering from slow network? We are >200.000 employees over here...
Re: Folder Scan incomplete
This sounds like an unusually long time..after reboot network takes up to 30min to establish
Please try increasing your folder_rescan_timeout:
- In Everything 1.5, from the Tools menu, click Options.
- Click the Advanced tab on the left.
- To the right of Show settings containing, search for:
rescan - Select folder_rescan_timeout.
- Set the value to: 3600000
(this is one hour, the default is 300000, which is 5 minutes) - Click OK.
Everything Server
Re: Folder Scan incomplete
I implemented 30min delay to skip scanning a folder.
I observe some folder shares are even longer as 30min offline.
Issue: last scanned folder over night “…Pro_m_L” just was skipped to another one without obvious reason.
No result was committed to the database.
Looking at the long time we try to get this working I doubt that we will be successful by doing it half-hearted.
To make it right it takes the measures I suggested in my previous comments:
- Never drop any data (possible option/manual step)
- Commit to database every minute...10min
- Add e.g. “last seen/deleted” to the database to identify files that (temporarily) might not be available
- Continue scanning a folder where it ended before and not start from start again and again
I know that is much of a work, but it also opens a path to new features based on the history of files.
=> I sent the logfile separately via mail
I observe some folder shares are even longer as 30min offline.
Issue: last scanned folder over night “…Pro_m_L” just was skipped to another one without obvious reason.
No result was committed to the database.
Looking at the long time we try to get this working I doubt that we will be successful by doing it half-hearted.
To make it right it takes the measures I suggested in my previous comments:
- Never drop any data (possible option/manual step)
- Commit to database every minute...10min
- Add e.g. “last seen/deleted” to the database to identify files that (temporarily) might not be available
- Continue scanning a folder where it ended before and not start from start again and again
I know that is much of a work, but it also opens a path to new features based on the history of files.
=> I sent the logfile separately via mail
Re: Folder Scan incomplete
Thank you for your feedback and logs Fred0815,
Adding a drive to your index is interrupting the folder scan.
I'll look into keeping the folder scan going when rebuilding the index.
For now, please disable automatically include NTFS volumes and please disable automatically remove NTFS volumes:
Please let me know if the issue persists.
Everything will timeout after the setting folder_rescan_timeout (5 minutes by default -30 minutes in your case).
A rebuild will currently interrupt a folder rescan.
I'll look into keeping the folder scan going when rebuilding the index.
I'll look into doing this.
For now, Last rescan information is available under Tools -> Debug -> Statistics -> Folder Index.
For now, please consider breaking your folder index into smaller folders.
For example, instead of indexing \\server\share, index:
\\server\share\folder1
\\server\share\folder2
\\server\share\folder3
Thank you for the suggestions.
Adding a drive to your index is interrupting the folder scan.
I'll look into keeping the folder scan going when rebuilding the index.
For now, please disable automatically include NTFS volumes and please disable automatically remove NTFS volumes:
- In Everything, from the Tools menu, click Options.
- Click the NTFS tab on the left.
- Uncheck Automatically include new fixed volumes.
- Uncheck Automatically include new removable volumes.
- Uncheck Automatically remove offline volumes.
- Click OK.
Please let me know if the issue persists.
Everything should keep trying to read a directory listing on any error.Never drop any data (possible option/manual step)
Everything will timeout after the setting folder_rescan_timeout (5 minutes by default -30 minutes in your case).
A rebuild will currently interrupt a folder rescan.
I'll look into keeping the folder scan going when rebuilding the index.
Tough to do without a full rescan.Commit to database every minute...10min
I'll look into doing this.
I will consider adding properties to show this information.Add e.g. “last seen/deleted” to the database to identify files that (temporarily) might not be available
For now, Last rescan information is available under Tools -> Debug -> Statistics -> Folder Index.
I will consider support for saving the current scan to disk.Continue scanning a folder where it ended before and not start from start again and again
For now, please consider breaking your folder index into smaller folders.
For example, instead of indexing \\server\share, index:
\\server\share\folder1
\\server\share\folder2
\\server\share\folder3
Thank you for the suggestions.