Everything Server Crashing Every Few Days

[JohnTravolski]

Hi, I've been running Everything 1.5a with Everything server for about a month now, and I've noticed that it has been consistently crashing after leaving it on for several days consecutively. I notice this each time because the Everything clients show the files as grayed out when this happens.

Do you have any suggestions for what I should do to try and understand what's going wrong? Are crash logs written somewhere?

[void]

Thank you for the crash report JohnTravolski,

How are you running Everything on the Server? with a logged in user or as a service?



If you are running Everything with a logged in user, is a crash dialog shown?
-If so:

• Please keep the crash dialog open.
• Download Process Explorer from Microsoft:
  https://docs.microsoft.com/en-us/sysinternals/downloads/process-explorer
• In Process Explorer, select Everything64.exe (If there is two, select the one using the most memory)
• Right click Everything64.exe and under Create Dump, select Create Mini Dump.
• Choose a filename and click Save.
• Please upload this file in a bug report.

To enable mini crash dumps for Everything:

• Create the following registry key:
  
  HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps\Everything.exe
  
  
• In this key, create the following dword value, with the name:
  
  DumpCount
  
  Set the value data to:
  
  3
  
  
• Create the following dword value, with the name:
  
  DumpType
  
  Set the value data to:
  
  1
  
  (1 = mini crash dump)
  
• Create the following expandable string value, with the name:
  
  DumpFolder
  
  Set the value data to:
  
  %LOCALAPPDATA%\Everything
  
  (or some writable location)
  
• Wait for Everything to crash
• Please upload the .dmp file from %LOCALAPPDATA%\Everything in a bug report.

In the time being, please monitor RAM usage and handle usage in the Task Manager.

What are you indexing with Everything?
Please upload your Help -> Troubleshooting information in a bug report.

[JohnTravolski]

Thank you for the crash report JohnTravolski,

How are you running Everything on the Server? with a logged in user or as a service?



If you are running Everything with a logged in user, is a crash dialog shown?
-If so:

• Please keep the crash dialog open.
• Download Process Explorer from Microsoft:
  https://docs.microsoft.com/en-us/sysinternals/downloads/process-explorer
• In Process Explorer, select Everything64.exe (If there is two, select the one using the most memory)
• Right click Everything64.exe and under Create Dump, select Create Mini Dump.
• Choose a filename and click Save.
• Please upload this file in a bug report.

To enable mini crash dumps for Everything:

• Create the following registry key:
  
  HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps\Everything.exe
  
  
• In this key, create the following dword value, with the name:
  
  DumpCount
  
  Set the value data to:
  
  3
  
  
• Create the following dword value, with the name:
  
  DumpType
  
  Set the value data to:
  
  1
  
  (1 = mini crash dump)
  
• Create the following expandable string value, with the name:
  
  DumpFolder
  
  Set the value data to:
  
  %LOCALAPPDATA%\Everything
  
  (or some writable location)
  
• Wait for Everything to crash
• Please upload the .dmp file from %LOCALAPPDATA%\Everything in a bug report.

In the time being, please monitor RAM usage and handle usage in the Task Manager.

What are you indexing with Everything?
Please upload your Help -> Troubleshooting information in a bug report.

I do not get any crash dialog. I think it's running as a service (services.msc shows "Everything (1.5a)" as running, and the GUI is closed (but there is the magnifying glass icon in the tray)).

I went ahead and enabled mini crash dumps using your instructions. I will report back the next time this occurs. Thanks.

[JohnTravolski]

It crashed again last night. I filed a bug report. I had to attach the minidump via google drive because it's too big (it's 19.4 MB but the bug report is limited to 10 MB). I'm indexing three local NTFS drives, a file list (734 MB in size), and another machine. Regarding the troubleshooting information:

Code: Select all

Everything 1.5.0.1404a (x64)
OS:	Windows NT 10.0 19044 (x64)
Admin:	0
Service:	1 (connected / installed and running)
Command line:	-is-relaunch-command
Binary:	C:\Program Files\Everything 1.5a\Everything.exe
Profile:	C:\Users\John\AppData\Roaming\Everything\Everything-1.5a.ini
Database:	C:\Users\John\AppData\Local\Everything\Everything-1.5a.db
Instance:	1.5a

Config:
open_file_command2=$exec("C:\Include\John_Scripts\AutoHotkey\SingleUseScripts\7z_navigate_to_file.ahk" "%1")
auto_move_ntfs_volumes=0
auto_move_refs_volumes=0
auto_move_fat_volumes=0
filelist_hide_reconstructed_folders=0
exclude_files=~$*.xlsx;~$*.xlsm
include_file_content=1
content_include_only_files=*.doc;*.docx;*.pdf;*.txt;*.xls;*.xlsx;*.xlsm;*.py;*.pyw;*.ipynb;*.ps1;*.psm1;*.bat;*.reg
content_max_size=10
journal_max_size=33554432
filter=Not In Archives
columns=[{"name":"Name","width":517},{"name":"Path","width":553},{"name":"Size","width":82},{"name":"Date Modified","width":114}]
ntfs_volumes=[{"path":"C:","volume_name":"\\\\?\\Volume{24}"},{"path":"D:","volume_name":"\\\\?\\Volume{ed}"},{"path":"F:","volume_name":"\\\\?\\Volume{c1}"}]
folders=[{"path":"\\\\the-server\\c\\Server"}]
filelists=[{"filename":"C:\\Users\\John\\AppData\\Roaming\\Everything\\John_archive_filelist.efu"}]
exclude_folders="C:\\$Recycle.Bin","C:\\Users\\John\\AppData\\Roaming\\Code\\User\\History","C:\\Users\\John\\AppData\\Roaming\\Microsoft\\Excel","C:\\Users\\John\\AppData\\Roaming\\Microsoft\\Office\\Recent","D:\\$RECYCLE.BIN","F:\\$RECYCLE.BIN"

Policy:

Plugins:
Everything Server 1.0.1.2 (everything_server64.dll)

I did notice that the crash happened shortly after my nightly filelist indexing operation completed. I have that scheduled to run at 3:50 AM and it takes about 8 minutes to complete. The minidump file was created at 3:59 AM. Maybe it has something to do with me writing to that file while Everything is still running. If I notice this pattern repeatedly, that would make sense. However, since Everything doesn't crash every night, this doesn't appear to be a guaranteed trigger.

In Python, I write to the file like this:

Code: Select all

print('Writing now...')
with open(archive_filelist, 'w', encoding='utf-8') as text_out:
  text_out.write('Filename,Size,Date Modified,Date Created,Attributes\n')
  for line in lines_to_write:
    text_out.write(line + '\n')

Perhaps I should write to a different file first rather than overwrite the existing one, and then after the write is finished, then delete the old one and rename the new one to the old name. Not sure but maybe that would fix it.

[void]

Thank you for the crash dump and troubleshooting information.

The dump shows the NTFS monitoring state is corrupt.

I am investigating the issue.
It looks like the state could be corrupted if the service gets unexpectedly disconnected.

Is the Everything Service running when Everything crashes?

[JohnTravolski]

Thank you for the crash dump and troubleshooting information.

The dump shows the NTFS monitoring state is corrupt.

I am investigating the issue.
It looks like the state could be corrupted if the service gets unexpectedly disconnected.

Is the Everything Service running when Everything crashes?

I did check services.msc after it had crashed (and before I had started Everything.exe back up again), and it did say that it was still running, so I believe so.

[void]

Thank you for the Everything Service information JohnTravolski,

I am doing some tests with filelists my end.

In the meantime, please try disabling content indexing and see if the issue persists.

[JohnTravolski]

I wanted to mention that I changed how I wrote my filelist write code. I changed it from:

Code: Select all

with open(archive_filelist, 'w', encoding='utf-8') as text_out:
  text_out.write('Filename,Size,Date Modified,Date Created,Attributes\n')
  for line in lines_to_write:
    text_out.write(line + '\n')

to this:

Code: Select all

with open(archive_filelist_temp, 'w', encoding='utf-8') as text_out:
    text_out.write('Filename,Size,Date Modified,Date Created,Attributes\n')
    for line in lines_to_write:
      text_out.write(line + '\n')
os.remove(archive_filelist)
os.rename(archive_filelist_temp, archive_filelist)

And I've not had a single crash since. I'm guessing the crashes were related to the write taking a while since the file is so big (700+ MB). I could be speaking too soon, though, so I'll keep monitoring this. I have not disabled content indexing, by the way.

[void]

Everything 1.5.0.1405a fixes a crash when disconnected from the Everything Service.

Everything 1.5.0.1405a improves handling of loading file lists that are opened for writing.
Everything will now delay loading the file lists for up to one second after any writes are made to the file list.