Memory leak during indexing (Windows 10, dynamic cache, metafile, ReFS)

Found a bug in "Everything"? report it here
Post Reply
roBsonn
Posts: 16
Joined: Wed Sep 06, 2017 4:36 am

Memory leak during indexing (Windows 10, dynamic cache, metafile, ReFS)

Post by roBsonn »

I've a Windows 10 and have encountered a serious problem during indexing my ReFS partition (I don't know if this happens also with NTFS). I spotted a huge increase of memory usage while reindex is in progress. I diagnosed this problem and came to these conclusions:
  • 1. It is not Everything's fault. It's Windows 10's dynamic cache used for file IO that gets bloated (link, link). In short, if there's a lot of quick/frequent IO demand, Windows caches it in RAM. The demand can be caused by, of course, Everything's full partition reindex, but also as simple as selecting all folders on your root partition and invoking Properties. Even if it seems like Windows problem, I still decided to report this here, because I hope Everything code can be upgraded to overcome this issue.
    2017.09.23 12.48 - Leak, Reindex Stop.png
    2017.09.23 12.48 - Leak, Reindex Stop.png (39.15 KiB) Viewed 4800 times
    2. The memory section where the leak is happening is metafile section, which is somewhat invisible in Windows Task Manager: It won't show in any column in Processes nor Details tab, but it's a part of the total In use bytes charted on Performance tab. Because of that you can see a 100% memory usage, but if you sum the memory column on Processes tab, it won't match (by many GiB). The missing part is the metafile section. which you can see in Performance Monitor by creating a counter for System Cache Residual Bytes (not Process Private Bytes, where usual memory leaks happen). You can also (and easier) see it using RAMMap app from Windows Internals (link). The problem is clearly visible on the screenshot I attached:
    2017.09.23 12.59 - Leak, Before Emptying system working set.png
    2017.09.23 12.59 - Leak, Before Emptying system working set.png (26.4 KiB) Viewed 4800 times
    3. When I force emptying of this section (Empty > Empty System working set in RAMMap), the memory is released (or rather, flagged as inactive, but Task Manager can see this RAM as now free):
    2017.09.23 12.01 - Leak, After Emptying system working set + Chart.png
    2017.09.23 12.01 - Leak, After Emptying system working set + Chart.png (61.11 KiB) Viewed 4800 times
My guesses are that ReFS partition or Windows 10's kernel are at fault. If you could diagnose this problem deeper and present a workaround in Everything, that would be great. More screenshots and logs for this issue are available at my Dropbox (couldn't attach more than 3 files to this post).
Last edited by roBsonn on Mon Sep 25, 2017 10:27 am, edited 1 time in total.
void
Developer
Posts: 15096
Joined: Fri Oct 16, 2009 11:31 pm

Re: Memory leak during indexing (Windows 10, dynamic cache, metafile, ReFS)

Post by void »

Thanks for dropbox screenshots and bringing this issue to my attention.

I'll look into a fix for Everything 1.5.
roBsonn
Posts: 16
Joined: Wed Sep 06, 2017 4:36 am

Re: Memory leak during indexing (Windows 10, dynamic cache, metafile, ReFS)

Post by roBsonn »

I must report a great discovery in this topic. Information from this Microsoft support page resolved the memory leak issue for me. The used RAM still increases during reindex, but now it is being freed (in batches, at random moments). They suggested three different keys to add to registry, I added only the first two (as they said the third is most aggressive and can lead to diminishing performance) and since then the problem is actually gone. I suspect even the first option might be enough (because the second is just a "rate" at which RAM space is reclaimed).

So, maybe Everything installer could check if user has these registry keys and if they don't, then add it? I shall leave the decisions to you, the author, whom I am grateful for this software. I am hardly productive without Everything anymore, for a couple of years now - thank you!
Post Reply