Memory leak during indexing (Windows 10, dynamic cache, metafile, ReFS)

Found a bug in "Everything"? report it here
Posts: 15
Joined: Wed Sep 06, 2017 4:36 am

Memory leak during indexing (Windows 10, dynamic cache, metafile, ReFS)

Postby roBsonn » Sat Sep 23, 2017 11:27 am

I've a Windows 10 and have encountered a serious problem during indexing my ReFS partition (I don't know if this happens also with NTFS). I spotted a huge increase of memory usage while reindex is in progress. I diagnosed this problem and came to these conclusions:

    1. It is not Everything's fault. It's Windows 10's dynamic cache used for file IO that gets bloated (link, link). In short, if there's a lot of quick/frequent IO demand, Windows caches it in RAM. The demand can be caused by, of course, Everything's full partition reindex, but also as simple as selecting all folders on your root partition and invoking Properties. Even if it seems like Windows problem, I still decided to report this here, because I hope Everything code can be upgraded to overcome this issue.
    2017.09.23 12.48 - Leak, Reindex Stop.png

    2. The memory section where the leak is happening is metafile section, which is somewhat invisible in Windows Task Manager: It won't show in any column in Processes nor Details tab, but it's a part of the total In use bytes charted on Performance tab. Because of that you can see a 100% memory usage, but if you sum the memory column on Processes tab, it won't match (by many GiB). The missing part is the metafile section. which you can see in Performance Monitor by creating a counter for System Cache Residual Bytes (not Process Private Bytes, where usual memory leaks happen). You can also (and easier) see it using RAMMap app from Windows Internals (link). The problem is clearly visible on the screenshot I attached:
    2017.09.23 12.59 - Leak, Before Emptying system working set.png

    3. When I force emptying of this section (Empty > Empty System working set in RAMMap), the memory is released (or rather, flagged as inactive, but Task Manager can see this RAM as now free):
    2017.09.23 12.01 - Leak, After Emptying system working set + Chart.png

My guesses are that ReFS partition or Windows 10's kernel are at fault. If you could diagnose this problem deeper and present a workaround in Everything, that would be great. More screenshots and logs for this issue are available at my Dropbox (couldn't attach more than 3 files to this post).
You do not have the required permissions to view the files attached to this post.
Last edited by roBsonn on Mon Sep 25, 2017 10:27 am, edited 1 time in total.

Site Admin
Posts: 3631
Joined: Fri Oct 16, 2009 11:31 pm

Re: Memory leak during indexing (Windows 10, dynamic cache, metafile, ReFS)

Postby void » Mon Sep 25, 2017 8:06 am

Thanks for dropbox screenshots and bringing this issue to my attention.

I'll look into a fix for Everything 1.5.

Posts: 15
Joined: Wed Sep 06, 2017 4:36 am

Re: Memory leak during indexing (Windows 10, dynamic cache, metafile, ReFS)

Postby roBsonn » Thu Sep 28, 2017 9:36 am

I must report a great discovery in this topic. Information from this Microsoft support page resolved the memory leak issue for me. The used RAM still increases during reindex, but now it is being freed (in batches, at random moments). They suggested three different keys to add to registry, I added only the first two (as they said the third is most aggressive and can lead to diminishing performance) and since then the problem is actually gone. I suspect even the first option might be enough (because the second is just a "rate" at which RAM space is reclaimed).

So, maybe Everything installer could check if user has these registry keys and if they don't, then add it? I shall leave the decisions to you, the author, whom I am grateful for this software. I am hardly productive without Everything anymore, for a couple of years now - thank you!

Posts: 1
Joined: Tue Jan 02, 2018 9:36 am

Re: Memory leak during indexing (Windows 10, dynamic cache, metafile, ReFS)

Postby brianmanee » Tue Jan 02, 2018 9:39 am

A memory leak happens when there's a bug in the code that forgets to send that signal, so every time it needs that piece of info, it'll keep calling it back up, but the old pieces will never leave, so it will continue to use tons of memory (RAM) over time to the point of making whatever game or application you're using unusable.

Using Window's Resource Monitor

To find a memory leak, you’ve got to look at the system’s RAM usage. This can be accomplished in Windows by using the Resource Monitor.

In Windows 7:

Start > All Programs > Accessories > System Tools > Resource Monitor

In Windows 8.1/10:

Press Windows+R to open the Run dialog; enter "resmon" and click OK.

With Resource Monitor open, select the Memory tab.

Return to “Bug report”