[Workaround] Content index 70GB+ txts?

[Herkules97]

I have files I want to search within over and over.
I use Agent Ransack when searching one-off stuff, but I was thinking of EBV for long-term repeated searching..
I am currently using Agent Ransack to do the search I'd want with EBV as I am writing this but this will only be done once I think. Unless the way to go is to search for all relevant lines and then make a new document with only those..As long as it takes less than 20GB to read I can run that with Notepad++ 64-bit.
For now I only want .inis from MPC-HC which start each relevant line with something I can search for. Same can't be said about non-mpc-hc.ini files
But the issue is content indexing would surely take at least 30GB in db file size and all 70GB+ in RAM?
I have neither of these to spare. Plus any time I add more I presume it'd go above 100GB in RAM to do all the rebuilding stages.
Is there a way to compress both to rely on CPU for operations or do I need a different program entirely that maybe compresses it all and then searches that compressed database? If the db for it all takes 10GB or less and less than 20GB commit it should be fine to run from time to time.
There are at least 150K files totalling these 70GB+(MPC-HC .inis) but I might even want to go all the way which would be 353K files totalling 162GB(Any files I have in the same root folder, should mostly if not all be plain text so should compress well).

I had my eyes on docfetcher some years ago but I couldn't figure out how to make it work. Not that I know if it achieves the desired end result anyway.

[void]

Everything isn't designed to index 70GB of text.

I recommend storing your content on a very fast NVMe SSD.
Let Windows naturally cache the content.

Search the content from disk.
The first search will be a little slow, but subsequent searches will be much faster.

But the issue is content indexing would surely take at least 30GB in db file size and all 70GB+ in RAM?

70GB db
70GB in RAM.

Plus any time I add more I presume it'd go above 100GB in RAM to do all the rebuilding stages.

Correct, expect a rebuild to use 140GB of RAM.

Is there a way to compress both to rely on CPU for operations or do I need a different program entirely that maybe compresses it all and then searches that compressed database?

No, there's no compressed database option.
I explored compression and it completely kills performance to halve RAM usage.
You're better off searching from disk or Windows file cache.

[Herkules97]

So I turned the search service on and got it indexing a small folder..
I tried searching si: with my Monolith instance and it found results immediately. (Monolith 1423b, editing so it's easier to know)
But the instance that only targets the C:\ device finds nothing? (C OS 1386a)
Total amount Monolith found were 44 files/folders. It also found a specific word in the files that had it so it seems to work.
Using Monolith to search isn't a big deal, especially if as I mentioned below, shell_execute_thread, works as I hope it does. Monolith can fail to index extra properties and when running a file when it's in that state, it will freeze indefinitely. I only save Monolith manually due to its size. Auto-saving means it will stop indexing and not retroactively add back at least to the journal and also I often don't have the space to save it casually. If a new thread freezes instead of Monolith itself then I can still exit it and not have to concern myself with knowing if it's in the basic-properties-only state or not and thus not lose any data and also avoid rebuilds.
Anyway, the indexing window itself says 1002 files indexed. Hmm..

My C OS instance is 1386a, does si: only work properly in newer versions? Monolith is currently 1423b as I updated it today for shell_execute_thread.
Si: is in the search syntax window, but maybe there was a bug with 1386a..

Debug log part where I searched, not that it helps.
C OS

2026-09-22 03:51:19.096: search 'si:' filter '' sort 2 ascending 0
2026-09-22 03:51:19.096: FOLDER TERM START 0000000085eb90f8 M 000000000037e880 N 000000000037e630
2026-09-22 03:51:19.096: 0000000085eb90f8 20e01100 M 000000000037e880 N 000000000037e630 OP 509
2026-09-22 03:51:19.096: FILE TERM START 0000000085eb90f8 M 000000000037e880 N 000000000037e630
2026-09-22 03:51:19.096: 0000000085eb90f8 20e01100 M 000000000037e880 N 000000000037e630 OP 509
2026-09-22 03:51:19.096: same results
2026-09-22 03:51:19.100: count 0
2026-09-22 03:51:19.100: new results 0
2026-09-22 03:51:19.100: update index C:

Monolith

2026-09-22 03:53:18.598: search 'si:' filter '' sort 5 ascending 0
2026-09-22 03:53:18.600: FOLDER TERM START 00000005ca63c218 RETM 00000000006fe3e0 RETN 00000000006fe2b0
2026-09-22 03:53:18.600: 00000005ca63c218 20e01100 M 00000000006fe3e0 N 00000000006fe2b0 OP 509
2026-09-22 03:53:18.600: FILE TERM START 00000005ca63c218 RETM 00000000006fe3e0 RETN 00000000006fe2b0
2026-09-22 03:53:18.600: 00000005ca63c218 20e01100 M 00000000006fe3e0 N 00000000006fe2b0 OP 509
2026-09-22 03:53:18.600: same results
2026-09-22 03:53:18.609: new results 44
2026-09-22 03:53:18.609: update index C:

Spent too much time re-writing, editing this message so I'm sending it..If I've missed something I'll just write it in a reply.

[void]

Please check the results returned from the windows index:

• Search for:
  c: si:
• In Everything, from the Tools menu, under the Debug submenu, check Verbose.
• In Everything, from the Tools menu, under the Debug submenu, click Start Debug Logging.
• Press F5.
• In Everything, from the Tools menu, under the Debug submenu, click Stop Debug Logging.
• This will open your %TEMP%\Everything Debug Log.txt
• Please upload this file in a bug report.

[void]

Thanks for the logs.

si: is broken in Everything 1388a and earlier for non-English languages.
1389a fixes these issues.

si:

without a parameter also does nothing in Everything 1389a and earlier.
This was fixed in 1390a.

[Herkules97]

I'm a sucker for indexable content for search engines, so I will reply here instead of mail..I was writing in Outlook before realising I shouldn't.
The relevant mail reply to my bug report:

Hi Herkules97,

Thank you for the logs.

si: is broken in Everything 1388a and earlier for non-English languages.
1389a uses a newer method (that isn't translated) to parse result items.

si:
without a parameter also does nothing in Everything 1389a and earlier.
This was fixed in 1390a.

Does the following return any results:
c: si:*



There's also way more debug logging information in the later versions.
It will be difficult to get this going on 1386a. You'll need to update to
1390a or later.

If I made it seem like I want to stick with 1386a but have this bug fixed, that was not the intent.
I can update just fine, really I just wanted to know if it IS a bug with 1386a(Thus if I can update to fix it, rather than it being a corruption with the db or whatnot). If you knew beforehand, I don't think you needed to read any debug logs as "yes" would've sufficed
I can't check si:* right now on the original instance as I accidentally sent the entire debug log into the search bar and the window has been not responding for at least 45min .

Hopefully it's working in the background and can eventually unfreeze so I don't lose any index journal entries' original capture. I am running a barebones secondary instance that's only going to be used to continue from a 3 hour old version of the db in case I have to force-exit the main one. If so I'll move the secondary's db into the main C OS instance and replace the old db and then finally try si:* before updating to 1423b. If you still want that result, Idk what the point is for a much older build.

[Xavierarmand]

Unrelated to EV or windows.SI... have you tried Grep?

I use dnGrep daily. https://github.com/dnGrep/dnGrep Love it. It can use EV SDK in the back ground it filter files to search inside of. I have throw 40+ gig for at it, with no filter. It will search most everything, binary and text, compressed etc.

It's become my #1 goto context searching tool before I try anything else.

X

[Herkules97]

I don't know if I wrote it anywhere outside of an unsent mail, but I decided to manually go through and find any inis I thought I should search for.
This turns it from 70.4GB to I think less than 50MB heh. I still haven't tried if I can add all the individual files to a new instance for content indexing. But sometimes when adding large chunks, I will modify the ini directly so maybe that's a path if the wildcard filters work for adding specific files.
Benefit of content indexing directly in EBV surely is that it can always be given to other systems. Probably isn't a way to copy over the Windows Search cache to a new system. It'd probably reject it..or maybe it could rebuild it, Idk.

Either way I do have other files, so maybe this Grep can be used for that or I change the location of the Windows Search cache to another SSD. I don't have any other that's via SATA/M.2 though so it would be slower..but I only have 500GB on OS SSD and it's often near 15GB.

And for si:*, I finally gave up and force-exited the instance and moved over the temporary panic database and this search did nothing.
I think even if I update it to 1423b, I'll use Monolith. This makes more sense given that indexing will take place across multiple devices and Monolith is the instance with all of the devices I've plugged in since 2024 or whenever. Unless I decide to only use EBV content indexing in which case it will be its own instance.

[Xavierarmand]

Stick with the grep. It's power sadly overlooked or forgotten these days. I only discovered it myself a couple years ago.

There are multiple programs that can do it going back to the dos days.

Dngrep has the best bells and whistles that I've found. Exstsive display options, syntax highlighting, hot keys, robost cli, you can cache and store certain binary files like PDFs or office docs for faster searching the 2nd time around., etc etc.

the devs on GitHub are involved. I've posted issues and requests, all have been tended to.

I run dnGrep portablely between 3 computers with 5 different os installs. One app for all of them and 95% of the time it finds what I need, if it there.

Yes throwing a 40g folder on an HDD might take 15 mins to search everything but it's less overhead on your CPU than windows system Index take every time you turn your computer on. Haha. I know it when windows is scrapping my system in the background.

[phil2search]

Hello,

I don't know if you are still looking for something but recoll should work.
I use it for hundreds of GB including 75+GB of documents (60k+ pdf, office, txt, code...).
My index is about 12GB. It has been very reliable for me.
Docfetcher did not work well for me in the past for that.

Recoll can scale to much bigger sizes than you seem to need (https://www.recoll.org/usermanual/webhe ... E.BIG.html)
It's open source (you can compile it yourself) or the author asks for a small donation (starting at 5 euros) for the windows executable.

Good luck
Philippe

PS: I am not related to recoll in any way. Just a nice tool to use along everything when filenames are not enough.

[Herkules97]

I tried Windows Search but at some point it decided around 50K files to restart the entire thing. It was bad enough that adding new filetypes also caused a reset, but then even when I don't touch it it'll still reset in the background..

Seems for some Windows stuff, they just like to undo themselves randomly. Same with the Windows 10 start menu.
Windows 10 Start Menu pins weren't seemingly exportable and importable cause of course system-tied stuff has to be more difficult for no good reason and in an update decided to break half of them.
I started using lnks, bats and cmds and it can't fail. Plus I get run history for all but the 2 EBV instances that runs everything else..They have zero history as I have to run them from Explorer. One admin, one non-admin for all other user-run programs.
So it seems I will yet again have to rely on other methods.

dnGrep or Recoll could be used, anything will probably keep an index better than Windows Search.
Not that dnGrep keeps an index, just..anything third-party will probably work better.

I do have an EBV instance that I intended for content indexing and any searches related to that for text stuff.
But I can't keep much before it goes out of hand due to it needing to index the whole files.
I was ready to lose about 25GB to Windows Search so it doesn't matter to me if it takes about that much for Recoll or similar.
I just hope whatever I can find can add to the index, append I guess, and not reset on additions nor reset randomly.

Maybe one day EBV can itself support something like Windows Search type indexing, but not decide to randomly reset. Presumably will be a lot of work for void..Idk coding, he already has so much else to add..Presumably, I think the todo is outdated IIRC but he probably has his internal todo.

Maybe I even find something third-party that integrates to EBV so I can still use it to search. I prefer searching using EBV.
For example with Agent Ransack it seems to decide not to search stuff and Idk why. That one PDF setting can be on or off, other stuff can be on or off..Just refuses. Plus I couldn't search the folder anyway as it 100% commits. I forgot that it can bloat like that on larger folders.
EBV simply works for what it supports.