[Workaround-ish] Delay a specific indexed property up to at least 48 hours?

[Herkules97]

As per viewtopic.php?p=83710#p83710
To repeat, what if you want to index SHA-256 but doing it at all times is very expensive on HDDs and they might chug more than usual.
If you could delay it to say 48 hours for only SHA-256 indexing, then it's unlikely those files will be modified on day 3 and beyond and the hash remains.
If it's based on milliseconds like the below one, you could of course delay less or more.

"property_read_on_change_delay" accomplishes something similar but for all properties and then some.
It does seem to go up to 48 hours in milliseconds, presuming 172,800,000 = 48 hours.

[void]

property_read_on_change_delay works for all properties, icons, and thumbnails.
There's no option to delay a specific property.



property_read_on_change_delay is in milliseconds.
172800000 is 48 hours.



Avoid indexing sha-256 in Everything.
Use hash sidecar files instead.

[Herkules97]

I do not want at least 72mil extra files to make that useable.

I imagined there would be a delay setting in every property page, where you can select max sizes and exclusions or inclusions currently.
Maybe one day.

Only other path I can think of that would be viable is if folders that I know gets files constantly modified are excluded. Like the entirety of C:\ as that rarely has any copies I want to find from anyway.
There might be lots of folders needing to be found and excluded, but once it's setup it could be fine.
Maybe that's a better system even if delaying a property was a thing.
Maybe a combination of excluding such folders and increasing the global delay to 20 seconds could work even better.
Only one way to find out..

[void]

What about pausing property indexing under Index -> Pause Property Indexing?

There's also .metadata.efu
Store all your hashes in a single .metadata.efu

[Herkules97]

I don't see how entirely pausing property indexing would be an option, then I wouldn't have any of them at all? The point is to have properties, except hashes which get delayed so only files that mostly remain the same don't update and stuff that constantly changes like on the system storage device don't get constantly read from. Thus saving constant disk read which can help when trying to do other stuff. Especially for struggling HDDs.

Metadata.efu would lower the file amount, but it would still be extra files in locations I wouldn't want.

Adding it to Monolith would still be better in this case. It would definitely be a lot more expensive on RAM. Presuming it's 64 bytes per SHA-256 entry.
If I manage to not find more than a few million to exclude from SHA-256, it could be 32-33GB at base when it's currently 26GB base.
I probably won't bother testing how it works on a smaller instance and instead go straight to the largest one(lazy).

If Monolith requires a rebuild from too many changes while it was restarting, is it USN Journal size and delta that controls that? Just delta?

Best I found about what sizes you should use is viewtopic.php?t=14486

is "too big" a thing?

For Maximum size: No

For Allocation delta: Yes
Using a really large allocation delta may cause the NTFS driver to become unresponsive while the USN Journal is trimmed after it is filled.

Please keep the allocation delta around 8192 KB.

One HDD has been slow recently and causes Monolith to take longer. Say I have it at 8MB and it's 3min before it finishes and just says "Ready" in status bar..Would 16MB make it 5-6min? If everything else stays the same.
Could this state itself cause it to rebuild from too many changes and get into a perpetual cycle of rebuilding?
This might be much more relevant as Monolith goes into 30GB+ base, as it will only take longer to exit.
So far I think the only times it has needed to rebuild on restart is when I forgot I have processes that are causing unusually many changes. Maybe it'll be fine with the extra time taken to exit at 33GB base at 8MB delta.

Another issue is that Monolith can get stuck and stop indexing extra properties. Not sure if USN Journal is at fault and if increasing delta or the total size could increase the likelihood it stops at any given time. Maybe not because of the journal, but because of EBV reading the journal as basic properties don't fail when this happens. So it's probably an EBV fault, dunno if it's tied to the journal.
I think once a single-device instance has failed in this way but that was probably with hundreds of thousands of new files in a few hours like a robocopy of an entire other HDD.
I think I've asked about Monolith getting stuck before, but as usual that's all gone from my memory..Or at least accessible memory.

[void]

I don't see how entirely pausing property indexing would be an option, then I wouldn't have any of them at all?

I was thinking you could pause when your using the PC, then unpause when your not at the PC, eg: unpause over night.

If Monolith requires a rebuild from too many changes while it was restarting, is it USN Journal size and delta that controls that? Just delta?

Just the USN Journal Size.
The delta is just how much is deleted at a time from the start of the log when it is filled.

One HDD has been slow recently and causes Monolith to take longer. Say I have it at 8MB and it's 3min before it finishes and just says "Ready" in status bar..Would 16MB make it 5-6min? If everything else stays the same.

It really depends on how much has changed, but yes, if it takes 3 mins to process 8MB of USN Journal changes, expect 6 minutes to process 16MB of changes.

Could this state itself cause it to rebuild from too many changes and get into a perpetual cycle of rebuilding?

Everything will rebuild if a required journal entry is deleted.
Yes, it's possible for Everything to index forever if it can't keep up with processing changes.

[Herkules97]

I was thinking you could pause when your using the PC, then unpause when your not at the PC, eg: unpause over night.

Originally wrote lines about maybe doing this, but then I realised..That's not even the issue.
I see, it wouldn't fix the issue of constantly reading files though.
I don't mind it running during the day. I don't see any method that's better than pausing individual properties for x number of milliseconds and secondly to exclude folders that should have lots of file changes to existing files that would cause it to constantly want to re-read hashes of them.

Of course the second one is only the one I currently know is second best.
I somehow brainfarted and thought pausing during the day was better, hopefully my edit come before anyone reads the original and replies to that

Also while sleeping I can't restart Monolith a bunch and when there is a progress bar it seems to have a much greater chance of pausing indexing extra properties.
Idk how to fix that without a force rebuild if it even it would fix it.

Just the USN Journal Size.
The delta is just how much is deleted at a time from the start of the log when it is filled.

And since it can be pseudo-infinite, I can just make it 1GB and never have to worry about lost changes?
I would probably make it 200MB, not 1GB but asking for something much larger as that should mean smaller is even more stable or at least as stable(If it can become unstable).

[void]

Maybe an option could be made to pause all properties

There's no option to do that.
Everything will lazy load column values, so the easiest option is to just hide the columns or make them offscreen.

And since it can be pseudo-infinite, I can just make it 1GB and never have to worry about lost changes?

1 GB is fine, just leave the delta reasonable. (1MB - 8MB)
If you had a 1 GB delta, you might notice system hangs while the NTFS driver deletes a large chunk of the USN Journal.

[Herkules97]

Oops, didn't manage to edit my brainfart away in time.
Here is the original message read for the reply above, or at least one of the versions..Maybe this just makes it messy. Delete if so I guess

I was thinking you could pause when your using the PC, then unpause when your not at the PC, eg: unpause over night.

Hmm..That could be. I've never tried it and didn't have it on my mind.

If I do this, is there anything I can do to prevent Monolith from getting stuck on extra properties? I won't be able to restart it a bunch while I sleep to fix it.
It's when it fails all but the basic properties and if you try to launch from it, it'll freeze and require force-exiting or some method of unfreezing that I don't know about.
Maybe the new shell_execute_thread or whatever it was called helps against the launch part..I've never wanted to risk opening a file with it on over restarting Monolith so I have no idea if it works.
It has a much greater chance of pausing when it has a queue of extra properties to index, as in when there is a progress bar.

I just tried pausing for an instance that isn't one of the ones I awfully care about and confirmed 2 questions..(I was originally going to ask cause lazy.)
It only pauses extra properties so I still get datetime modified and such..
(Maybe an option could be made to pause all properties, though in my case I don't want to do that..But there could be situations where even basic properties should be pauseable..Or does that tie into the journal and would just be unfeasible to add..)

It also doesn't affect the journal side of things.
(I also take it the journal side CAN'T work like the main interface due to it being limited by the origin journal's size. Like pausing its property building wouldn't work cause things get removed over time.)

[Herkules97]

I finally did it and it finished while I was sleeping. All below are just statements. Including the ? ones, just rhetorical.
The first rebuild took maybe 5 hours, I had changed some property exclusions settings so after it was done with that it went straight to rebuilding again so I couldn't check its stats time.
Peak Private Bytes, which is probably close enough to commit, was 85.9GB. (Peak pre-SHA256 was 48GB or so)

After it was all done and I woke up, it was 45GB with a file size of only 13.3GB. I recall the property queue doesn't take extra space but thought SHA-256 would still increase it by a few gigs..It's only 4.4GB in RAM so maybe that's only around 450MB in file size?
12.8-13.2GB sounds about right for a fresh Monolith db before SHA-256. I tried checking but didn't go far back enough in time and lost interest.
I excluded 35,188,728 files out of currently 72,794,779 online ones. Total file count is 133,394,233.
I think the offline ones might still take RAM for the property even if Monolith hasn't touched them in months, placeholders. At least as I understand it.

I also set the property indexing to a 60 seconds delay from 1 second. This is with the OS device being indexed, but with some folder exclusions to help against often-modified files being hashed over and over. I also excluded lnk, cmd and .aja from SHA-256 entirely. .aja alone are 64,209,360 across online and offline. So that explains why SHA-256 is only 4.4GB instead of an expected 6GB if offline files are still kept in memory with placeholder empty values. 8GB total, 2GB excluded on online devices and 2GB excluded on offline.
There are surely more I can exclude, but I was doing this sort of in a rush cause I didn't want Monolith to rebuild unnecessarily.
I took long enough that it already finished its original rebuild before SHA-256, so I could've searched Monolith for more filetypes to exclude but didn't.

Unfortunately don't know the peak of the second rebuild, so Idk if the exclusions only affected the first or all rebuilds. If say the next rebuild only took 60GB peak or near.
Rebuild for the second one was 5h 52min 25s it seems.
With that time, it was probably 85GB again.
A rebuild before SHA-256 was about an hour IIRC.

A day's worth of journal entries on the OS device is about 200MB, to be safe I increased it to 800MB journal size and smaller sizes for other devices.
Now it shouldn't rebuild ever..Even if it does it might take similar times to rebuild which gives me plenty of time to exit stuff in preparation.

When I woke up it was 15% done at the progress bar and I read online that with newer chips that SHA-256 is hardware-accelerated.
Chip I have might be not so great for much, but it seems that doesn't matter if there is hardware acceleration
15% accounted for 5GB it seems, as I restarted Monolith to save all the hours it just spent and it was 40GB PB after.
But maybe there were some other accumulation that was freed from that too..

This is similar to me avoiding 1.5x because I thought it would be unstable and fail too much. I waited way too long to do something.
Having SHA-256 should've been something I tried much sooner just to at least find out how it works.
I held off because of that time I did CRC-64 for one of the devices. SHA-256 is probably faster..Either way I also never did any excluding for it.
The point is to de-dup from SHA256 or otherwise find duplicates with only one property. I don't therefore need SHA-256 for everything.
Also when I tried CRC-64 I may have had 64GB sticks memory total..This Monolith rebuilding took at peak from what I've recorded at 83GB PB, 64GB WS. So presuming I had similar amount of files back then, I couldn't finish it anyway..Unless Monolith would've tried to work around lack of sticks memory.