Guide for auto-compare of hash files and hash properties

[rajah]

This is a guide to accomplish a specific task and not a question for anybody to respond to.

However, if anybody finds any errors I've made or suggestions for easier ways to do things then please feel free to comment!
I used Everything Version 1.5.0.1380a (x64) for this guide.

I've had more than one HDD die or have corrupted data after sitting in a drawer for some time. (Yeah, time flies - it turned out to be more like 12 years they were sitting there. Sigh) So I figured it was time to 'refresh' my stored data every once in a while to keep it intact. Below is my plan and hopefully someone else can find some of this useful.

I have managed to put together a procedure for checking the integrity of the data stored on an external drive that I plan on removing and placing into storage at another location (in case of theft or a disaster such as fire or electrical surge). I figure that every so often (maybe every few years) I can retrieve the drive and copy the contents to another drive. I can then repeat the procedure outlined below to validate the data using SHA-256 hashes.

I'm posting this guide because I had a difficult time piecing together exactly how to accomplish this and had I found a post with a step-by-step example like this I could have modified it to suit my needs much more easily than figuring it out from scratch.

Please take note: Generating SHA-256 hashes on large amounts of data is very time-consuming (as in possibly many hours). And you will have to generate the hashes twice for this procedure initially. Later you will only need to re-generate them one time.

The following steps are an example and you will need to modify as necessary for your setup and needs.

conditions for the example:
I have an external archive drive that is currently mounted as drive D:
I want to use a SHA-256 hash type.
I don't normally index on the SHA-256 property or generate hash files for my normal computer usage so I will be removing the properties that were added during this procedure from my indexing settings after verifying the external drive and placing it in storage.

Steps to follow:
1. use voidhash.exe to create the .sha256 hash files for the entire archive using the following command and wait for it to complete:
voidhash.exe -sha256 D:\

2. In the everything Tools->Options dialog box, click on Indexes->Properties then do the following:
Click the 'Add...' button and add the SHA-256 property.
Set "Include only folders:" field to "D:\" (without the quotes).
Set "Exclude files:" to "*.sha256" (again without the quotes).
Check the 'Fast sort' box. Don't close the dialog box yet.

3. While still in the dialog box from step 2:
Click on the 'Add...' button again and choose the property "sha256sum SHA-256".
Check Fast Sort on this property.
Click 'Apply' and 'Ok' to close the dialog box.
Wait until indexing of properties finishes. Could be several hours depending on amount of data.
There will be an Indexing Properties: progress bar on the status bar. You can hover your mouse over it to see what file it is working on.

4. Make sure that you have Folders turned on (make sure view->folders is checked) and then click on just the archival drive (D: in my case)
because you only want to see and check files on that drive.

5. In the search results list area:
Right-click on any column header and add the "SHA-256" and the "sha256sum SHA-256" columns to the result list.
Enter the following search in the search area:
!folder: !*.sha256
Then make a note of the number of items from the bottom left corner of the window.
Then change the search to:
!folder: !*.sha256 sha256:==sha256sumsha256:
Assuming everything was done properly, the new number of items in the search result list should match the noted number of items.
Now for the hard-earned result. Change the '==' to '!=' in the search so it looks like this:
!folder: !*.sha256 sha256:!=sha256sumsha256:
Hopefully, this gives you a result list with zero items. This would indicate that all pre-generated hashes match the generated hashes.

6. Do a safe removal of the storage device. If the system keeps complaining that some process is using a file on the drive, you may need to do a normal computer shutdown and remove the drive while the computer is off. Store the drive in a safe place off-site.

7. Now get back into Everything and remove the 2 property indexes that were added in steps 2 and 3 (assuming you don't index these properties normally)

8. When you decide to 'refresh' the data (maybe a few years down the road) then (after cloning or copying the drive to a new drive) you just need to follow steps 2-7 as the generated .sha256 files will all still be on the drive and can be used to compare with the newly generated hashes in step 3.

Note: If you later want to add files to the drive, you should follow steps 1 and 6 at a minimum. voidhash.exe will not overwrite any previously created hash files and will only create ones for the new data. Do steps 1-6 for a full verify if you are of a more cautious type.

[omHSTa]

Thank You Mate, learned a lot from Your Post ♥

[Herkules97]

Could one use the index property itself, export all of it as efu then compare the file lists?
As in skip making all the sha256 files, save disk write.
It still has to read everything again maybe. One could potentially save even more disk read if using NTFS..Presuming you can put it off for a year, then come back and if you run the same instance it'll find anything that's different when it comes to SHA-256?
Unless bit rot or whathaveyou won't show up like that..
Presumably if you instead index as folders, you can rescan the folder and it'll only re-index sha-256 for files that it finds to be different? Might still need to re-read all the files, but not re-read all the sha-256 values thus saving disk read?

Say you make at least one file list, then you make a temporary instance and another file list after a year..Then rename the device for the old one within the efu(If everything starts with C:, replace-all to A: or whatever), then add both into the database and dupe-from whichever is the old disk? If file count is the same, nothing is changed. I presume the dupe you have to use is sha-256dupe: or dupe:sha-256 whichever you prefer. "dupe-from:A: dupe:sha-256 !folder:" or some such.
Would that save more on disk read and "infinitely" more on disk write as you never write anything additional? I don't really use hashes, I just hope everything still works heh..I think it would very impractical to attempt anyway, especially if files can corrupt while the devices are still plugged in. Then you'd have a sha-256 that already is the post-ruined state.

So far any file I've wanted to read has read since back to 2016 when I began using external HDDs proper. I've done nothing to keep these HDDs working properly.
Maybe there was one questionable time I ran a file, but IIRC I soon after found out it was broken from the start so something else probably happened to it.

If I DID index SHA-256 or similar, I'd be able to more easily find uniques among HDDs that still work..I think I tried CRC-64 but it was just too heavy.
Maybe if there was a way to delay hashing. Typically a file from 2 days ago won't change on day 3. So if I could tell the SHA-256 property to delay indexing for 48 hours, maybe it'd work much better. Maybe I should make a post requesting this. Maybe it'd require a new db version and void would wait for client v1.6 to add it.

Google AI Summary(via viewtopic.php?t=15816) says
property_delay or read_properties_delay. which are in milliseconds. Idk if that could be used if the milliseconds go up to 48 hours.
It should be 172800000 which seems to work but that's for ALL properties.
I'll make a new post

[therube]

Click on the 'Add...' button again and choose the property "sha256sum SHA-256".

I'm missing something, I don't see that?
Well, I certainly missed the [A]dvanced checkbox. But still not seeing that.

Is it, 'Sidecar SHA-256' ?
Must be, no?

Oh, & so 'Sidecar SHA-256' physically reads the (voidhash) generated .sha256.
And the [Everything.exe] compares, compare the (Everything generated) SHA-256 values against those (gathered from) the (voidhash generated) .sha256 file. (And that then also answers my question below.)


And how is voidhash.exe playing into the procedure you're describing (other then generating & saving the hashes)?
(I'd think you'd want to also backup the saved .sha256 on different devices in case it should corrupt.)

[therube]

Aside from using Everything to do the later validation, you could use any program that can validate .sha256 files.

(Any program*.
A file manager, if supported, CHK hash program, sha256sum.exe -c [to some extent]...
Other programs could have limitations, LFN, UTF-8,... but in general should work.)

[therube]

If you later want to add files to the drive, you should follow steps 1 and 6 at a minimum. voidhash.exe will not overwrite any previously created hash files and will only create ones for the new data.

Note that voidhash does not UPDATE an existing .sha256, so neither added nor removed files will be accounted for.

If there were newly created directories, with no pre-existing .sha256 file within, it would (newly) create .sha256 files for them.


In order to update, you would have to first remove existing .sha256...
(Maybe even rename, recreate, then compare renamed against new?)

[therube]

I have a (known) bad flash drive (that I typically don't use) but I happened to use it yesterday.
Source files, were on C:, & I copied them & a xxHash "sidecar" to the flash drive.

Then when I copied the files off (onto a different computer, sneaker-net), I re-compared the files, against the sidecar, & sure enough, some were corrupted.

Code: Select all

generate:
xxhsum.exe * > xxx.xxh64

validate:
xxhsum.exe -c xxx.xxh64

gui-validate
CHK.exe xxx.xxh64
(or drag xxx.xxh64 into CHK.exe)

(I'll use the same procedure later today to copy/validate the files that failed the first time - which may or may not work - but at least I'll know.)