Find duplicates in various folders and keep or delete only those present in specific folder/s.

[Turok]

I know how to find dupes. But I need some way to achieve what follows:

I have a few folders, let's say 3.
Folder A has 500 files and 100 duplicates.
Folder B has 300 files and 60 duplicates.
Folder C has 200 files and 50 duplicates.

These duplicates are both, part of dupes are only inside each folder and part dupes on all the three folders. For example, Folder B may have 60 dupes from which 50 are also in A, while the rest are only in B.
Folder A is the recent one, so it must have most (but not all) of the files that are also present in B and C.
So I need a way to keep all the files in A and delete dupes (only those that are also in A) from B and C.

Is there a quick way (not manually : ) to achieve this?

PS: I know how to find dupes but, is find dupes by size 100% precise? does it compare them in bytes?

[void]

Please try the following search:

dupe-from:"C:\FolderA" dupe:size

This will give you size duplicates where there is at least one duplicate from C:\FolderA

Sort by path and delete any files outside C:\FolderA



Please try the following in your search to restrict results to C:\FolderA, C:\FolderB and C:\FolderC:

"C:\FolderA\" | "C:\FolderB\" | "C:\FolderC\" dupe-from:"C:\FolderA" dupe:size

dupe:size will give you instant results.
Use the results as a guide only as the same size doesn't mean the same contents.

To find duplicated content, use the following dupe search instead:

dupe:size;sha256

Everything will find duplicate files by size first, then duplicated files by sha256 (content)
It's not instant, but it shouldn't be too slow either.



dupe-from:
Finding duplicates in Everything

[Turok]

Please try the following search:

dupe-from:"C:\FolderA" dupe:size

This will give you size duplicates where there is at least one duplicate from C:\FolderA

Sort by path and delete any files outside C:\FolderA



Please try the following in your search to restrict results to C:\FolderA, C:\FolderB and C:\FolderC:

"C:\FolderA\" | "C:\FolderB\" | "C:\FolderC\" dupe-from:"C:\FolderA" dupe:size

dupe:size will give you instant results.
Use the results as a guide only as the same size doesn't mean the same contents.

To find duplicated content, use the following dupe search instead:

dupe:size;sha256

Everything will find duplicate files by size first, then duplicated files by sha256 (content)
It's not instant, but it shouldn't be too slow either.



dupe-from:
Finding duplicates in Everything

This search "dupe-from:" reports 0 items. Tried different combinations.

[void]

Please make sure you specify a full path with dupe-from:

For example:

dupe-from:"C:\Program Files\Everything"

Use double quotes to escape any spaces.



dupe-from: MUST be combined with a dupe: search

dupe-from:"C:\Program Files\Everything" dupe:size

What version of Everything are you using?
-Please make sure you are using a recent version as dupe-from: is pretty new.

[Turok]

Please make sure you specify a full path with dupe-from:

For example:

dupe-from:"C:\Program Files\Everything"

Use double quotes to escape any spaces.



dupe-from: MUST be combined with a dupe: search

dupe-from:"C:\Program Files\Everything" dupe:size

What version of Everything are you using?
-Please make sure you are using a recent version as dupe-from: is pretty new.

Version 1.5.0.1383a (x64)
Probably is the version. Gonna update then.

[Turok]

Problem, after updating it started to index properties (seems that for all the files ) and says also rescanning: drive/path... maybe because I left ticked the option "automatically index NTFS volumes?" so I tried to use a .db backup file from 2 days ago, but it still insists in indexing and rescanning.

Tried as well with a backup from hours ago for the contents in Roaming folder but keeps rescanning and indexing.

It seems stuck indexing properties for a file. In the progress bar it says 3% and shows this file permanently.

[void]

Check Tools -> Debug -> Statistics -> Build -> Last Rebuild Reason

What type of the file is getting stuck? (eg: .pdf)

Please try moving that file out of your index or delete the file.

[Turok]

Check Tools -> Debug -> Statistics -> Build -> Last Rebuild Reason

What type of the file is getting stuck? (eg: .pdf)

Please try moving that file out of your index or delete the file.

It passed that file already.
Last rebuild reason: Out of date NTFS volume C: is available

[Turok]

What it means?
It's not rescanning anymore, but It is indexing properties for all the drives, not just C.
This rebuild will take a few hours so I'll leave it tonight and will try the search for dupes tomorrow.

[void]

Possibly too many changes occurred since the old db backup.

If you want results now, try the current db (not the backup db) and check Tools -> Debug -> Statistics -> Build -> Last Rebuild Reason.

If it shows added new NTFS volume X:, then remove X: from Tools -> Options -> NTFS, click OK in the options window, terminate Everything and try again.

[Turok]

Possibly too many changes occurred since the old db backup.

If you want results now, try the current db (not the backup db) and check Tools -> Debug -> Statistics -> Build -> Last Rebuild Reason.

If it shows added new NTFS volume X:, then remove X: from Tools -> Options -> NTFS, click OK in the options window, terminate Everything and try again.

It's fine, I left it and its finished now.
I've tried the dupe-from: query and it works now so my original question is solved, thank you for your help.

One more thing. How can I include to the index the calculated size;sha256 for the contents of any search I do with that query? Or is done automatically? And does it has a hit on performance in any way -I mean not when calculating but for the running process or during the start-? Like more RAM used?



I'm also wondering if there's a way to compare many folders between them to get a correlation of the amount of dupes. For example, I have 10 folders with dupes, and would like to get some kind of list with the amount of dupes that are in Folder A, and also in B, C, D etc sorted by the total amount of dupes in each folder against A (or against all of them if possible).

As example against A:
Folder A total files: 100.
Folder B total files: 100 of which 50 dupes, 25 present in A.
Folder C total files: 100 of which 50 dupes, 20 present in A.
Folder D total files: 100 of which 50 dupes, 15 present in A.

As example against all of them:
Folder A total files: 100 of which 50 dupes, 25 present in B, 15 present in C, 10 present in D.
etc

Is there a way to get this or anything similar?

[void]

One more thing. How can I include to the index the calculated size;sha256 for the contents of any search I do with that query? Or is done automatically? And does it has a hit on performance in any way -I mean not when calculating but for the running process or during the start-? Like more RAM used?

I don't recommend indexing sha256.
However, you can index sha256 under Tools -> Options -> Properties -> Add property -> Select SHA256.

If sha256 is not indexed, Everything will just gather sha256 values when needed.

Indexing sha256 will have a performance hit.
Every time a file changes, the sha256 will have to be updated.
It you are making a lot of file changes, the performance hit will increase.
If you do index sha256, consider limiting which folders are indexed, for example, just an archive folder.

There is a small increase in RAM usage, +33MB per 1million files.

I'm also wondering if there's a way to compare many folders between them to get a correlation of the amount of dupes. For example, I have 10 folders with dupes, and would like to get some kind of list with the amount of dupes that are in Folder A, and also in B, C, D etc sorted by the total amount of dupes in each folder against A (or against all of them if possible).

Not possible with Everything.
Everything only offers a simple flat result list.
Try sorting by path or other properties after finding duplicates.

[Turok]

One more thing. How can I include to the index the calculated size;sha256 for the contents of any search I do with that query? Or is done automatically? And does it has a hit on performance in any way -I mean not when calculating but for the running process or during the start-? Like more RAM used?

I don't recommend indexing sha256.
However, you can index sha256 under Tools -> Options -> Properties -> Add property -> Select SHA256.

If sha256 is not indexed, Everything will just gather sha256 values when needed.

Indexing sha256 will have a performance hit.
Every time a file changes, the sha256 will have to be updated.
It you are making a lot of file changes, the performance hit will increase.
If you do index sha256, consider limiting which folders are indexed, for example, just an archive folder.

There is a small increase in RAM usage, +33MB per 1million files.

Thanks for the detailed explanation. So is there a way to index sha256 only for specific folders? or you mean to just index specific folders excluding the rest in the global settings?

I'm also wondering if there's a way to compare many folders between them to get a correlation of the amount of dupes. For example, I have 10 folders with dupes, and would like to get some kind of list with the amount of dupes that are in Folder A, and also in B, C, D etc sorted by the total amount of dupes in each folder against A (or against all of them if possible).

Not possible with Everything.
Everything only offers a simple flat result list.
Try sorting by path or other properties after finding duplicates.

Understood. I'll try some workaround like that.

[void]

Thanks for the detailed explanation. So is there a way to index sha256 only for specific folders?

There's two options:

1). No sha256 indexing.
sha256 is gathered when required.

2). Index sha256
You can index all files/folders or just a few files/folders.
Searching for sha256 will be instant.
Sorting by sha256 will be fast.
You can set which folders to index and which folders to exclude.
files/folders that are not included will not have a sha256 value.

You cannot index some sha256 folders and expect Everything to gather sha256 for other folders.
What you index is what you get.



To index sha256 for a specific folder only:

• In Everything, from the Tools menu, click Options.
• Click the Properties tab on the left.
• Click the sha256 property.
• Change Include only folders to your specific folder.
• For example:
  D:\archive
• Click OK.

To exclude a specific folder:

• In Everything, from the Tools menu, click Options.
• Click the Properties tab on the left.
• Click the sha256 property.
• Change Exclude folder to your specific folder.
• For example:
  C:\Windows
• Click OK.

or you mean to just index specific folders excluding the rest in the global settings?

It depends what you are trying to do.
I recommend not indexing sha256 and just gather it when needed.

A better option might be to use .sha256 sidecar files.
Then use the "sha256sum SHA-256" property instead of sha256.

[Turok]

Thanks for the detailed explanation. So is there a way to index sha256 only for specific folders?

There's two options:

1). No sha256 indexing.
sha256 is gathered when required.

2). Index sha256
You can index all files/folders or just a few files/folders.
Searching for sha256 will be instant.
Sorting by sha256 will be fast.
You can set which folders to index and which folders to exclude.
files/folders that are not included will not have a sha256 value.

You cannot index some sha256 folders and expect Everything to gather sha256 for other folders.
What you index is what you get.



To index sha256 for a specific folder only:

• In Everything, from the Tools menu, click Options.
• Click the Properties tab on the left.
• Click the sha256 property.
• Change Include only folders to your specific folder.
• For example:
  D:\archive
• Click OK.

To exclude a specific folder:

• In Everything, from the Tools menu, click Options.
• Click the Properties tab on the left.
• Click the sha256 property.
• Change Exclude folder to your specific folder.
• For example:
  C:\Windows
• Click OK.

or you mean to just index specific folders excluding the rest in the global settings?

It depends what you are trying to do.
I recommend not indexing sha256 and just gather it when needed.

A better option might be to use .sha256 sidecar files.
Then use the "sha256sum SHA-256" property instead of sha256.

Uhm, I see, it's a handicap the part of not being able to index sha256 for specific folders and later then compare to others non indexed by gathering their contents sha256.

Is sha256 cached for the current session when gathered?

I'll learn more about this sha256sum SHA-256. Thanks.

[void]

Is sha256 cached for the current session when gathered?

Only while the tab is opened.
-The cache is cleared when you close the tab.