Question: Comparing drives/folders for files

If you are experiencing problems with "Everything", post here for assistance.
Post Reply
CarlRCraw
Posts: 5
Joined: Wed Mar 07, 2018 6:08 pm

Question: Comparing drives/folders for files

Post by CarlRCraw »

I have two drives/paths/folders (for example (a) C:\A-folder and (b) F:\B-folder) - each with multiple different sub-folders and files.

I need to verify that all the files of (a) exist somewhere on (b) without regard to path/folder.

My main purpose is to verify that the files of (a), which are proposed for deletion, exist somewhere on (b), or that I agree that the file is not necessary on (b) and can be deleted from (a) without impact.

Is there some way with Everything that I can do this analysis in an automated (or semi-automated) fashion?

Thanks for your help, Carl
NotNull
Posts: 5167
Joined: Wed May 24, 2017 9:22 pm

Re: Question: Comparing drives/folders for files

Post by NotNull »

On first thought, that is not possible "out of the box", but should be do-able with some extra effort.

But first, an important question: when do you consider files to be the same? When they have the same name? Same name + same size? Same name + same size + content bit-exactly the same?
CarlRCraw
Posts: 5
Joined: Wed Mar 07, 2018 6:08 pm

Re: Question: Comparing drives/folders for files

Post by CarlRCraw »

For my purpose probably just the name, but if it is not difficult to verify size/date as well then that might be preferable.

I want to verify that files still exist in the destination after a backup operation of a renamed folder. Many backup programs have to delete the old destination files and folder and than rebuild a new folder with the files rather than just renaming the destination backup folder to match the origin folder. My backup program saves all deleted files in a separate directory that allows me to verify the deletions. Doing this verification manually is very time consuming. Can't seem to find any software that will do this, so hoping Everything could help.
NotNull
Posts: 5167
Joined: Wed May 24, 2017 9:22 pm

Re: Question: Comparing drives/folders for files

Post by NotNull »

Thanks for clarification. I do not have time today to write something, but tomorrow should be manageable.
What do you want to do with the found duplicates? Delete them? / List them (and what details)? / ... ?
CarlRCraw wrote: Fri Jan 11, 2019 3:11 pm Can't seem to find any software that will do this,

You might want to take a look at the free version of Easy Duplicate Finder.
It has a scan mode: Folder comparison that does what (I think that) you want.
CarlRCraw
Posts: 5
Joined: Wed Mar 07, 2018 6:08 pm

Re: Question: Comparing drives/folders for files

Post by CarlRCraw »

Thank You.

I simply want to visually see the files and their path for all (a) files and if it finds a (b) file, it's path. If it does not find a (b) file, then simply list the (a) file and path.

Similar to what the program lists now for files that it finds (listing them all together with their paths), but limiting/filtering the search/listing to only those files in folder/path (a) (rather than the whole disk drive).

Thanks for your help.

Carl
vanisk
Posts: 152
Joined: Sat Oct 27, 2018 11:33 am

Re: Question: Comparing drives/folders for files

Post by vanisk »

Try WinMerge (File and Folder comparision utility) which compares 2 different location and lists out identical files and "Location A only" files and "Location B only" files.

2019-01-11 23_29_27-WinMerge.jpg
2019-01-11 23_29_27-WinMerge.jpg (23.33 KiB) Viewed 15585 times

2019-01-11 23_21_32-WinMerge.jpg
2019-01-11 23_21_32-WinMerge.jpg (21.93 KiB) Viewed 15589 times

http://winmerge.org/

or if u prefer portable release
https://portableapps.com/apps/utilities ... e_portable
CarlRCraw
Posts: 5
Joined: Wed Mar 07, 2018 6:08 pm

Re: Question: Comparing drives/folders for files

Post by CarlRCraw »

Hi,

Thanks for WinMerge idea. Tested it. It doesn't do what I need done. It only looks at current folder directories and files; it doesn't drill down into all folders for differences.

This is what I am looking for:
(1) For each file in the origin folder [A], search all folders/subfolders in the destination for its existence, displaying the path for each file found.
(2) Repeat this process for all files in all folders/sub-folders in [A].

Carl
NotNull
Posts: 5167
Joined: Wed May 24, 2017 9:22 pm

Re: Question: Comparing drives/folders for files

Post by NotNull »

CarlRCraw wrote: Mon Jan 14, 2019 9:10 pm This is what I am looking for:
(1) For each file in the origin folder [A], search all folders/subfolders in the destination for its existence, displaying the path for each file found.
(2) Repeat this process for all files in all folders/sub-folders in [A].
Did you take a look at Easy Duplicate Finder? It seems to do what you want (also in portable version):
2019-01-14 22_41_07-Window.png
2019-01-14 22_41_07-Window.png (47.84 KiB) Viewed 15561 times
2019-01-14 22_52_11-Window.png
2019-01-14 22_52_11-Window.png (47.3 KiB) Viewed 15561 times
(I did not have any time last weekend to write something, btw. Will do when Easy Duplicate Finder turns out to be not what you are looking for.)
therube
Posts: 4580
Joined: Thu Sep 03, 2009 6:48 pm

Re: Question: Comparing drives/folders for files

Post by therube »

Do a search on your two wanted paths:

Code: Select all

X: | M:
File | Export -> music-m+x.efu

File | Open File List -> music-m+x.efu

Do a search to find duplicates:

Code: Select all

dupe:
Or unique's:

Code: Select all

!dupe:

Now there can be some gotchas with that.
It will turn up two files with the same name, within the same directory tree.
(And probably some other things I'm not thinking of off the top of my head.)

Code: Select all

M:\___fromC\The Cinematic Orchestra\The Cinematic Orchestra - Ma Fleur (2007)\02 - Familiar Ground.mp3
X:\_MUSIC\___fromC\The Cinematic Orchestra\The Cinematic Orchestra - Ma Fleur (2007)\02 - Familiar Ground.mp3
X:\_MUSIC\jazz\cinematic orchestra\Ma Fleur\02 - Familiar Ground.mp3
therube
Posts: 4580
Joined: Thu Sep 03, 2009 6:48 pm

Re: Question: Comparing drives/folders for files

Post by therube »

Duplicate Cleaner: Verify files between two drives/paths

You can also set DC to find duplicates between 2 directory trees (ignoring any dupes that may happen to be within the same tree).
(Ditto with AllDup, "Compare only files between different source folders".)
NotNull
Posts: 5167
Joined: Wed May 24, 2017 9:22 pm

Re: Question: Comparing drives/folders for files

Post by NotNull »

Tonight I had some time to write something (just for fun; I still think the aforementioned tools are much better equipped for the task).
It's a basic CMD script that requires ES.exe (the command-line interface for Everything):
2019-01-17 00_03_42-C__Windows_System32_cmd.exe.png
2019-01-17 00_03_42-C__Windows_System32_cmd.exe.png (17.35 KiB) Viewed 15540 times


And that generates 2 outputfiles: Output.txt and CanBeDeleted.txt.
It is really fast as it uses Everything :) (although my code could be optimized ...)

Output.txt

Code: Select all

[ "T:\DELETEST\NOTFOUND\AutoHotkeyU64.exe" ] 
C:\develop\JumpToFolder\AutoHotkeyU64.exe
C:\develop\JumpToFolder\GUI menu\AutoHotkeyU64.exe
C:\develop\JumpToFolder\GUI menu\IconTest\AutoHotkeyU64.exe
C:\develop\Tweaks\AutoHotkeyU64.exe
C:\Tools\AutoHotkey_1.1.27.07\AutoHotkeyU64.exe
[ "T:\DELETEST\Everything.chm" ] 
C:\Tools\Everything\Everything.chm
[ "T:\DELETEST\Everything.exe" ] 
C:\develop\Everything Installer\Everything-1.4.1.895.x64-Setup\$PLUGINSDIR\Everything\Everything.exe
C:\Tools\Everything\Everything.exe
[ "T:\DELETEST\NOTFOUND\Unique1 - Copy.txt" ] 
[ "T:\DELETEST\NOTFOUND\Unique1.txt" ] 

CanBeDeleted.txt

Code: Select all

"T:\DELETEST\NOTFOUND\AutoHotkeyU64.exe" 
"T:\DELETEST\Everything.chm" 
"T:\DELETEST\Everything.exe" 

NotNull
Posts: 5167
Joined: Wed May 24, 2017 9:22 pm

Re: Question: Comparing drives/folders for files

Post by NotNull »

therube wrote: Wed Jan 16, 2019 4:34 pm Duplicate Cleaner: Verify files between two drives/paths

You can also set DC to find duplicates between 2 directory trees (ignoring any dupes that may happen to be within the same tree).
(Ditto with AllDup, "Compare only files between different source folders".)
Was followingv your links and in the first one the question was asked by someone with the same username as "our" CarlRCraw :D
vanisk
Posts: 152
Joined: Sat Oct 27, 2018 11:33 am

Re: Question: Comparing drives/folders for files

Post by vanisk »

NotNull wrote: Wed Jan 16, 2019 11:08 pm It's a basic CMD script that requires ES.exe (the command-line interface for Everything):
Thank you & Waiting for the upload to try it :).
NotNull
Posts: 5167
Joined: Wed May 24, 2017 9:22 pm

Re: Question: Comparing drives/folders for files

Post by NotNull »

vanisk wrote: Thu Jan 17, 2019 10:34 am Thank you & Waiting for the upload to try it :).
And I was waiting for feedback from @CarlRCraw ..
But now there is more interest, will post it tonight (European timezone).
therube
Posts: 4580
Joined: Thu Sep 03, 2009 6:48 pm

Re: Question: Comparing drives/folders for files

Post by therube »

(Heh. Nope, I didn't catch that.)

-------

Another possibility to find unique files between two trees:

Code: Select all

dir  x:  m:  /b /s  |  sed "s/.*\\//"  |   sort   |  uniq -d  >  list_uniq
NotNull
Posts: 5167
Joined: Wed May 24, 2017 9:22 pm

Re: Question: Comparing drives/folders for files

Post by NotNull »

This is tailor-made for @CarlRCraw, but can be made quite easily in something more universal
Files are considered the same when they have equal name, date modified AND size

Instructions
  • If you haven't done so already, download ES.exe from here
  • Save this script in some folder on your system.
    I named it DELTEST.cmd, but name it whatever you feel like (DualDirectoryDuplicateDetector.cmd ? :))
  • In the script, change the path to ES.exe to match your situation
  • Check if Everything is running. If not: start it.
  • Double-click your 'Whatever.cmd' in Explorer
  • Fill in the path to your back-ups by (*) dragging the folder to this window, (*) typing the folder name or (*) use Tab-completion (type c:\p and <Tab> till you reach "Program Files").
  • Same procedure for the other folder

    When done, you can view the results in Output.txt and CanBeDeleted.txt

    ( Build and (quickly) tested with Everything 1.4.1.895 x64 /ES 1.1.0.9 @Win10 )

    DELTEST.cmd

    Code: Select all

    @echo off
    setlocal
    
    ::_______________________________________________
    ::
    ::	SETTINGS
    ::_______________________________________________
    ::
    
    ::	PAth to ES.exe
    	set EXE=c:\Tools\Everything\ES.exe
    
    ::	Location outputfile
    	set OUTPUT=.\Output.txt
    ::	Location list of deletable files
    	set DELETE=.\CanBeDeleted.txt
    	
    
    ::_______________________________________________
    ::
    ::	INIT
    ::_______________________________________________
    ::
    
    	pushd "%~dp0"
    	set TEMPFILE=.\zzzFolder1.txt
    	del "%DELETE%" 2>nul
    	del "%OUTPUT%" 2>nul
    
    
    ::_______________________________________________
    ::
    ::	ACTION!
    ::_______________________________________________
    ::
    
    ::	Get backup folder
    	echo.
    	echo.   Type the foldername or
    	echo.	drag/drop the folder where your BACKUPS are
    	echo.   from Explorer to this window and press [Enter]
    	echo.
    	set /p FOLDER1=Backup folder : 
    
    ::	Get original folder
    	echo.
    	echo.   Type the foldername or
    	echo.   drag/drop the folder where your ORIGINAL FILES are
    	echo.   from Explorer to this window and press [Enter]
    	echo.
    	set /p FOLDER2=Original folder : 
      
    	cls
    	echo.
    	echo.___________________________________________________
    	echo.
    	echo.	Backup folder     : %FOLDER1%
    	echo.	Original files in : %FOLDER2%
    	echo.
    	echo.
    	echo.	Remember: Everything must be running!
    	echo.___________________________________________________
    	pause
    
    
    	
    ::	Strip "" from foldernames and 
    ::	remove possible trailing \ in foldername
    
    
    	for /f "useback delims=" %%x in ('%FOLDER1%\.') DO set FOLDER1=%%~dpnxx
    	for /f "useback delims=" %%x in ('%FOLDER2%\.') DO set FOLDER2=%%~dpnxx
    
    ::	Escape special chars & and ^ (according to ES help)
    	rem seems not needed :-s. Probably because those are ""
    	
    ::	"Index" files in Backup folder
    	"%EXE%" -full-path-and-name -dm -size -size-format 1 "%FOLDER1%\" file: -export-csv "%TEMPFILE%"
    
    ::	Compare each file with "Original folder"
    	for /F "skip=1 usebackq tokens=1-3 delims=," %%x  in ("%TEMPFILE%") DO call :PARSEFILE %%x "%%y" "%%z"  
    	del "%TEMPFILE%"
    	pause
    	
    goto :EOF
    
    
    
    :PARSEFILE
    ::	Skip empty lines
    	if [%1]==[] goto :EOF
    
    ::	The CALL command doubles a "^" (bug). Undo this (= THISFILE)
    	set THISFILE=%1
    	set THISFILE=%THISFILE:^^=^%
    
    ::	Strip path from filemae (= THISNAME)
    	for /f "delims=" %%a in (%THISFILE%) do set THISNAME="%%~nxa"
    
    ::	Log THISFILE
    	echo [ %THISFILE% ] >> "%OUTPUT%"
    
    ::	Find THISNAME in FOLDER2
    ::	If found: add to logfile  and to DELETE file
    	"%EXE%" "%FOLDER2%\" file: %THISNAME% dm:%2 size:%3 -size-format 1 | findstr "\\" >> "%OUTPUT%" && echo %THISFILE%>> "%DELETE%"
    
    ::	Done. Next file from FOLDER1 ...
    
    goto :EOF
    
    
Last edited by NotNull on Sun Jan 20, 2019 8:27 pm, edited 1 time in total.
vanisk
Posts: 152
Joined: Sat Oct 27, 2018 11:33 am

Re: Question: Comparing drives/folders for files

Post by vanisk »

therube wrote: Thu Jan 17, 2019 4:50 pm

Code: Select all

dir  x:  m:  /b /s  |  sed "s/.*\\//"  |   sort   |  uniq -d  >  list_uniq
Thanks for the code @therube (Learning something new everyday)

NotNull wrote: Thu Jan 17, 2019 7:01 pm DELTEST.cmd
Thanks @NotNull for sharing the script. Tested it and working fine.
therube
Posts: 4580
Joined: Thu Sep 03, 2009 6:48 pm

Re: Question: Comparing drives/folders for files

Post by therube »

dir x: m: /b /s | sed "s/.*\\//" | sort | uniq -d > list_uniq

Revised:

Code: Select all

dir  x:  /b /s  |  sed "s/.*\\//"  |   sort -u    >  list_tmp
dir  m:  /b /s  |  sed "s/.*\\//"  |   sort -u   >>  list_tmp
uniq -d  <  list_tmp  > list_uniq
The sort -u eliminates duplicates found in a particular directory tree (so either X: or M:)
& then the uniq eliminates duplicates between both trees (so X: and M:).


Can you please tell me, where to get sed & uniq exe files.
GnuWin
GnuWin Packages
I've never "installed" (so can't comment on that).
I download the Zip binaries (extract the .exe's) & stick them in a place in my path (C:\BIN\UNIX\).
Some "packages" have "dependencies", so you might need those also.
Some programs may have stand-alone links, like sed & will contain "sed".
Some programs package a bunch of small stand-alone utilities together, like CoreUtils, which include lots, including uniq.


If you needed (even older) 16-bit DOS programs, there is DJGPP.
NotNull
Posts: 5167
Joined: Wed May 24, 2017 9:22 pm

Re: Question: Comparing drives/folders for files

Post by NotNull »

therube wrote: Thu Jan 17, 2019 9:12 pm GnuWin
GnuWin Packages
I've never "installed" (so can't comment on that).
I download the Zip binaries (extract the .exe's) & stick them in a place in my path (C:\BIN\UNIX\).
I switched to Cygwin as GnuWin seemed a bit abandoned and Cygwin had a 64-bit version. For the rest:same difference ...
NotNull
Posts: 5167
Joined: Wed May 24, 2017 9:22 pm

Re: Question: Comparing drives/folders for files

Post by NotNull »

vanisk wrote: Thu Jan 17, 2019 8:32 pm Thanks @NotNull for sharing the script. Tested it and working fine.
Thanks for testing!

It still has a couple of issues, which I will iron out later:
- problems with folders with spaces in their name (the default issue in any script/program :))
- If you use folder c:\abc, but also have a folder like c:\abcd or "c:\abc something else", it will scan those too.
- I need to do some extra testing with special characters "&" and "^" (special meaning in ES)
- No idea what will happen with (for example) Cyrillic characters in filenames

Update will follow.
NotNull
Posts: 5167
Joined: Wed May 24, 2017 9:22 pm

Re: Question: Comparing drives/folders for files

Post by NotNull »

NotNull wrote: Thu Jan 17, 2019 9:43 pm Update will follow.
Updated.
Still no idea what will happen with (for example) Cyrillic characters in filename, though.
Stamimail
Posts: 1121
Joined: Sat Aug 31, 2013 9:05 pm

Re: Question: Comparing drives/folders for files

Post by Stamimail »

Another tool might help here - the new version of ToDownload by ovg:
viewtopic.php?f=7&t=5998#p19575
Post Reply