Compare larger folders for migration

General discussion related to "Everything".
Post Reply
Just Visiting
Posts: 12
Joined: Sat Jul 16, 2011 10:38 pm

Compare larger folders for migration

Post by Just Visiting » Tue Jun 08, 2021 2:17 am

I am in the process of slowly migrating one my large folders with thousands of files from my legacy W7 PC to a new laptop. At the moment, I wish to keep both folders on both drives and then merge the folders back and forth as necessary.

Is there a simple Everything script or command I can use to scan both folders (the one my laptop and the one on my desktop) when I copy them over to a temporary directly on each PC and show me only the most recent ones? Or perhaps there's an even simpler approach you can recommend? For example. The main folder is called MYdocuments on Laptop and the other is called the same on my PC. I use a USB drive to copy the folder onto and then would run Everything to compare the differences.

horst.epp
Posts: 483
Joined: Fri Apr 04, 2014 3:24 pm

Re: Compare larger folders for migration

Post by horst.epp » Tue Jun 08, 2021 7:39 am

Thats a easy job for any file manager you may try (not Windows Explorer of course).
Synchronize Directories in Total Commander for example.

therube
Posts: 3026
Joined: Thu Sep 03, 2009 6:48 pm

Re: Compare larger folders for migration

Post by therube » Wed Jun 09, 2021 5:41 pm

If it's a straight 1 directory to 1 directory comparison, I'd agree with the file manager argument that should easily point out differences.

For a mixed method, also including Everything, see [freefilesync] Problem using the Mirror function.

Just Visiting
Posts: 12
Joined: Sat Jul 16, 2011 10:38 pm

Re: Compare larger folders for migration

Post by Just Visiting » Thu Jun 17, 2021 3:57 am

Ok. so I have downloaded Total Commander and Free. Now what do I do?

How do I use Freefilesync?
therube wrote:
Wed Jun 09, 2021 5:41 pm
If it's a straight 1 directory to 1 directory comparison, I'd agree with the file manager argument that should easily point out differences.

For a mixed method, also including Everything, see [freefilesync] Problem using the Mirror function.
horst.epp wrote:
Tue Jun 08, 2021 7:39 am
Thats a easy job for any file manager you may try (not Windows Explorer of course).
Synchronize Directories in Total Commander for example.

horst.epp
Posts: 483
Joined: Fri Apr 04, 2014 3:24 pm

Re: Compare larger folders for migration

Post by horst.epp » Thu Jun 17, 2021 6:14 am

Just Visiting wrote:
Thu Jun 17, 2021 3:57 am
Ok. so I have downloaded Total Commander and Free. Now what do I do?

How do I use Freefilesync?
therube wrote:
Wed Jun 09, 2021 5:41 pm
If it's a straight 1 directory to 1 directory comparison, I'd agree with the file manager argument that should easily point out differences.

For a mixed method, also including Everything, see [freefilesync] Problem using the Mirror function.
horst.epp wrote:
Tue Jun 08, 2021 7:39 am
Thats a easy job for any file manager you may try (not Windows Explorer of course).
Synchronize Directories in Total Commander for example.
In TC open the root of your file sets in the 2 panels.
Then execute the command cm_FileSync in the command line on the bottom.
There are ways to start it from the Button bar but I don't know your layout.
See the 4 buttons in the Show section and select what you want to see and hit Compare.
Screenshot - 17.06.2021 , 08_16_31.png
Screenshot - 17.06.2021 , 08_16_31.png (58.97 KiB) Viewed 7222 times

NotNull
Posts: 3131
Joined: Wed May 24, 2017 9:22 pm

Re: Compare larger folders for migration

Post by NotNull » Thu Jun 17, 2021 1:33 pm

Be aware that Total Commander's Synchronize Directories is not reliable with 'modern' files. Simple (DOS-like) files should be OK though.

For example: TC doesn't handle files with named datastreams (/ADS) well. Maybe other 'modern' files too, but I stopped testing after that.
(TC 9.50B7 x64 btw)


I don't have an alternative to offer.

horst.epp
Posts: 483
Joined: Fri Apr 04, 2014 3:24 pm

Re: Compare larger folders for migration

Post by horst.epp » Thu Jun 17, 2021 3:41 pm

NotNull wrote:
Thu Jun 17, 2021 1:33 pm
Be aware that Total Commander's Synchronize Directories is not reliable with 'modern' files. Simple (DOS-like) files should be OK though.

For example: TC doesn't handle files with named datastreams (/ADS) well. Maybe other 'modern' files too, but I stopped testing after that.
(TC 9.50B7 x64 btw)


I don't have an alternative to offer.
What do you mean with "modern" files ?
TC copies files with the official Windows API.
There are no known problems with any sort of files as the content is not relevant.

NotNull
Posts: 3131
Joined: Wed May 24, 2017 9:22 pm

Re: Compare larger folders for migration

Post by NotNull » Thu Jun 17, 2021 3:56 pm

It is not about the copying, it's about the comparing.

Try this:
Create a new folder, start PowerShell and go to that folder.
Run the following commands (just select all and paste it in PowerShell).

That will create a abc.txt in folder1 as well as folder2.
Difference being that they have different named datastreams and those datastreams not only have a different name, they even have different content.
TC sync comparison sees both folders as being equal ....

Code: Select all

md .\Folder1, .\Folder2
Set-Content .\folder1\abc.txt -Stream 'Folder1' "This is Folder1 stream"
Set-Content .\folder2\abc.txt -Stream 'Folder2' "Something else"
(gi .\Folder1\abc.txt).LastWriteTime = (gi .\Folder2\abc.txt).LastWriteTime
cat .\Folder1\abc.txt:Folder1
cat .\Folder2\abc.txt:Folder2



horst.epp
Posts: 483
Joined: Fri Apr 04, 2014 3:24 pm

Re: Compare larger folders for migration

Post by horst.epp » Thu Jun 17, 2021 4:14 pm

NotNull wrote:
Thu Jun 17, 2021 3:56 pm
It is not about the copying, it's about the comparing.

Try this:
Create a new folder, start PowerShell and go to that folder.
Run the following commands (just select all and paste it in PowerShell).

That will create a abc.txt in folder1 as well as folder2.
Difference being that they have different named datastreams and those datastreams not only have a different name, they even have different content.
TC sync comparison sees both folders as being equal ....

Code: Select all

md .\Folder1, .\Folder2
Set-Content .\folder1\abc.txt -Stream 'Folder1' "This is Folder1 stream"
Set-Content .\folder2\abc.txt -Stream 'Folder2' "Something else"
(gi .\Folder1\abc.txt).LastWriteTime = (gi .\Folder2\abc.txt).LastWriteTime
cat .\Folder1\abc.txt:Folder1
cat .\Folder2\abc.txt:Folder2


Zero sized files with the same name and time stamp are of no practical relevance.
Where do you have such files ?
Updating datastreams normaly also updates the time stamps
if not done by an virus.

NotNull
Posts: 3131
Joined: Wed May 24, 2017 9:22 pm

Re: Compare larger folders for migration

Post by NotNull » Thu Jun 17, 2021 5:02 pm

This is just a simple, minimal demo to show the effect. It would not be very hard to imagine a more realistic scenario.

The main point here is that TC does not detect this, where it should. It's a shortcoming in it's comparison routine. Enabling ignore date for comparisons would make it even worse.

Just Visiting
Posts: 12
Joined: Sat Jul 16, 2011 10:38 pm

Re: Compare larger folders for migration

Post by Just Visiting » Thu Jun 17, 2021 9:55 pm

Thanks everyone.

Unfortunately, you all LOST me completely. This is way beyond my level of computer expertise.

I will add that TC is the easiest to look at even though it uses a very primitive 1994 DOS like interface. Command lines are for Linux and power users, not modern day Windows users who look and need "a turn key solution"...in my humble opinion. I became totally confused with the RED X marks, also shown in the above screen shot.

I also played around with Freesync and each time Freesync kept the duplicate files scattered from everywhere. The interface started to become confusing the more I tried to find a way to delete the duplicates during the process.

With TC, on ONE occasion [ONLY] it deleted the duplicates, but other times it kept them BOTH as well. Fortunately, I had my backup copy and I lost a few hours messing with these applications. TC deleted NON duplicate files.

Clearly, these two applications are not simple and effective enough for someone who wants an automated process to compare, add, and delete files that are the same and different while prioritizing a particular drive or directory.

In other words, if my large directory called Dell has been modified on another PC, but needs to be merged into the same directory onto another PC called ASUS, that's really all I want. Any files which have been deleted, say from, Dell would have more priority over the files that still remain on ASUS. They would also be deleted [automatically] on ASUS as part of the process. At least that's what I need.

From this point on, I think it's best save any new files or documents on the cloud instead of repeating this time process hoping to obtain the results I need.

I will make one last attempt on Reddit to find a suitable solution for me. Perhaps I'll have better luck there. Thanks again for your efforts.

tuska
Posts: 410
Joined: Thu Jul 13, 2017 9:14 am

Re: Compare larger folders for migration

Post by tuska » Thu Jun 17, 2021 11:17 pm

Hi,
Just Visiting wrote:
Thu Jun 17, 2021 9:55 pm
In other words, if my large directory called Dell has been modified on another PC,
but needs to be merged into the same directory onto another PC called ASUS, that's really all I want.

Any files which have been deleted, say from, Dell would have more priority over the files that still remain on ASUS.
They would also be deleted [automatically] on ASUS as part of the process. At least that's what I need.
 
2021-06-18_Compare larger folders for migration_10141.png
2021-06-18_Compare larger folders for migration_10141.png (22.24 KiB) Viewed 7081 times
 
F1-Help - <customised for you> wrote:Asymmetric
If this option is checked, it is assumed that a copy of the left side (folder: "Dell") should be created on the right side (folder: Asus).
Simply press the F1 key in the "Synchronize directories" window to call up the help.
Pay close attention to the buttons I have pressed(!)
Just Visiting wrote:
Thu Jun 17, 2021 9:55 pm
I became totally confused with the RED X marks, also shown in the above screen shot.
F1-Help wrote: Blue: Copy from right to left, or delete on the right side (asymmetric mode, in this case, the file has a cross symbol x in the middle).
If the folder "Dell" is to be prioritised, then this means that after an asymmetrical synchronisation
all files/subfolders in the "Asus" folder should also be present ("1:1") - just as in the "Dell" folder.
With thousands of files, I think it is impossible that there would be no files to delete in the "Asus" folder...

It is best to test beforehand with 2 test folders and a few files that differ in these folders,
to get clarity about the process.

Regards,
Karl

PS:
If you want to synchronise the two folders "Dell" and "Asus" against each other, then it is necessary to uncheck "Asymmetric".
The following also applies in this case:
It is best to test beforehand with 2 test folders and a few files that differ in these folders.

PPS:
If you have questions or problems regarding synchronisations in Total Commander,
then I would recommend that you post a request in the Total Commander forums,
because this is not the right place...
________________________________________________
Duplicate files
  • Search for duplicate files in Total Commander
  • Alt+Shift+F7 Search in separate process...
  • Click on tab "Advanced"
  • Tick "Find duplicate files",...
  • Click on button "Start search"
  • Click on button "Feed to listbox"
More information is available on request in the TC Forum.

Post Reply