Page 1 of 1

Looking for a tool that can merge overlap two files

Posted: Fri Apr 10, 2020 9:27 am
by raccoon
I've got two files of the same size in bytes.
I want to merge both files, overlapping them, treating 0x00 (NUL) bytes as transparent.
Ideally, the program would notify me if any two non-NUL characters share the same byte offset that are dissimilar.

Goal: Take two incomplete and non-sequentially downloaded files from different download sources, that are supposed to be the same file, to build a slightly-more completed (or totally completed) file.

Anyone know of such a program?

Re: Looking for a tool that can merge overlap two files

Posted: Fri Apr 10, 2020 10:35 am
by therube
So you're saying you downloaded a file, & the 1st attempt only got part, & the second attempt picked up the remainder - but in a separate file, with both files ending up being the same size. With one having byte-byte-byte... and ending with nul-nul-nul. The other beginning with nul-nul-nul & ending with byte-byte-byte.

Use a hex editor, cut the 1st file after the end of the "data" (i.e. where the nul's start).
Use a hex editor, cut the 2nd file, from the beginning (which would be where the nul's start) to the start of the "data".

Then combine the two together.
cat (or type.exe /b, perhaps?) would do it.

Code: Select all

cat.exe  file1  file2  > filecomplete
filecomplete should be the same size as (both) file1 & file2, & have the "data" from file1 & the "data" from file2, nul's removed.

https://mh-nexus.de/en/hxd/


Depending on the file... it may be more or less difficult to get it "right".
Because there could be the odd nul that you've removed that does actually belong in the file.

With say a movie, a missing byte or so is immaterial. Your player will simply ignore it & you will never "see" it.
With a ZIP file, a missing byte or so, may end up being immaterial.

With a program, .exe, that could be the case too.
With a signed (certificate) program, the signing would be compromised & the program may not run.

So some trial & error may be in order.


How to find the start (end) of the nul's?
Well you could search for a consecutive set of strings of $00$, which may be end up being inaccurate.
Or you could jump to the end (beginning) of the file & search upwards (downwards) for say the letter 'e'.
That is likely to turn up near the beginning of the nul's you want to cut.
Once you find that point, cut downwards (upwards) to the end (beginning) of each respective file.

Code: Select all

Directory of C:\WLIB\LeechFTP\patch

04/10/2020  07:23 AM        29,672,012 file1
04/10/2020  07:00 AM       163,466,028 file2

04/10/2020  06:49 AM       193,138,040 file1.bak
04/09/2020  03:04 AM       193,138,040 file2.bak

04/09/2020  03:04 AM       193,138,040 filecomplete
file1 & file2 are the cut files
.bak are the original files; file1 with the nul's at the end, file2 with the nul's at the beginning.
filecomplete is the combined file1+file2, & is complete & accurate & works (it's a signed .exe in this case).

file1: beginning is "data", at some point, nothing but nul's until its end
Image

file2: beginning is nul's, at some point, its data begins until the eof
Image


(And since today's turbotax patch did not complain about its certificate being wrong, & since it did just update turbotax & since turbotax runs, & with the assistance the government is giving me due to coronavirus, I've been promised that within two weeks, my refund+stimulus incentive monies totally $1,000,000,000 will be here - within two weeks :-). Now they failed to say when that two weeks starts? I'm thinking two weeks after hell freezes over :o.)

Re: Looking for a tool that can merge overlap two files

Posted: Sat Apr 11, 2020 7:55 am
by raccoon
I'm afraid it's not so simple as one file containing the top half, and the second file containing the bottom half of the data. They're both quite interleaved in their completeness (and lack there of), with even overlapping bits of completeness.

What I described, treating NUL characters as transparent, is pretty straight forward. Imagine two image layers in Photoshop as a good analogy, where the top most layer has transparent bits you can see-through to the layer beneath it. The image need not be a top-half or bottom-half, but interspersed with transparent holes. Same goes for my part files I wish to merge together into one file.

They're an identical file from two different old stale incomplete torrents.