Looking for Software: Copy/Move/Make Files Sparse

Plug-in and third party software discussion.
Post Reply
raccoon
Posts: 90
Joined: Thu Oct 18, 2018 1:24 am

Looking for Software: Copy/Move/Make Files Sparse

Post by raccoon » Fri Jul 03, 2020 7:47 am

I am looking for a piece of software that can copy or move a Sparse File from one drive to another drive, and by the same token, will also convert a normal file into a Sparse File on that same drive.

Primer

Sparse Files are a feature of NTFS which allows an incompletely written file, such as a download-in-progress, to consume less disk space than it would on other simple file systems like FAT32 and exFAT. This is normally the behavior on any filesystem if the incomplete file is being written to sequentially, but, NTFS also allows for non-sequential writes anywhere within the file, without first having to allocate file space using zero-padding. On NTFS, a Sparse File lets you write chunks anywhere to the address mapping of a file, and the file will only consume the amount of disk space of the unique data that has been written. Consider how Bittorrent files are downloaded in random chunks, for example.

Microsoft Windows allows for Sparse Files to be renamed or moved from one directory to another on the same drive, but if you attempt to copy or move the file to a secondary file or another drive, a non-Sparse regular file is created that will consume the full disk allocation. If you want to move a folder with many Sparse Files from old Disk X to new Disk Y, then those files will consume a lot more disk space and there is no way around this.

My Request

I need a piece of software that can scan through the contents of a file, locate large sequences of 0x00, and then write a duplicate of this file while SEEKing (jumping) over those 0x00 sequences, thus ''compressing'' the normal file into a sparse file. The software should be able to do this with a file you wish to ultimately copy or move using a basic Source and Destination model. A move operation is equivalent to a Copy-then-Delete operation.

The software should be able to specify the minimum 0x00 sequence length to consider making Sparse. Eg: 512 bytes. 4096 bytes. 1048576 bytes.

I presently cannot find any such software that does this.

Why?

You can reasonably store 4 terabytes of incomplete sparse files on your 1 terabyte SSD (high speed) scratch disk. This is what I do. It is too costly to purchase a 4 terabyte SSD scratch disk, and it is impossible to move old projects from one scratch disk to another during hardware upgrades.

Who else would use this, though?

Sparse files can free up space on any NTFS disk. Any file can be, in theory, converted into a sparse file and remain fully functional without any performance hits. In fact, sparse files are faster to read into memory than normal files, given that the zero-fill does not have to be physically read from the source file... only implied. Normal users can try scanning for files that contain large 0x00 segments, and then have those files converted into sparse files to free up disk space.

NotNull
Posts: 2156
Joined: Wed May 24, 2017 9:22 pm

Re: Looking for Software: Copy/Move/Make Files Sparse

Post by NotNull » Fri Jul 03, 2020 8:34 pm

Directory Opus does support copying sparse files (as sparse file, of course). But I never used it.

See also this thread and more specific: @therube's contribution.

A (very) quick websearch resulted in this (and the link in aforementioned thread).

Post Reply