For those who are keeping score, I’m a dumbass.
So, I have a file server at my home that’s running FreeNAS, which is based on FreeBSD, a popular Unix distribution. I have around 7.4 terabytes of storage capacity across 5 disks; that’s a lot of personal files. I bought a single 8TB disk for the purposes of backing up this mountain of data. The problem is that my methods of making exact copies of files to the disk were failing or taking too long to complete, so the backups were slow and not always good.
When you’re dealing with terabytes of data, everything’s slow and insufficient.
Well, the problem is that I was looking at the problem from the wrong point of view. On my personal workstation, which runs Linux and uses the ext4 filesystem, I have a stack of scripts that basically run “rsync” to make exact copies to a portable 2TB disk. This method is what I tried on the FreeNAS box. Due to the way rsync operates under the hood, especially on complex filesystems where there are lots of hardlinked files, it was occasionally consuming all of the free memory on the fileserver and eventually dying after a few hours of churning, sometimes with no files actually copied.
“There has to be a better way to do this,” I thought. “I guess I’ll split the disk up into smaller chunks and make scripts to do it that way.”
And then, like a strike of lightning, it hit me: this FreeNAS box isn’t running on an ext4 filesystem — it’s running ZFS. What feature does ZFS have that ext4 doesn’t? That’s right — snapshots. Futuristic, amazing, wonderful snapshots. And what can you do with these snapshots? That’s right, you can replicate them, bit-for-bit, to another disk or remote system. Woah.
Just slap me now.
So, I did what any sane person would do: use the available tools as intended. I made a snapshot. I then wiped the backup disk clean, and am currently replicating that snapshot to the backup disk. Then, once the initial snapshot is there, I can take incremental snapshots at later dates and push them to the backup disk in a significantly shorter amount of time.
Use what’s available to you. That’s the key lesson here. This is liberating. Let’s hope this works!