Only the Hardest Links

At home I use backintime as my backup software (a FOSS clone of Apple’s Time Machine). It works fantastically (and has saved my ass several times). It’s a Python framework around some very clever rsync magic; daily incremental snapshots are light on space because it uses hard links on the backend storage if a file hasn’t changed between snapshots. So if you have 1TB of backups, and if nothing changes during any of the snapshots, the overall storage of the entire daily backup dataset is still 1TB. Magic.

Thing is, the moment I move files around, the next snapshot grows the dataset by the size of the diff, because ta-da those moved files look like new files, and the old ones don’t disappear, because they’re still referenced by the previous snapshots. Neither backintime nor the filesystem know they’re the same files.

So, if I put my files in some dumb bonehead layout (like the tons of photos I’ve shot over the years), and if I move them around into some smart layout, then my backup storage will blow up like a balloon. What to do‽

Remember the part where backintime makes hard links on the backend? Did you also wonder if backintime observes hard links on the frontend and preserves those on the backend? Let’s find out!

user@desktop:~/$ ln notes.txt notes-hardlink.txt
user@desktop:~/$ ll notes*
-rw-rw-r-- 2 user user 1453 Jun 20  2018 notes-hardlink.txt
-rw-rw-r-- 2 user user 1453 Jun 20  2018 notes.txt

Here I have a file notes.txt file that was previously backed up. I make a hard link (ln without the -s) on the desktop machine. You can tell there are now 2 inode references to the file (see the 2 after the file permissions).

I take note of how many inode refs are on the fileserver before the next snapshot: 41.

root@fileserver:/mnt/backintime/desktop/root/1/last_snapshot/backup/home/user/ # ll notes*
-rw-rw-r--  41 user  user  1453 Jun 20  2018 notes.txt

Then I let the nightly backups run.

The next morning I verify the new file gets picked up and that the inode ref count increases on the fileserver:

root@fileserver:/mnt/backintime/desktop/root/1/last_snapshot/backup/home/user/ # ll notes*
-rw-rw-r--  42 user  user  1453 Jun 20  2018 notes-hardlink.txt
-rw-rw-r--  42 user  user  1453 Jun 20  2018 notes.txt

Now it’s 42! I expected 43, but backintime purged a previous daily snapshot (by config), dropping 41 to 40 then adding 1 for new snapshot and 1 for new hard link. Magic!

So I think I now have a path forward for reorganizing my files without killing my backup storage (an ongoing concern weighing on my shoulders):

1. Duplicate files into new location, using hard links only
2. Wait for the nightly snapshot
3. Delete the old references the next day

Can it actually be that easy? Probably!

For final testing, I moved the notes-hardlink.txt to another folder, so we’ll see how it goes after tonight’s snapshot. If inode refcount increases by 1 (minus any snapshot purges), then I’ll consider it a success. Wish us luck!