Data Hoarding 101

I have digital backups of data, especially of my photos, going back almost a decade (I’m not that old) but it’s only recently that I got into creating time-snapshots of my backups, after I mistakenly deleted a bunch of (non-crucial) personal files with a stray rsync --delete on a directory I had not backed-up.

My home server used be compatible with macOS’s Time Machine and when I kept most things on my Mac that was useful, but I changed my data hoarding habits and that solution started to not be the optimal one.

Time Based Snapshots Using rsync

rsync has got to be one of my favourite and most used utilities. I use it constantly to transfer files between machine on my local network (and remote ones) but I’ve discovered it’s very useful as a backup tool. The following is a short script that uses rsync to create snapshots of the my data directory to another directory (they are mount points on two different drives):

#!/bin/sh

TIMESTAMP=`date "+%Y-%m-%dT%H-%M-%S"`
USER=user
SOURCEDIR=/var/data/backups
TARGETDIR=/var/redundancy

# Create new backup using rsync and output to log
rsync -avPh --delete --link-dest=$TARGETDIR/current $SOURCEDIR/$USER/ $TARGETDIR/$USER-$TIMESTAMP > /var/log/rsync/$TIMESTAMP.log 2>&1

# check exit status to see if backup failed
if [ "$?" = 0 ]; then
    # Remove link to current backup
    rm -f $TARGETDIR/current
    # Create link to the newest backup
    ln -s $TARGETDIR/$USER-$TIMESTAMP $TARGETDIR/current
else
    # Rename directory if failed
    mv $TARGETDIR/$USER-$TIMESTAMP $TARGETDIR/failed-$USER-$TIMESTAMP
fi

Essentially this script creates incremental copies of backups for a given user on the system by comparing any changes to that user’s data directory ($SOURCEDIR) with the current backup ($TARGETDIR) and making a new snapshot of any changes with the date.

Now all that remains is to save this script somewhere and create a cron job to run it on an automatic interval:

# daily at 5am
0 5 * * * bash /usr/local/bin/rsync-time-machine

Hard Links

This method uses rsync’s --link-dest option to create hard-links so that isn’t duplicative of file data (the contents are only stored once, so you don’t use twice as much) and you can delete snapshots at will. (The links do still take up some space but it’s relatively insignificant.)

Be careful when deleting old snapshots. A new full backup of should be made before further incremental ones. I have to give a shoutout to Mike Rubel whose post was helpful in coming up with this solution.