Backups aren't simple

See also: John Salvatier’s excellent blog, Reality has a surprising amount of detail

I read a comment somewhere that stuck with me, that went something like this:

“There are two types of people: those who have suffered a catastrophic loss of data, and those who will.”

Trying to find the source for it for this blog, it turned out that every other sysadmin has his rehashed version of the quote, but the gist of it is the same everywhere. Data loss is something that happens more often than we’d hope, and most of us are woefully unprepared for when it hits us (which is almost always at the worst possible time).

I can confirm that I had a similar experience once. When I was little we had pulled all our family photos from our home laptops and PCs onto an external hard drive, in order to free up some space. This worked beautifully until one day my dad wanted to use the drive as storage for our TV set-top box (one of these old things), and was prompted to format the drive. He went ahead with it, and the disk was reformatted. The index of files was deleted, and we were stuck with a nominally empty drive.

It would be easy to blame him for screwing up, but it takes beginning a career in tech to realise that there is a series of errors that lead to this kind of mistake. Firstly, we had put all of our photos in one place and didn’t bother with backups. Secondly, most consumer-facing software usually has bold disclaimers telling you that formatting a disk means losing data (which the set-top box didn’t, terrible UI). Besides, why would you even expect a non-technical person to even have to know any of this?

Thankfully we were able to get the photos restored, and it turned out to be a cheap lesson in handling data. You never keep important things in one place only. There’s about a million things that can go wrong. Your drive could die, it could be stolen, bits could rot in cold storage (hard drives have magnetic particles which can inexplicably shift, and SSDs are made of NAND transistors which leak electricity and over time, corrupt your data).

So our first principle is to have a backup, i.e. a copy of your files someplace else. So far so good.

This doesn’t cover the headaches of what a plugged in drive could do. Ransomware could encrypt your files, and you could do anything from an honest mistake like deleting the wrong file; up to catastrophic mistakes like running a script that overwrites everything with zeroes.

So our backup should not be a mirror of the first drive, because we also want to be able to go back in time if we mess up. Importantly, this means that mirroring your disk with something like RAID 1 is out. We need some other method that snapshots things.

How often do we want to take snapshots? Maybe in our case with the photos we should have run a backup every week. If we lose 6 days and 23 hours of data, that’s fine and we can live with it. This is what’s called a Recovery Point Objective (RPO) in IT, and in real cases, it ranges from <30 seconds for critical financial institutions which really can’t afford to lose data, to 24 hours or more for some small enterprises (if they even have a disaster recovery strategy).

Taking snapshots means that we have an increasing burden on our storage. With an RPO of 24 hours, you will end up having 7 snapshots per week. 30 per month. 365 per year, if you really don’t go and prune your snapshots. So you need to rotate your backups.

Let’s say I go with the naive approach and decide to keep 14 days’ worth of snapshots. When I take a new snapshot, I delete the oldest one and I add the new one. Pretty simple, but this now forces me to have a watchful eye. Maybe I keep lots of data and can’t be bothered to check if something got corrupted in the past two weeks? But then again, I can’t just store a year’s worth of backups and they’re simply not relevant to me. What happened between day 2 and day 3 of the year has almost no significance when it’s day 364. So the granularity at which we take backups must change. The closer we are to today, the more frequent the snapshots. The further back, the less frequent the snapshots.

So maybe we rotate our daily backups every 14 days, but also take weekly backups that we rotate every 7 weeks, and monthly backups we rotate every 12 months. This should be much more efficient. But again our complexity grows. We now have something called a GFS-rotated, snapshot-based backup. This list of adjectives will continue growing, as we’ll see in a bit.

Maybe then you take a look at how MPEG compresses video, and get fascinated by how a calm scene in a movie, where the protagonist speaks but otherwise doesn’t move against a completely still background can be used for compressing video. You notice that videos are composed of frames that are mostly similar to each other, only changing with a certain movement that can be represented as a vector for a fraction of the storage. Which leads you to the very logical conclusion that your snapshots also follow the same pattern! Even more, it turns out that file changes follow a fat-tailed distribution, so over a given period there are a vast majority of files that don’t get changed at all, and a very tiny minority that change all the time.

So it becomes obvious that we shouldn’t store identical copies of files, but rather deduplicate. We can use hard links when we need to reference an already existing file. This way, we store one file on disk, and then reference it from each of our snapshots. This also survives backup rotation because we never delete files, we only delete directory entries. This exact approach is used by rsnapshot, and is best described as an incremental backup, because we store only the changes between two adjacent snapshots, instead of all the changes since the latest full backup (these are called differential backups and are more robust when restoring, but I won’t get into it for the sake of brevity).

The savings in storage are not the only benefit we get from doing this. We briefly mentioned in the beginning that we don’t store everything on a single machine. Obviously, there is also networking involved in this process, since we need to actually transfer the files from one machine to another. Deduplicated backups save a lot of bandwidth, which is especially important if you use a cloud service as your second machine. It directly affects you financially.

To sum up, by this time we have created an incremental, deduplicated, GFS-rotated, snapshot-based backup. We can use rsync to pull the files from the main machine, and cronjobs to run our backup scripts. We can run backups on as many machines as we’d like, and adding another one is trivial. Even better, file metadata is preserved, so things like access permissions and file ownership are fine.

Motivated by our success in developing this solution, we try to use it to backup the homelab with its 10 Docker containers. But later we find out from logs on the individual machines that backups are failing. The reason being that many Docker containers like to create root-owned files, and if you’re not careful you can create a cronjob running as the default user.

To make matters worse, almost every web app uses a database of some kind. Databases sometimes like to store things in-memory and flush them to disk in batches to improve performance. This practically means that restoring from backup will fail due to data corruption if we are unlucky. So we make the backup also dump the databases, and give it full filesystem permissions on our Docker volumes. That should make it work!

Then you read about incidents in which a model of hard drive had famously high failure rates, and start to wonder if you should maybe store your backups on two machines with different types of media. That way a hardware-specific failure would be unlikely to wipe out your backups. And while we’re on the topic of physical security, have one offsite backup on the cloud or a machine at a family member’s house. This way you make it really unlikely that a power surge, flood or fire will destroy everything. This is where the 3-2-1 backup gets its name: 3 copies, on 2 different types of media, with 1 offsite.

Let’s say you decide on a cloud provider for your offsite backup. Specifically object storage like Amazon S3. You quickly find out that our current setup won’t work because one, files lose their metadata when you upload them to S3, and two, the price for uploading many small files to S3 is punitively high. (file sizes also follow a fat-tailed distribution) These two facts make it best for you to stick many files into a tarball. That way you retain both your file metadata as well as your low costs. But the question is, how do you do that? Do you stick everything in one giant tarball? Obviously not, then your incremental backups with the hard links stop making sense. Best to split everything in clean 50MB chunks, but good luck with doing that in a way that is verifiably safe!

Up until this point, rolling your own backups sounded like something you should be able to do in an afternoon, but this is where I’d give up. It simply isn’t worth the mental load to do all this. Instead, you just use tried and tested tools like Borg or Restic which handle all this and much more (encryption, chunk-level deduplication, checksums). And just give your utmost thanks to the wonderful open-source community for building, maintaining, and live-testing these tools, while respecting how much trial and error was necessary to get to the point where all this complexity is abstracted away for us.

Obviously, none of this is worth anything if you don’t actually test restores. So that’s also a little digital hygiene article that you will have added to your to-do list. So, as long as you run restores every 6 months, you can enjoy your:

encrypted, chunk-level deduplicated, GFS-rotated, point-in-time archived, cloud, 3-2-1 backup solution

Also make sure not to run backups at 2AM or 3AM, or things may get scary.