You can play with it here. It retains state between hibernations, kind of like running a view count. Here you can see the preview:

However, the challenge lies in speed. Memory snapshot files are big, spanning multiple GBs. Saving a memory snapshot takes 1 second per gigabyte (so an 8GB VM takes 8 seconds to snapshot), and copying a memory snapshot takes the same time.

So if you're looking at a sandbox and press fork, we would have to:

1. Pause the VM (~16ms)
2. Save the snapshot (~4s)
3. Copy the memory files + disk (~6s)
4. Start a new VM from those files (~300ms)

Together, you would have to wait ~10s, which is faster than waiting for all dev servers to start, but it's still too slow if you want to quickly test some changes.

Just the fact that this works is incredible — cloning VMs is actually a possibility! However, we need to seriously cut down on serialisation time.

Saving snapshots faster

When we call create_snapshot on the Firecracker VM, it takes about 1 second per gigabyte to write the memory snapshot file. Meaning that if you have a VM with 12GB of memory, it would take 12 seconds to create the snapshot. Sadly, if you're looking at a sandbox, and you press fork, you would have to wait at least 12 seconds before you could open the new sandbox.

We need to find a way to make the creation of a snapshot faster, down to less than a second, but how?

In this case, we're restricted by I/O. Most time is spent on writing the memory file. Even if we throw many NVMe drives at the problem, it still will take more than a couple seconds to write the memory snapshot. We need to find a way where we don't have to write so many bytes to disk.

We've tried plenty of approaches. We tried incremental snapshotting, sparse snapshotting, compression. In the end, we found a solution that reduced our timings tenfold—but to explain it, we first need to understand how Firecracker saves a snapshot.

When Firecracker loads a memory snapshot for a VM, it does not read the whole file into memory. If it would read the whole file, it would take much longer to resume a VM from hibernation.

Instead, Firecracker uses mmap. mmap is a Linux syscall that creates a “mapping” of a given file to memory. This means that the file is not loaded directly into memory, but there is a reservation in memory saying “this part of the memory corresponds to this file on disk”.

Whenever we try to read from this memory region, the kernel will first check if the memory is already loaded. If that's not the case, it will “page fault”. During a page fault, the kernel will read the corresponding data from the backing file (our memory snapshot), load that into memory, and return it.

The most impressive thing about this is that by using mmap, we will only load parts of the file into memory that are actually read. This allows VMs to resume quickly, because a resume only requires 300-400MB of memory.

It's pretty interesting to see how much memory most VMs actually read after a resume. It turns out that most VMs load less than 1GB into memory. Inside the VM it will actually say that 3-4GB is used, but most of that memory is still stored on disk, not actually stored in memory.

So what happens if you write to memory? Does it get synced back to the memory file? By default, no. Normally, the changes are kept in memory, and are not synced to the backing file. The changes are only synced back when we call create_snapshot, which often results in saves that are 1-2GB in size. This takes too long to write.

However, there is a flag we can pass. If we pass MAP_SHARED to the mmap call, it actually will sync back changes to the backing file! The kernel does this lazily: whenever it has a bit of time on its hands, it will flush the changes back to the file.

This is perfect for us, because we can move most of the I/O work of saving the snapshot upfront. When we actually want to save the snapshot, we'll only have to sync back a little amount!

This seriously reduced our snapshot timings. Here's a graph of the average time it takes to save a memory snapshot, before and after the deployment of this change:

With this change, we went from ~8-12s of saving snapshots to ~30-100ms!

Getting the clone time down to milliseconds

We can now quickly save a snapshot, but what about cloning? When cloning a memory snapshot, we still need to copy everything byte-for-byte to the new file, which takes again ~8-12s.

But… do we really have to clone everything byte-for-byte? When we clone a VM, >90% of the data will be reused, since it resumes from the same point. So is there a way that we can reuse the data?

The answer is in using copy-on-write (CoW). Copy-on-write, like the name implies, will only copy data when we start writing to it. Our previous mmap example also uses copy-on-write if MAP_SHARED is not passed.

By using copy-on-write, we do not copy the data for a clone. Instead, we tell the new VM to use the same data as the old VM. Whenever the new VM needs to make a change to its data, it will copy the data from the old VM and apply the change to that data.

Here's an example. Let's say VM B is created from VM A. VM B will directly use all the data from VM A. When VM B wants to make a change to block 3, it will copy block 3 from VM A, and only then apply the change. Whenever it reads from block 3 after this, it will read from its own block 3.

With copy-on-write, the copies are lazy. We only copy data when we need to mutate it, and this is a perfect fit for our forking model!

As a side-note, copy-on-write has been used for a long time already in many places. Some well-known examples of CoW being used are Git (every change is a new object), modern filesystems (btrfs/ zfs) and Unix itself (two examples are fork and mmap).

This technique does not only make our copies instant, it also saves a lot of disk space. If someone is looking at a sandbox, makes a fork, and only changes a single file, we will only have to save that changed file for the whole fork!

We use this technique both for our disks (by creating disk CoW snapshots) and for our memory snapshots. It reduced our copy times from several seconds to ~50ms.

But… can it clone Minecraft?

By applying copy-on-write and shared mmaping of the memory file, we can clone a VM extremely fast. Looking back at the steps, the new timings are:

1. Pause the VM (~16ms)
2. Save snapshot (~100ms)
3. Copy the memory files + disk (~800ms)
4. Start new VM from those files (~400ms)

Which gives us clone timings that are well below two seconds! Here's a fork of Vite (you can try for yourself here):