Replicating Firecracker Snapshots Across Regions

The first time you try to move a Firecracker snapshot to another region, it looks like a storage problem. You have some files in one bucket, you want them in another bucket, and there is a well-known command for that. You run it, it finishes, you feel productive, and then a machine three thousand miles away tries to restore from what you copied and produces one of the more interesting failure modes in systems work: a guest that boots, appears healthy, and is subtly wrong about the world.

I'm Ajay; I built PandaStack, which restores a baked Firecracker snapshot on every sandbox create rather than keeping a warm pool of idle VMs — the restore step lands around 49ms, end-to-end create is p50 179ms and p99 roughly 203ms, and only the very first cold boot of a template costs about 3 seconds. That model only works if every host in every region has a snapshot it can trust. Getting there took me through most of the failure modes below, including one that took a fleet down because a pointer was confidently aiming at an object that had been garbage-collected nine minutes earlier. This post is what a snapshot actually is, why replicating one is a distributed publish problem rather than a copy, and how you make multi-gigabyte memory images stop being the thing that gates your cold start.

A snapshot is not a file

Firecracker's snapshot is a set of artifacts that are only meaningful together, and the coupling between them is much tighter than the filenames suggest. At minimum you have a memory image — the guest's entire physical RAM, serialized — and a VM state file describing vCPU register state, device state, virtio queue positions, interrupt controller state, and the machine configuration. Alongside those sits the rootfs disk the guest was using when it was paused, plus whatever metadata your platform bakes on top.

The important property is mutual consistency. The memory image contains a kernel whose in-memory data structures point at devices described by the state file, which in turn describe queues whose contents reference pages in the memory image, which reference blocks on the disk. Every one of those references is a raw offset or address captured at one instant. Mix a memory image from one bake with a state file from another and you don't get a warning — you get a kernel dereferencing a pointer into memory that means something different now. Sometimes it panics immediately, which is the merciful case. Sometimes it runs.

The metadata layer is where most of the surprises live, because it's the part people forget is part of the snapshot at all:

• Machine configuration — vCPU count and RAM size are frozen at snapshot time. You cannot restore a 4 GiB memory image into a VM configured for 2 GiB, because the memory image *is* the machine's RAM. Resizing means re-baking.
• Baked guest network identity — the guest's kernel has an IP, a MAC, a gateway, and a routing table in its memory image. When you restore it somewhere else, the guest still believes all of that. The host has to bend to match, not the other way around.
• CPU feature expectations — the guest probed CPUID at boot, cached what it found, and made decisions accordingly. Those decisions are now baked into the memory image.
• Disk geometry and identity — the block device's size and layout must match what the guest's mounted filesystem expects, because the guest has cached filesystem metadata in the memory you're restoring.
• The guest's clock — a restored guest wakes up believing it's whatever time it was at bake. That's benign until it isn't, which is the day a TLS certificate rotates past your bake date and every outbound HTTPS call in the guest starts failing validation.

Why this is a publish problem, not a copy problem

Once you accept that the artifact set is atomic in meaning, the requirement follows immediately: it must also be atomic in visibility. A consumer in the destination region must see either the complete old set or the complete new set, and never a mixture. Object stores do not give you multi-object transactions, so you have to build that guarantee out of the primitives they do give you: immutable objects and a single small pointer.

The scheme that works is boring, which is a compliment. Every publish writes to a fresh, immutable generation prefix — a monotonically increasing integer, a timestamp, a content hash, whatever, as long as it is never reused. Nothing under that prefix is ever mutated after it's written. When every artifact is uploaded and verified, you flip one tiny pointer object, conventionally called `CURRENT`, to name the new generation. Readers do exactly two operations: read `CURRENT`, then read artifacts under the generation it names. The pointer flip is the atomic commit, and it's atomic because it's a single-object write.

This buys you a few things beyond atomicity. Rollback is writing an older generation number into `CURRENT`. A reader that grabbed a generation before the flip keeps reading a coherent set rather than having the ground shift under it. And because generations are immutable, aggressive caching is trivially correct — a host that has generation 47 cached locally never has to ask whether generation 47 changed, because it can't have.

The pointer race that will eat your fleet

Here's the bug I promised. A mutable pointer flipped without a compare-and-swap is a race, and at fleet scale it is not a theoretical one. The setup: several hosts each build and publish the same template, roughly concurrently, because that's what a fan-out deploy does. Host A publishes generation 47 and flips `CURRENT` to 47. Host B, which started slightly earlier and finished slightly later, publishes generation 46 and flips `CURRENT` to 46. Last writer wins, and the last writer is now the *older* generation.

On its own that's merely wasteful. It becomes an outage when you add garbage collection, which you must, because memory images are large and old generations accumulate. A reasonable-sounding GC policy is "keep the newest N generations." Generation 46 is not among the newest, so GC deletes it — and `CURRENT` is pointing at it. Now every host in every region that reads `CURRENT` gets a generation number, asks for its artifacts, and receives a 404 that your code paths interpret as an internal error. Every pull fails. Every create that needs that template fails. And the pointer is aiming, with total confidence, at an object that was deleted nine minutes ago.

Two changes make this structurally impossible rather than merely unlikely. First, the flip must be a compare-and-swap that also enforces monotonicity: read `CURRENT` with its object generation number, and write the new value conditional on that generation being unchanged, refusing the write if the new value is numerically older than what's already there. If the precondition fails, re-read and re-evaluate — someone else committed and you may not need to publish at all. Second, GC must be defined relative to the pointer, not to recency: delete only generations strictly below `CURRENT`, and never the one `CURRENT` names. Under those two rules, a pointer to a deleted generation is not a bug you fix, it's a state you cannot construct.

#!/usr/bin/env bash
# Publish a snapshot artifact set atomically: immutable generation prefix,
# then a compare-and-swap flip of CURRENT. Object-store generation numbers
# are the CAS token (GCS: --if-generation-match; S3: If-Match on ETag).
set -euo pipefail

BUCKET="gs://pandastack-seeds"
TEMPLATE="base"
BASE="$BUCKET/seeds/$TEMPLATE"

# 1. Read CURRENT *and* the object generation of the pointer itself.
#    Use `objects describe` — a formatted `ls` will not reliably give you
#    the generation, and guessing it defeats the entire mechanism.
ptr_gen=$(gcloud storage objects describe "$BASE/CURRENT" \
            --format='value(generation)' 2>/dev/null || echo "")
cur=$(gcloud storage cat "$BASE/CURRENT" 2>/dev/null || echo "0")
next=$(( cur + 1 ))

# 2. Upload every artifact under an immutable, never-reused prefix.
#    Nothing reads this prefix yet: CURRENT still names the old generation.
GEN="$BASE/$next"
gcloud storage cp vm.mem        "$GEN/vm.mem"
gcloud storage cp vm.state      "$GEN/vm.state"
gcloud storage cp clone.ext4    "$GEN/clone.ext4"
gcloud storage cp meta.json     "$GEN/meta.json"      # baked vCPU/RAM/net identity
gcloud storage cp vm.mem.header "$GEN/vm.mem.header"  # non-zero chunk bitmap

# 3. Manifest last. Its presence is the marker that the set is complete.
sha256sum vm.mem vm.state clone.ext4 meta.json > manifest.sha256
gcloud storage cp manifest.sha256 "$GEN/manifest.sha256"

# 4. CAS flip. If someone else moved CURRENT since step 1, this fails
#    loudly instead of silently clobbering a newer generation.
precond="${ptr_gen:-0}"   # 0 == "create only if absent"
printf '%s' "$next" | gcloud storage cp - "$BASE/CURRENT" \
  --if-generation-match="$precond"

# 5. GC is defined against the pointer, never against recency.
#    Delete strictly BELOW current; never the generation CURRENT names.
keep=3
for g in $(gcloud storage ls "$BASE/" | grep -oE '[0-9]+/$' | tr -d '/'); do
  if [ "$g" -lt "$(( next - keep ))" ]; then
    gcloud storage rm -r "$BASE/$g"
  fi
done

Checksums, manifests, and the half-uploaded memory image

Immutable generations stop you from mixing artifact sets. They do not stop you from publishing a set where one artifact is truncated, corrupted in transit, or silently zero-length because a disk filled up mid-upload. That's what the manifest is for, and the ordering rule is the whole trick: upload every artifact first, upload the manifest last, and treat the manifest's existence as the only signal that a generation is complete. A generation prefix without a manifest is debris, not a snapshot. Readers skip it; GC eventually removes it.

Then verify on the consuming side, before restore, not after. A truncated memory image is the worst possible thing to discover lazily. Firecracker will happily start restoring it, the guest will page in whatever the tail of the file contained, and you will spend an afternoon reading kernel oops traces trying to work out which of your own subsystems corrupted memory — when the answer is that a TCP connection got reset at 94% and nobody checked.

#!/usr/bin/env bash
# Consumer side: resolve CURRENT, fetch the set, verify BEFORE restoring.
# A snapshot that fails verification must never reach Firecracker.
set -euo pipefail

BASE="gs://pandastack-seeds/seeds/base"
DEST="/var/lib/pandastack/seeds/base"

gen=$(gcloud storage cat "$BASE/CURRENT")
src="$BASE/$gen"

# The manifest is the completeness marker. No manifest == not a snapshot,
# no matter how many bytes are sitting under that prefix.
if ! gcloud storage cat "$src/manifest.sha256" > /tmp/manifest.sha256; then
  echo "gen $gen has no manifest — incomplete publish, refusing" >&2
  exit 1
fi

stage="$DEST/.pull-$gen.$$"          # stage into a temp dir, never in place
mkdir -p "$stage"
gcloud storage cp "$src/*" "$stage/"

# Verify every artifact against the manifest.
( cd "$stage" && sha256sum -c manifest.sha256 --quiet ) || {
  echo "checksum mismatch in gen $gen — discarding" >&2
  rm -rf "$stage"; exit 1
}

# Cross-check the machine config the guest was baked with. Restoring a
# 4096 MiB memory image into a 2048 MiB VM is not a resize, it's a crash.
mem_mb=$(jq -r '.mem_size_mib' "$stage/meta.json")
actual=$(( $(stat -c %s "$stage/vm.mem") / 1024 / 1024 ))
[ "$mem_mb" -eq "$actual" ] || {
  echo "meta says ${mem_mb}MiB, vm.mem is ${actual}MiB — refusing" >&2
  rm -rf "$stage"; exit 1
}

# Publish locally by rename. Remove any stale destination FIRST: rename
# cannot replace a non-empty directory, and a crashed earlier pull leaves
# exactly that behind.
rm -rf "$DEST/$gen"
mv "$stage" "$DEST/$gen"
ln -sfn "$DEST/$gen" "$DEST/current"   # local atomic flip, same idea

That last block — clear the destination before the rename — is another small thing that ruins a Tuesday. A rename cannot replace a non-empty directory, so any interrupted earlier pull leaves a partial directory that makes every subsequent pull fail with a message about the file existing, which is technically accurate and completely unhelpful. Self-healing beats alerting: assume the previous attempt died halfway, because eventually one did.

The CPU-feature problem, and the honest answer

Here's the part that is genuinely hard, where the honest answer is a constraint rather than a clever trick. When a guest boots, its kernel and libc probe CPUID, discover which instruction set extensions are available, and specialize accordingly. Function pointers get resolved to AVX-512 implementations of memcpy. The kernel enables features based on what it saw. glibc picks optimized string routines at load time via IFUNC resolvers. All of that is decided once, at boot, and then frozen into the memory image when you snapshot.

Restore that memory image on a host with a different CPU model and the guest does not re-probe. It has no reason to — from inside, no time passed and nothing changed. It keeps calling the AVX-512 memcpy it resolved at boot, on a host whose CPU doesn't implement AVX-512, and the result is an illegal-instruction fault somewhere deep inside a library, in a stack trace that will not mention CPU features anywhere. You have handed a running program a CPU that changed underneath it, which is not a situation the x86 architecture has ever promised to handle.

The mechanism the platform gives you is CPU templates: masking the CPUID leaves Firecracker exposes to the guest so it sees a normalized, lowest-common-denominator feature set rather than whatever the host physically has. Bake the snapshot against that masked view, restore against the same masked view, and the guest's boot-time decisions stay valid everywhere. The costs are real and worth naming plainly:

• You lose performance you paid for. Masking off AVX-512 on hosts that have it means guests never use it — you are trading peak throughput for portability, deliberately.
• The mask is a compatibility contract, so it's forward-only in practice. Widening it later means re-baking every template; narrowing it after guests have specialized against the wider set means illegal instructions.
• Cross-vendor portability is mostly a fantasy. Normalizing Intel and AMD to a common feature set is not just a matter of masking flags — errata handling, microcode-dependent behavior, and topology differ. Treat vendor boundaries as hard pool boundaries.
• Architecture boundaries are absolute. An arm64 snapshot does not restore on x86_64 under any masking scheme; that's a different machine, not a different feature set.
• Whatever mask you choose, verify the specifics against Firecracker's own CPU-template documentation for your version and host generation rather than against my summary — this is an area where the details move.

The alternative, which is what many production fleets quietly do, is to skip normalization and pin snapshots to homogeneous host pools: bake per CPU generation, tag the artifacts with the host class they're valid for, and make the scheduler refuse to place a restore on a host outside that class. It's more artifacts and more bookkeeping, and it costs you fleet flexibility exactly when you want it most — during a capacity crunch, when the only free machines are the wrong generation. But it's honest, it's easy to reason about, and it fails at scheduling time with a clear error instead of at runtime with a SIGILL. Pick one of these two. Picking neither means your snapshots work until the day your provider adds a new instance generation to the pool.

The fat-artifact problem

Now the arithmetic. A memory image is the guest's entire RAM, so a template baked at 4 GiB produces a 4 GiB memory image. Multiply by templates, multiply by regions, multiply by generations you keep for rollback, and the storage bill is annoying but survivable. The latency is the actual problem: the naive replication model says a host must download the entire memory image before it can restore anything from it. A host that just came up, or an autoscaler adding capacity into a traffic spike, sits there transferring gigabytes while requests queue. Your beautiful ~49ms restore is now gated by a multi-gigabyte transfer that has nothing to do with the guest's actual working set.

And it *is* mostly unrelated to the working set. A freshly-booted guest has touched a fraction of its RAM. Much of a memory image is zero pages the guest never wrote, plus pages it touched once during boot and never looked at again. Downloading all of it to run a workload that touches a slice of it is the storage-layer equivalent of shipping a filing cabinet because someone asked for a receipt.

Demand-paged memory: userfaultfd plus range GETs

The fix is to make the memory image lazy. Linux `userfaultfd` lets a userspace process register a memory region and receive an event whenever the kernel takes a page fault there, then satisfy the fault by installing page contents with `UFFDIO_COPY`. Firecracker supports handing memory restoration to a UFFD handler over a Unix socket: it passes the file descriptor and the region mappings, and from then on every page the guest touches becomes a fault your handler answers.

Which turns replication inside out. Instead of "download 4 GiB, then restore," the host writes a small sidecar describing where the memory image lives in object storage — bucket, object, size — and restores immediately. The guest starts running. It touches a page, the kernel faults, the handler maps the fault address to an offset in the memory image, issues an HTTP range GET for the chunk containing that offset, and installs it. The guest never knows. It's paging from another region, which is a sentence that sounds worse than it behaves.

The naive version of that is too chatty, so there are three refinements that carry most of the weight. First, a zero-map: at bake time, record which chunks of the memory image are entirely zero in a small header alongside it, and serve those faults by zero-filling with no network call at all. Since a large fraction of a memory image is untouched pages, this eliminates a large fraction of potential fetches by construction. Second, a prefetch trace: record at bake time which chunks the guest touched during a normal startup, then on restore replay that set in the background so the hot pages are already resident by the time the guest asks. Third — the one that matters most on a busy host — a shared on-disk chunk cache.

The shared cache is the piece that makes this economical rather than merely clever. Chunks fetched by any restore land in a per-host, per-generation sparse file, and every subsequent restore of the same generation reads them from local disk. The first restore on a fresh host pays remote latency; every one after that is local. And because the cache is keyed by the generation's identity — a hash of bucket and object path — a re-bake produces a different key and self-invalidates, so there is no stale-cache class of bug to reason about. That's the immutable-generation discipline paying a second dividend: content that never changes is content you can cache without a coherence protocol.

One durability detail that's easy to get wrong: the cache's "I have this chunk" bitmap must only be advanced after the chunk data is durably on disk — write, fsync, then flip the bit, with the bitmap updated atomically. Do it in the other order and a host that loses power mid-write comes back believing it has a chunk it actually has as a hole, and serves a page of zeros into the middle of a guest's kernel. Chunk size is a design choice rather than a measured optimum: PandaStack uses 4 MiB, which is large enough that per-request overhead amortizes and small enough that a single fault doesn't drag in a lot of unrelated memory. Tune it against your own object store's behavior, not against my number.

• Time to first instruction — Full artifact replication: gated by transferring the entire memory image before restore can begin. Demand-paged streaming: gated by the restore itself; pages arrive as the guest asks for them.
• Bytes moved per host — Full artifact replication: the whole image, including every zero page the guest never touched. Demand-paged streaming: roughly the working set, minus everything the zero-map answers locally.
• Local disk per template per region — Full artifact replication: a full copy of every generation you keep. Demand-paged streaming: a sidecar plus a bounded, LRU-evicted chunk cache you size deliberately.
• Behavior on a cold host — Full artifact replication: worst case, a full transfer before the first sandbox. Demand-paged streaming: first restore pays remote latency on its faults; subsequent restores of the same generation hit local disk.
• Failure mode — Full artifact replication: fails early and loudly at pull time, which is genuinely nice. Demand-paged streaming: an object-store failure surfaces as a page fault that can't be satisfied, mid-execution, so it needs retries and a real error path.
• Operational complexity — Full artifact replication: a copy command and a checksum. Demand-paged streaming: a UFFD handler, a chunk resolver, a shared cache with crash-safe bookkeeping. Not free.
• Rootfs disk — Full artifact replication: local, because copy-on-write requires it. Demand-paged streaming: also local, for exactly the same reason. Streaming solves memory, not disk.

That last row is the constraint people miss, so it deserves its own paragraph.

The rootfs stays local

Streaming memory is possible because userfaultfd gives you a fault handler for anonymous memory. The rootfs has no equivalent free lunch, because of what you want to do with it: copy-on-write. A fast create clones the template disk with a reflink or a device-mapper snapshot, so the clone costs metadata rather than gigabytes and the shared blocks are only copied when written. Both of those mechanisms require a real local block device or a filesystem that supports reflinks locally. You cannot reflink an HTTP endpoint.

So the artifact model ends up asymmetric, and that asymmetry is a design decision rather than an oversight. The rootfs template disk is replicated eagerly and lives on local disk in every region, because CoW demands it. The memory image is streamed lazily, because it's the fat artifact and userfaultfd gives you the hook. Getting a demand-paged rootfs too is possible — you'd be building a block device backed by remote reads, via something like ublk or NBD — but that's a real project with real failure modes, not a config flag, and it should be justified by numbers you measured rather than by symmetry being aesthetically pleasing.

What this looks like from the consuming end

The point of all this machinery is that none of it is visible to the person creating a sandbox. They ask for a machine on a template; a scheduler picks a host in some region; that host resolves the current generation, restores from a snapshot it can prove is intact, and pages memory in as the guest touches it. What comes back is a running VM, and the interesting number is that a fresh machine is cheap enough to be the default rather than a thing you ration.

from pandastack import Sandbox

# Every create is a snapshot restore — there is no warm pool of idle VMs
# waiting around. The template's artifact set was published as an immutable
# generation, verified against its manifest, and is streamed on demand.
sbx = Sandbox.create(template="base", ttl_seconds=600)

# The guest resumed exactly where the bake left off. Its uptime is a lie
# it believes sincerely: from inside, no time passed between the snapshot
# and now, which is also why its clock needs a nudge on restore.
print(sbx.exec("uptime -p").stdout)
print(sbx.exec("date -u").stdout)

# Memory the guest touches now is faulted in from object storage behind
# the scenes. Nothing in the guest can tell the difference; a page fault
# is a page fault whether it's answered from RAM, disk, or a range GET.
r = sbx.exec("python3 -c \"print(sum(range(10_000_000)))\"", timeout_seconds=60)
print(r.exit_code, r.stdout.strip())

# Confirm the machine size is the *baked* size, not something you asked for.
# vCPU and RAM are frozen in the memory image; the host matches the snapshot.
print(sbx.exec("nproc && free -m | awk '/Mem:/ {print $2}'").stdout)

sbx.kill()

There's a scheduling wrinkle worth mentioning, because it shows how the replication design leaks into placement. If some hosts can stream memory and some can't, they aren't interchangeable: a streaming host can start restoring immediately while a non-streaming host may first have to pull a full memory image. So the scheduler wants a small preference for streaming-capable hosts, on top of the usual free-CPU and free-memory terms. It's a tiebreaker, not a dominant term — you still want load spread — but it's the kind of thing that only becomes obvious after you've watched a create get scheduled onto the one host that had to download everything first.

The failure modes in snapshot replication are all the same shape: something believed a thing about the world that was true at bake time and isn't true now. A pointer to a deleted generation, a guest expecting an instruction the host doesn't have, a clock frozen before a certificate rotated. Design against staleness, not against copy errors.

The short version

If you're building this yourself, the parts that are load-bearing rather than nice-to-have:

1. Treat the artifact set as one indivisible unit. Memory image, VM state, disk, and metadata are only meaningful together; there is no valid partial set.
2. Write to immutable, never-reused generation prefixes. Nothing is ever mutated in place, which makes caching correct for free.
3. Flip a single pointer to commit, and flip it with a compare-and-swap that enforces monotonicity. Last-writer-wins on a mutable pointer is the outage.
4. Garbage-collect strictly below the pointer, never by recency. A pointer to a collected generation should be an unconstructable state, not a postmortem.
5. Upload the manifest last and treat its presence as the completeness marker. Verify checksums before restore, not after — a truncated memory image fails as memory corruption, not as an I/O error.
6. Decide your CPU compatibility story at bake time: normalize the feature set with a CPU template, or pin snapshots to homogeneous host pools. There is no restore-time fix.
7. Make the memory image lazy — userfaultfd plus range GETs, a zero-map, a prefetch trace, and a shared per-host chunk cache with crash-safe bookkeeping.
8. Keep the rootfs local. Copy-on-write needs a block device, and no amount of streaming changes that.

Other platforms solve some subset of this, and several of them do it well; check their documentation for the specifics rather than trusting my characterization, since this is an area where implementations differ meaningfully and change between versions. But the underlying constraints aren't vendor-specific. Any system that restores a saved machine somewhere other than where it was saved has to answer the same three questions: how do you publish a set of files atomically to something with no transactions, how do you keep a running guest from discovering that its CPU changed, and how do you avoid moving gigabytes you're never going to read. Get those right and cross-region snapshot replication is boring. Boring is the goal. Boring is what lets a machine on the other side of the planet come up in milliseconds and never find out how far it traveled.