Firecracker Snapshot Version Compatibility & Cross-Version Restore

A Firecracker snapshot looks like a portable blob — three files you can copy to any host and restore. It is not. A snapshot is a serialized capture of a very specific machine: a specific VMM version's view of device and CPU state, guest RAM that was populated by a specific guest kernel, and a set of CPU features the guest was told it had. Move any of those variables and restore can fail loudly, or — worse — succeed and then have the guest fall over minutes later when it executes an instruction the new host doesn't actually support. The mental model that survives contact with production: a snapshot is a photograph of a machine that only develops correctly in the same darkroom. This post is about what "the same darkroom" means — the versioned snapshot format, the CPU-template angle, kernel considerations — and the operational reality that when you upgrade Firecracker or roll out new hardware, existing snapshots may become un-restorable and have to be re-baked.

I'm Ajay — I build PandaStack, an open-source Firecracker microVM platform whose entire create path is snapshot-restore. Every sandbox, database, and app is a restore of a baked snapshot, so "can this snapshot restore on that host?" isn't a trivia question for us; it's the difference between a fleet that boots and a fleet that 500s. Everything below is the internals plus the operational discipline we learned the boring way: pin the FC version, pin CPU templates across heterogeneous hardware, treat snapshots as version-coupled artifacts, and keep a re-bake pipeline you can pull the trigger on.

Snapshots encode a versioned serialization format

When Firecracker takes a snapshot, it serializes the VMM's in-memory state — device model, vCPU registers, MSRs, interrupt controller state, memory-region layout — into `vm.state`, alongside the guest RAM image (`vm.mem`) and the rootfs. That `vm.state` is not a freeform dump; it's written in a versioned format so a future Firecracker can read a snapshot an older one produced. Historically Firecracker built this on a versionize/serde-style serialization layer, and each Firecracker release advertises which snapshot data-format version it emits and which it can consume. The key consequence: a snapshot carries a format version, and restore is a compatibility check between that version and the Firecracker binary you're restoring with.

The two rules people get backwards are forward and backward compatibility, so state them precisely. Backward compatibility is the useful direction: a newer Firecracker can generally restore a snapshot produced by an older Firecracker, within a documented support window. Forward compatibility is the direction that bites: an older Firecracker generally cannot restore a snapshot produced by a newer one, because it doesn't understand fields the newer format added. And even backward compatibility is bounded — it's not "any old snapshot forever," it's a supported range that each release documents and that eventually drops old versions. Do not memorize a matrix from a blog post (including this one). Verify the exact forward/backward support window against the Firecracker snapshot-support / versioning docs for the specific versions you're running.

The CPU-template angle: the silent killer

The format-version check is the compatibility problem people expect. The one that actually takes fleets down is subtler, because it isn't about Firecracker's version at all — it's about the physical CPU under the host. When a guest boots, it queries CPUID to learn which instruction-set extensions it has: AVX2, AVX-512, BMI, specific feature bits, and so on. Software inside the guest — the libc, the JIT, numpy's dispatcher, a Go binary's runtime detection — reads those bits once and then happily emits instructions that rely on them. A snapshot freezes that advertised feature set. The guest already believes, permanently, that it has AVX-512 (or doesn't).

Now restore that snapshot on a different host. If the bake host had AVX-512 and the restore host is an older or different microarchitecture that lacks it, the guest is a ticking bomb: it was told it had AVX-512, some library cached that fact, and the moment that code path runs an AVX-512 instruction the CPU faults with an illegal-instruction (SIGILL) and the process — or the guest — dies. Nothing at restore time necessarily errors. The snapshot loads, the vCPUs resume, everything looks green, and then a workload crashes ten minutes later on an instruction the silicon never supported. This is the single most confusing snapshot-restore failure mode, precisely because restore succeeded.

The fix is a CPU template. Firecracker lets you mask and normalize the CPUID the guest sees — either a static template you author (an explicit allow/deny of feature bits) or a normalization that presents a lowest-common-denominator view across your hardware. You choose a template that only advertises features present on every host you'll ever restore on, bake the snapshot with that template applied, and configure restore hosts to enforce the same template. Now the guest's frozen belief about its CPU is true everywhere, because you deliberately made the advertised feature set the intersection of your whole fleet — not whatever the bake host happened to have. On a homogeneous fleet you can sometimes get away without this; the day you add a second CPU generation, or a cloud provider silently reschedules you onto different silicon, an un-templated snapshot becomes a landmine.

# 1) Know exactly which Firecracker you baked with. The snapshot's data-format
#    version is a property of THIS binary's release.
$ firecracker --version
Firecracker v1.16.0

# 2) The snapshot's format version lives in vm.state, and restore is a
#    compatibility check against the RESTORING binary's supported range.
#    Treat the FC version as part of the snapshot's identity: store it in the
#    snapshot's manifest/metadata so you can refuse an incompatible restore
#    up front instead of discovering it when /snapshot/load 500s.
#    -> Verify the exact forward/backward window in the FC snapshot-support
#       docs for YOUR version. There is no universal matrix.

# 3) CPU template: bake and restore against the SAME normalized CPUID so a
#    heterogeneous fleet advertises one consistent feature set. A static CPU
#    template is applied at boot-config time (machine-config / cpu-config),
#    then baked into the snapshot; restore hosts must enforce the same one.
#    Templating to your fleet's INTERSECTION of features is what stops the
#    later illegal-instruction (SIGILL) crash on a lower-spec restore host.

Guest-kernel and host-kernel considerations

Two more variables travel with a snapshot, and both are easy to forget because they usually just work until they don't. The first is the guest kernel. The guest RAM in `vm.mem` was populated by a specific guest kernel that had already probed its virtual hardware, set up its page tables, and pinned its idea of the device layout. You do not swap the guest kernel underneath a snapshot — a restore reinstates the exact guest that was running, kernel and all. So "upgrade the guest kernel" is not a live operation on existing snapshots; it's a reason to re-bake. Change the kernel that backs a template and the old snapshots are snapshots of the old kernel, full stop.

The second is the host kernel and KVM. Restore hands serialized vCPU and device state back to KVM on the restore host, and KVM's acceptance of MSRs, interrupt-controller state, and CPU features is a function of the host kernel version and the physical CPU. A snapshot that restores cleanly on one host-kernel/KVM combination can be rejected on another that handles a particular MSR or feature differently. In practice this means your restore fleet wants a controlled, consistent host-kernel and KVM baseline, not a grab-bag of whatever each host happens to run. The CPU template handles the feature-advertisement half; a consistent host-kernel/KVM baseline handles the acceptance half. Both are verify-against-your-versions territory — don't assume a snapshot that works on your dev box works on a differently-patched production host.

Treat snapshots as version-coupled artifacts

Put the pieces together and a snapshot stops looking like a portable file and starts looking like a build artifact with a dependency set. A given snapshot is only valid against a compatible Firecracker version, a CPU whose CPUID satisfies the template it was baked with, a matching guest kernel, and a host-kernel/KVM baseline that will accept its serialized state. That's four coupling axes, and every one of them is a version you can bump out from under yourself. The discipline that keeps this sane is to treat the snapshot exactly like a compiled binary: it was built against a toolchain, and if you change the toolchain you rebuild.

• Pin the Firecracker version. Run one known FC version across bake and restore, and record it in the snapshot's metadata. Upgrading Firecracker is a deliberate migration, not a rolling `apt upgrade` you do host-by-host — because a newer FC restoring old snapshots is fine, but old FC on a host that hasn't upgraded yet cannot restore snapshots the newer bakers already produced.
• Pin a CPU template for any heterogeneous fleet. Advertise the intersection of features across every host you might restore on, bake with it, enforce it on restore. This is the single highest-leverage step against silent SIGILL-later failures. The moment you add a second CPU generation, this stops being optional.
• Pin the guest kernel per template. The kernel is baked into the snapshot; changing it means re-baking. Version the template so 'which kernel is in this snapshot' is answerable from metadata, not archaeology.
• Keep a controlled host-kernel / KVM baseline on restore hosts. A differently-patched host can reject state another accepted. Consistency here is a reliability feature, not a nicety.
• Store the compatibility metadata with the snapshot. FC version, CPU template id, guest-kernel version, format version — in the manifest. Then restore can refuse an incompatible pairing up front instead of failing deep in `/snapshot/load` (or, worse, succeeding and crashing later).

The operational reality: re-baking on upgrade

Here's the part nobody wants to hear on upgrade day: when you bump Firecracker past the compatibility window, swap the guest kernel, change the CPU template, or roll out hardware that no longer satisfies an old snapshot's advertised CPUID, the existing snapshots may become un-restorable. There is no in-place migration that rewrites a stale snapshot into the new world — a snapshot is a frozen capture, and the honest answer is to re-bake: cold-boot the template on the new toolchain (new FC, new kernel, new CPU template), let it reach a ready state, and capture a fresh snapshot. The old snapshots are retired. This is why 'just upgrade Firecracker' is never just an upgrade; it's an upgrade plus a re-bake sweep across every template you serve.

The good news is that once you accept snapshots are version-coupled artifacts, the re-bake pipeline is a normal build pipeline. You want it to be a button, not a fire drill. On PandaStack, re-baking a template invalidates the existing snapshots for it by publishing a new generation, and the create path handles the cutover: auto-bake triggers on the first spawn of a template that has no valid current snapshot, so the very first create after an upgrade pays a one-time cold boot (around 3s) to produce the fresh snapshot, and every create after that is back to restore prices — a restore lands around 49ms, with an end-to-end create at roughly 179ms p50 and 203ms p99. The re-bake isn't a special emergency mode; it's the same cold-boot-then-capture path the platform already uses to make the very first snapshot.

# The re-bake pipeline, conceptually. Trigger it whenever ANY coupling axis
# moves: Firecracker version, guest kernel, CPU template, or restore hardware.

# 1) Bring up the new toolchain on the bake host:
#      - pinned Firecracker version (the one your restore fleet also runs)
#      - new guest kernel (if that's what changed)
#      - CPU template = intersection of features across the whole fleet

# 2) Cold-boot the template to a ready state, then capture a fresh snapshot
#    against the new toolchain. This is a one-time ~3s cold boot per template.

# 3) Publish it as a NEW generation and flip the CURRENT pointer atomically.
#    Old snapshots are retired, not migrated. Re-baking a template
#    invalidates its existing snapshots by design.

# 4) Restores of the new generation are back to ~49ms restore / ~179ms p50
#    create. On PandaStack, auto-bake triggers on the first spawn if a
#    template has no valid current snapshot, so the cutover is self-healing:
#    first create eats the cold boot, every create after is restore-fast.

Why a snapshot-restore platform pins all of it

PandaStack's create path has no warm pool of idle VMs — every sandbox, database, and app is a snapshot-restore. That makes snapshot compatibility a first-class concern rather than a footnote, because a snapshot that can't restore isn't a degraded experience, it's a failed create. So we pin the Firecracker version across the whole fleet, bake and restore against a CPU template so a create can land on any agent regardless of which physical CPU that box has, version guest kernels per template, and keep the re-bake pipeline as a routine generation-publish with an atomic current-pointer flip. When we upgrade Firecracker or roll new hardware, we upgrade the restoring agents first, then re-bake, then let auto-bake self-heal any template on its first spawn.

The cross-host story is where the CPU template earns its keep most visibly. A fork can land on a different agent than its parent — a same-host copy-on-write fork is 400–750ms and a cross-host fork is 1.2–3.5s — and a scheduler that spreads load will happily place a restore on whichever agent has capacity, across a fleet where each agent pre-allocates 16,384 /30 subnets for its VMs. That only works because the advertised CPUID is templated to be identical everywhere; without it, 'schedule anywhere' would mean 'restore onto silicon that doesn't match the snapshot's frozen CPU beliefs,' which is the SIGILL-later trap at fleet scale. Pinning FC + CPU templates + a re-bake pipeline isn't ceremony. It's the thing that lets a snapshot baked once restore correctly on any host, any time, which is the whole promise of a restore-on-create platform.

The honest bottom line

A Firecracker snapshot is not a portable blob; it's a version-coupled artifact with four dependency axes. It carries a versioned serialization format, so a newer Firecracker can generally restore an older snapshot but not the reverse, within a support window you must verify per version. It freezes the guest's advertised CPUID, so restoring on a CPU that lacks a baked-in feature can crash the guest with an illegal-instruction fault long after a green restore — which is why CPU heterogeneity is the #1 silent failure and a CPU template pinned to your fleet's feature-intersection is the fix. It's tied to its guest kernel and to a host-kernel/KVM baseline that will accept its state. So pin the FC version, pin CPU templates across heterogeneous hardware, version guest kernels, upgrade restorers before producers, and keep a re-bake pipeline you can trigger without ceremony — because when the toolchain moves, existing snapshots may become un-restorable and have to be re-baked. Verify every specific version and feature claim against the Firecracker snapshot-support and versioning docs for the exact versions you run. A snapshot only develops correctly in the same darkroom; your job is to keep the darkroom the same everywhere.