Restoring a Firecracker Snapshot on a Different CPU

Here's the failure that teaches everyone this lesson exactly once. You bake a Firecracker snapshot on a shiny new host — say an Ice Lake box with AVX-512 — publish it, and later an agent on an older machine without AVX-512 restores it. The restore succeeds. Every log line is green. Then, minutes later, the guest confidently issues an AVX-512 instruction the new host has never heard of, and the illegal-instruction trap has opinions: SIGILL, no recovery, a dead process (or a dead guest) with a stack trace that points at a memcpy that has no business crashing. Nothing at restore time warned you. This post is specifically about that portability problem — restoring a snapshot on a *different* CPU than the one it was baked on — and the one tool that fixes it: a CPU template that normalizes the CPU the guest sees down to a baseline every host in your fleet can satisfy.

I'm Ajay — I build PandaStack, an open-source Firecracker microVM platform whose entire create path is snapshot-restore. Every sandbox is a restore of a baked snapshot onto whichever host the scheduler picks, so "will this snapshot survive on that box's CPU?" isn't trivia for us; it's the difference between a fleet that runs and a fleet that SIGILLs an hour into a customer's job. If you want the generic tour of what CPU templates are — static templates, custom-JSON modifiers, the MSR side — read /blog/firecracker-cpu-templates-explained first; this post deliberately doesn't rehash that. It goes deeper on one axis only: portability across heterogeneous silicon, the vendor caveat nobody wants to hear, and why the decision has to be made at bake time.

Why a guest freezes its CPU detection at boot

To understand why a snapshot can't just move to a different CPU, you have to understand that CPU feature detection is a one-time boot-time act, and its results get cached deep inside the guest — permanently, for the life of that VM's memory.

When a guest boots, software inside it runs the CPUID instruction to ask the processor what it can do: does this chip have AVX2? AVX-512? SHA extensions? BMI2? RDRAND? CPUID answers with long bitfields of feature flags plus the vendor string and family/model/stepping. The guest kernel reads those flags once, at startup, and wires itself up accordingly — it picks an AVX-512 memcpy over a scalar one, enables instruction sets, sizes structures, sets function pointers. glibc does the same with its IFUNC resolvers: the first time a program calls memcpy, the dynamic linker resolves it to whichever implementation matches the CPUID it saw, and that choice is baked into the GOT for the rest of the process's life. numpy's dispatcher, a JIT, a Go binary's runtime detection — they all probe once and cache.

None of them ever re-probe. Why would they? On real hardware the CPU doesn't change underneath a running kernel. So the guest's entire picture of "what CPU am I on" is a set of decisions made in the first second of boot and then frozen into live memory as function pointers, jump tables, and cached feature words. A snapshot captures exactly that live memory. The moment you snapshot, the guest's belief about its CPU — and every code-path choice it made on the strength of that belief — is frozen into the artifact.

The illegal-instruction crash: restore succeeds, guest dies later

Now play the tape forward. You baked on a host with AVX-512, so the guest's frozen memory contains an AVX-512 memcpy pointer, an AVX-512 code path selected in some hot loop, a cached feature word that says "AVX-512: yes." You restore that frozen memory on a host whose CPU lacks AVX-512.

Restore itself does not fail. Firecracker loads the memory image, hands the vCPU state back to KVM, resumes the vCPUs — and none of that touches an AVX-512 instruction. The machine comes up. It passes a health check. It serves a few requests. Then a workload finally hits the code path that was compiled to assume AVX-512, the vCPU executes a 512-bit vector instruction the silicon literally cannot decode, and the CPU raises #UD — an illegal-instruction fault. In the guest that surfaces as SIGILL, and the process (or, if it's the kernel's path, the guest) is gone. No recovery, because the instruction stream itself is now invalid for this hardware.

This is the single most confusing snapshot-restore failure mode precisely because restore reported success. You go looking for a bug in the workload, in the memcpy, in the library — and there isn't one. The bug is that you moved a frozen set of CPU assumptions onto a CPU that doesn't honor them. The direction matters, and it's asymmetric: baking on a *lesser* CPU and restoring on a *richer* one is generally fine (the guest just never uses the extra instructions it doesn't know exist), but baking on a richer CPU and restoring on a poorer one is the landmine. Since your scheduler can't guarantee it always lands the restore on a host at least as capable as the bake host, the only safe move is to make the bake host's advertised feature set no richer than the poorest host in the fleet. That's what a template does.

How a CPU template makes the snapshot portable

A CPU template is Firecracker masking and normalizing the CPUID (and a set of MSRs) that the guest is allowed to see, so instead of passing through the physical host's real feature set, the guest sees a curated, deliberate one. Applied at boot, before the guest ever runs CPUID, it means the guest's one-time feature detection reads the template's baseline — not the silicon's.

The portability trick falls straight out of that. Pick a template that advertises only the features present on *every* host you might ever restore on — the intersection of your whole fleet's capabilities, the lowest common denominator. Bake the snapshot under that template. The guest, probing at boot, sees (say) no AVX-512 because the template hid it, so it never selects an AVX-512 code path, never caches an AVX-512 memcpy pointer, never freezes an instruction into memory that some host in the fleet can't run. Now the frozen guest's beliefs about its CPU are true on every host — because you deliberately made them true everywhere in advance. The snapshot became portable by construction, not by luck.

Selecting a static template is one field on the machine-config you PUT before the guest boots. The name (T2, C3, and the rest) encodes the baseline; verify the exact feature set of each against the Firecracker CPU-templates docs for your version, because it evolves:

// PUT /machine-config — before InstanceStart, BEFORE the guest runs CPUID.
// T2 pins the guest to a Skylake-era Intel baseline (notably no AVX-512),
// so a snapshot baked here restores across every Intel host generation
// in the fleet — the guest never learns AVX-512 exists, so it can never
// freeze an AVX-512 instruction into memory that a poorer host can't run.
{
  "vcpu_count": 2,
  "mem_size_mib": 2048,
  "smt": false,
  "cpu_template": "T2"
}

You can watch the template do its job from inside two guests. Boot the same template on two physically different hosts, read the advertised flags, and — with a correct template — they match, even though the bare metal doesn't:

# Inside a guest: what feature flags did this VM detect at boot?
# /proc/cpuinfo reflects exactly what CPUID advertised through the template.
$ grep -o -m1 -E 'avx512[a-z]*|avx2|avx|sha_ni' /proc/cpuinfo | sort -u
avx
avx2
#   ^ note: NO avx512 line — the T2 template hid it, so the guest never
#     detected it and never compiled itself to depend on it.

# The same template booted on a DIFFERENT physical host must show the
# identical set. If host A (with AVX-512) and host B (without) both boot the
# T2 template and both omit avx512 here, the snapshot is portable between them.
# If host A leaks avx512 into the guest, that snapshot is a landmine on B.

# (`cpuid -1` gives the raw leaves if you want to inspect bit by bit.)

Static templates vs. custom templates for portability

There are two ways to get that normalized baseline. Static templates are Firecracker's built-in named baselines — you pick one by name and Firecracker owns the CPUID/MSR masks it implies (T2 for a broad Intel baseline, C3 for an older/more conservative Intel one, and the AMD and mitigation-flavored variants covered in the generic post). Custom templates are JSON you write yourself: lists of CPUID and MSR modifiers that set or clear specific bits, giving per-bit control down to your fleet's exact intersection. Static is right when your hardware matches a named shape; custom is the escape hatch when the menu doesn't fit your silicon mix and you need to mask precisely the features your poorest host lacks and no more.

For the portability decision specifically, the trade-offs line up like this:

• No template (bare host CPUID pass-through) — Portability: none across mixed CPUs; the snapshot is pinned to hosts with the exact feature set of the bake host. A: newer instructions — the guest gets every feature the bake host has, so single-machine speed is maximal. B: complexity — zero config, but the portability cost is a #UD crash the first time the scheduler lands a restore on a poorer host. Only safe on a genuinely homogeneous fleet (one CPU model, one microcode).
• Static template (e.g. T2, C3) — Portability: high across whatever shape the name targets (all-Intel, older-Intel-inclusive, Intel+AMD via the AMD variant). A: newer instructions — hidden down to the named baseline, so you lose the freshest vector/crypto paths on your newest silicon. B: complexity — one string in machine-config, Firecracker owns correctness; the cost is you inherit whatever baseline the name implies, tightest-fit or not.
• Custom template (CPUID + MSR modifiers in JSON) — Portability: highest and most precise — you mask down to exactly your fleet's intersection and no lower. A: newer instructions — you keep every feature every host shares and hide only the ones some host lacks, so you give up the least performance for the portability you need. B: complexity — you own the correctness; a wrong bitmap is a subtly broken or unbootable guest, and you re-derive the template whenever the hardware fleet changes.

The through-line: every step toward portability is a step away from the newest instructions on your best hardware. Masking AVX-512 to make a snapshot run everywhere means a vector-heavy workload takes a narrower path on the boxes that *do* have AVX-512. That's a real, measurable cost for numerical or encoding code and a rounding error for orchestration code — which is exactly why the choice has to be deliberate and workload-aware, not a default you back into.

The Intel↔AMD caveat: templates don't make vendors interchangeable

Here's the caveat that surprises people who assume a template is a universal solvent: masking feature flags across Intel and AMD is generally *not* enough to make a snapshot freely portable between the two vendors, and you should treat cross-vendor portability as unsupported unless the Firecracker docs for your version explicitly say otherwise for the specific template you're using.

The reason is that a CPU's identity to a guest is more than its feature bitfields. CPUID also reports the vendor string ("GenuineIntel" vs "AuthenticAMD"), the family/model/stepping, and vendor-specific leaves; and the MSR layout — which registers exist, what bits mean — differs between vendors in ways that aren't just "a flag is on or off." Guest software branches on vendor identity, not only on feature flags: kernels apply vendor-specific errata workarounds and microcode-dependent behavior, libraries pick vendor-tuned code paths, and some MSRs a guest touched on one vendor simply don't exist or behave differently on the other. A frozen guest that made Intel-specific decisions can misbehave when its state is handed to KVM on an AMD host even if the raw feature intersection looks compatible on paper. Firecracker's AMD-flavored template (the T2A family) exists to make an AMD guest present a feature set *close to* the Intel T2 baseline so workloads behave consistently — but "behaves consistently for workloads" is a weaker claim than "an Intel-baked snapshot's frozen state will safely resume on AMD silicon."

In practice this means the pragmatic design is to template *within* a vendor and keep Intel and AMD as separate snapshot pools. You bake a template's snapshot under an Intel baseline for your Intel hosts and (if you also run AMD) a separate AMD-baseline snapshot for your AMD hosts, and the scheduler places a restore only onto a host in the matching pool. That's less elegant than "one snapshot everywhere," but it's honest about the hardware, and it avoids betting a customer's workload on a cross-vendor edge case the docs don't promise.

The operational rule: choose your template before you bake

Everything above collapses into one operational rule: the CPU template is a bake-time decision, and re-baking is the only way to change it. There is no in-place migration that reaches into a stale snapshot and strips a feature the guest already committed to — the code-path choices are already frozen in memory. So you decide the baseline before you capture the snapshot, or you live with whatever the bake host happened to advertise.

1. Inventory your restore fleet's CPUs. Enumerate every distinct CPU model (and vendor) a restore might land on. This set defines your portability pool — one pool per vendor, at minimum.
2. Pick the template that fits the poorest host in each pool. Static if your pool matches a named baseline; custom (mask to the exact feature intersection) if it doesn't. Advertise no feature that any host in the pool lacks.
3. Bake under that template. Cold-boot the template guest with the CPU template applied in machine-config, reach a ready state, then capture the snapshot. The guest's frozen feature detection now reflects the baseline, not the bake host.
4. Record the template in the snapshot's metadata. Store the template id (and vendor, guest kernel, and Firecracker version) alongside the snapshot so a restore can refuse an incompatible pairing up front instead of discovering it via a SIGILL an hour later.
5. Re-bake when the fleet changes. Add a poorer CPU generation, add a vendor, or change the template — any of those invalidates existing snapshots for that template, and the fix is a fresh bake, not a patch.

The reason this rule is worth internalizing is that the failure it prevents is invisible until production. An un-templated (or over-generously templated) snapshot works perfectly on your dev box and in every test that happens to run on the bake-host CPU. It fails only when a real create lands a restore on a host you didn't test against — which, on a scheduler that spreads load, is a matter of *when*, not *if*. Choosing the template at bake time is how you turn a random production crash into a decision you made on purpose.

Why a snapshot-per-request platform can't skip this

PandaStack has no warm pool of idle VMs — every single create is a snapshot-restore of a baked template onto whichever agent the global scheduler picks. A restore lands the snapshot in roughly a 49ms restore step, with an end-to-end create around 179ms p50 and 203ms p99, versus the ~3s of a first cold boot. That model only holds if the baked snapshot is portable across the fleet, because the scheduler scores agents purely on free CPU and memory and places the restore on whichever box is least loaded. The instant that fleet spans more than one CPU model, "schedule anywhere" means "restore onto whatever silicon has capacity" — and without a normalized CPU template, that's the SIGILL-later trap at fleet scale. So we bake templates under a CPU template pinned to the fleet's feature intersection: any agent can restore any baked snapshot because every guest was told it's running on the same, deliberately-conservative CPU.

The same requirement shows up in forks, harder. A same-host fork (roughly 400–750ms) copies the parent's frozen memory on the same box, so the CPU view already matches — no template concern. But a cross-host fork (1.2–3.5s once artifacts move over the network) restores the parent's frozen guest on a *different* machine, which means that machine's CPU has to satisfy every feature the parent committed to at bake time. The template the parent was baked under is exactly what guarantees the child can land elsewhere. The primitive that lets a snapshot restore across a heterogeneous fleet is the same primitive that lets a fork cross hosts at all — which is why "which template did we bake under, and does it fit the poorest host in the pool?" is a first-class question here, not a footnote.

PandaStack's core is open source under Apache-2.0, so you can run the control-plane API and per-host agent on your own mixed-CPU Linux KVM hosts, bake templates under whatever CPU template your hardware requires, and watch one snapshot restore across mismatched machines yourself. For the mechanics of the CPU template itself (static names, custom JSON, MSR modifiers) start at /blog/firecracker-cpu-templates-explained; for how snapshot format versioning couples to all this, see /blog/firecracker-snapshot-version-compatibility; and for the memory-restore and fork internals underneath, /blog/snapshot-and-fork-explained.