Firecracker CPU Templates, Explained

Here's a problem you hit the moment you run a serverless-microVM fleet across more than one machine: you bake a snapshot on a host with an Intel Ice Lake CPU, publish it, and an agent on an AMD EPYC box picks it up and tries to restore it. The guest inside that snapshot booted believing it had a specific set of CPU features — a specific instruction set, specific model bits, specific microarchitectural quirks. Resume it on a CPU that doesn't have those features and, at best, the guest is confused; at worst it executes an instruction the new host doesn't implement and dies. CPU templates are Firecracker's fix. A CPU template normalizes what the guest sees — the CPUID leaves and a set of MSRs — down to a common baseline that every host in your fleet can satisfy. This post is the working engineer's tour: what CPUID and MSRs actually are, the static templates (T2, T2S, T2CL, T2A, C3), the newer custom templates you write as JSON, the AVX and vulnerability-mitigation angle, and the honest cost of masking features. It's the backbone of restoring one snapshot across a heterogeneous host fleet — which is exactly why a sub-second snapshot-restore platform cares a great deal about it.

CPUID and MSRs: what the guest is asking about

Two mechanisms let software discover and configure the CPU it's running on, and a CPU template is entirely about controlling both of them for the guest.

• CPUID — a single instruction. Software runs CPUID with a leaf (and sometimes sub-leaf) selector, and the CPU answers in four registers: vendor string, family/model/stepping, and long bitfields advertising features — does this chip have AVX2? AVX-512? SHA extensions? RDRAND? The guest kernel and libc read these at boot and branch on them, choosing code paths, enabling instruction sets, sizing structures. CPUID is your VM asking the host 'what can you do?' — and a CPU template is you telling it to answer with a polite, curated lie.
• MSRs (Model-Specific Registers) — a bank of control and status registers read/written with RDMSR/WRMSR. Some are architectural and stable; many are, as the name says, specific to a model. They govern things like the speculative-execution mitigation controls, performance counters, and various microarchitectural knobs. A guest that reads an MSR expecting one layout and gets another — or writes one the host doesn't implement — is a fault waiting to happen.

When Firecracker boots a guest without a template, it presents a lightly-filtered pass-through of the host's real CPUID and MSRs — so the guest sees, more or less, the actual physical CPU. That's fine on a single machine. It becomes a portability problem the instant a snapshot taken against one CPU has to resume against another. The template is the layer that stops the guest from ever seeing host-specific details in the first place.

Why snapshot portability needs a common baseline

The core constraint is simple and unforgiving: a guest's view of its CPU is captured at boot and frozen into the snapshot. CPU feature detection is a boot-time act. The kernel probes CPUID once, decides 'this machine has AVX2, so I'll use the AVX2 memcpy,' wires up function pointers accordingly, and never asks again. glibc does the same. Those decisions are baked into live memory the moment you snapshot.

So if the guest decided at bake time that AVX-512 was available and selected AVX-512 code paths, and you then resume that exact frozen memory on a host CPU without AVX-512, the guest will eventually execute an AVX-512 instruction the new host can't decode. The result is an illegal-instruction fault — the microVM equivalent of a segfault, with no recovery. Nothing re-probes; the guest is running on assumptions it made against a CPU that no longer exists under it.

A CPU template breaks this by making every host in the fleet present the same curated feature set to the guest, regardless of the physical silicon. If T2 says 'no AVX-512,' then the guest never sees AVX-512 at bake time, never selects those code paths, and never executes an instruction some host in the fleet might lack. You are trading the fastest-possible instructions on any single machine for the guarantee that the same frozen guest runs on all of them. For a fleet that restores one baked template onto whatever host the scheduler picks, that guarantee is the whole game.

Static CPU templates: T2, T2S, T2CL, T2A, C3

Firecracker ships a handful of built-in named templates. You pick one by name in the machine-config, and Firecracker applies the corresponding CPUID/MSR normalization. The names encode a target baseline; treat the exact feature set of each as something to confirm against the Firecracker CPU-templates docs, because it evolves — but the intent of each is stable.

• T2 — Intel baseline. Normalizes the guest to a broad Intel feature set (roughly Skylake-era, notably without AVX-512), so a snapshot is portable across a range of Intel host generations. The common default when your fleet is Intel and spans multiple CPU generations.
• T2S — a T2 variant tuned for a specific security posture around speculative-execution behavior; it exposes/sets the MSR bits so guests apply the intended mitigations consistently. Reach for it when you want the mitigation-consistent flavor of the T2 baseline.
• T2CL — a T2-family template oriented toward Cascade Lake, again with the mitigation-related bits set so the guest sees a consistent, hardened view across that Intel line.
• T2A — the AMD counterpart to T2. It presents an AMD guest a feature set close enough to the Intel T2 baseline that workloads behave consistently across Intel and AMD hosts — the template that lets one snapshot family span both vendors.
• C3 — an older, more conservative Intel baseline (roughly Cascade-Lake-conservative / pre-T2 lineage). Broader compatibility, fewer features exposed; you'd use it when you need to include older Intel silicon in the portability set.

Selecting one is a single field on the machine-config you PUT before the guest starts. Here's the whole thing:

// PUT /machine-config — before InstanceStart.
// Pin the guest to the T2 Intel baseline so a snapshot baked
// here restores across every Intel host generation in the fleet.
{
  "vcpu_count": 2,
  "mem_size_mib": 2048,
  "smt": false,
  "cpu_template": "T2"
}

That's the static path in its entirety: one string. Firecracker knows the CPUID masks and MSR settings that name implies and applies them as it constructs the guest's CPU view. The named templates are the right tool when your fleet fits one of the shapes they target (all Intel, Intel+AMD, older-Intel-inclusive) and you don't need to shave the feature set any finer.

Custom CPU templates: CPUID and MSR modifiers in JSON

The static templates are a fixed menu. Custom CPU templates are Firecracker letting you write the recipe yourself as JSON: a list of CPUID modifiers and MSR modifiers that spell out, register by register and bit by bit, exactly what the guest should see. This is the mechanism when your fleet's lowest common denominator doesn't match any named template, or when you need to toggle one specific feature bit that a static template gets wrong for your hardware mix.

The two sections do what you'd expect. A CPUID modifier names a leaf and sub-leaf and a register (eax/ebx/ecx/edx), then supplies a bitmap that sets or clears specific bits of that register — that's how you mask off a feature flag. An MSR modifier names an MSR by address and gives the value (or a masked patch of it) the guest should read. The shape of the JSON, verbatim from the Firecracker CPU-templates docs, looks like this:

// A custom CPU template: clear one CPUID feature bit and pin one MSR.
// Field names/format follow the Firecracker CPU-templates docs — verify
// leaf/register/bitmap details against your Firecracker version.
{
  "cpuid_modifiers": [
    {
      "leaf": "0x7",
      "subleaf": "0x0",
      "flags": 0,
      "modifiers": [
        {
          "register": "ebx",
          "bitmap": "0b_____________________0____________________"
        }
      ]
    }
  ],
  "msr_modifiers": [
    {
      "addr": "0x10a",
      "bitmap": "0b00000000000000000000000000001000"
    }
  ]
}

The bitmap notation is the clever bit: each position is 0 (force-clear), 1 (force-set), or x (leave the host value untouched). That lets a template surgically clear one feature flag — say, mask off AVX-512 in leaf 0x7 — while passing everything else through from the host. You reference the file the same way you'd name a static template, pointing the machine-config at the JSON instead of a built-in name (check your Firecracker version's docs for whether that's a config-file field or a CLI flag). And Firecracker ships tooling to help you build these against your actual hardware:

# Dump the host's real CPUID + MSR configuration as a starting
# point, then hand-edit it down to your fleet's common baseline.
# (Tool/subcommand names vary by version — check the Firecracker
# CPU-templates docs for the exact invocation on your build.)
cpu-template-helper template dump \
  --config vm-config.json \
  --output baseline.json

# Once you have a custom template, point the machine-config at it
# instead of a built-in name and start the guest as usual.
curl --unix-socket /run/fc.sock -X PUT 'http://localhost/machine-config' \
  -d @machine-config-with-custom-template.json

The workflow is: dump the real CPUID/MSR set from each distinct CPU model in your fleet, intersect them to find the common baseline, and write a custom template that forces the guest down to that intersection. Then every host presents the guest the same view, and one snapshot restores everywhere. It's more work than typing 'T2,' and it's the escape hatch you want when the named menu doesn't fit your silicon.

Static vs. custom: which to reach for

• Static templates — pros: one string, zero maintenance, Firecracker owns the correctness of the masks. A: choose when your fleet matches a target shape (all-Intel, Intel+AMD via T2A, older-Intel via C3). B: cons: fixed menu, no per-bit control, and you inherit whatever baseline the name implies whether or not it's the tightest fit for your hardware.
• Custom templates — pros: exact per-bit control over CPUID leaves and MSRs, so you can hit precisely your fleet's lowest common denominator and no lower. A: choose when the named templates don't fit your CPU mix or you must toggle one specific feature. B: cons: you own the correctness — a wrong bitmap is a subtly-broken or unbootable guest, and you re-derive the template whenever your hardware fleet changes.
• Portability vs. raw performance — this is the real axis underneath both. A: masking features (either path) maximizes how many different host CPUs one snapshot can restore on. B: exposing everything the host has maximizes single-machine speed but pins the snapshot to hosts with that exact feature set. You're picking a point on that line, and a template is how you pick it deliberately instead of by accident.

The AVX and vulnerability-mitigation angle

Two categories of feature are the usual reason you reach for a template at all, and they pull in the same direction — toward masking — for different reasons.

AVX and wide vector instructions

AVX-512 and other wide vector extensions are the classic portability trap. They're fast when present, wildly uneven in availability across CPU generations and between Intel and AMD, and — because feature detection is frozen at snapshot time — the single most likely thing to make a snapshot non-portable. A guest that saw AVX-512 at bake time and selected AVX-512 code paths simply cannot run on a host that lacks them. So the standard move for a portable fleet is to mask the wide-vector features down to a baseline every host has (which is a big part of what T2 does by leaving AVX-512 out). You give up the fastest vector path on your newest silicon in exchange for one snapshot that runs on all of it.

Speculative-execution mitigations

The other reason templates exist is determinism and security around the Spectre/Meltdown family of speculative-execution issues. Those mitigations are driven by specific CPUID bits and MSR controls, and you generally want the guest to see a consistent, correct picture of which mitigations are in effect — regardless of which physical host it landed on. The mitigation-flavored templates (the T2S/T2CL family) set those bits so the guest applies the intended hardening uniformly across the fleet, rather than one guest getting a mitigation because its host happened to expose the bit and another guest silently not. In a multi-tenant sandbox platform, 'every guest sees the same, correctly-mitigated CPU' is not a nice-to-have — it's part of the isolation story.

The honest cost: masking features isn't free

None of this is a free lunch, and it's worth being blunt about the trade. When you mask a CPU feature, the guest stops using it — which means it stops getting the performance that feature would have given. Hide AVX-512 and a workload that would have taken the AVX-512 path takes a narrower vector path or a scalar one instead. For vector-heavy numerical code the difference can be real and measurable.

So the decision is a genuine trade-off, not a default you can ignore. If your fleet is one CPU model, you may not need a template at all — pass through the real CPU and get every instruction it has. If your fleet spans generations or vendors and you need one snapshot to run everywhere, a template is mandatory, and you're accepting some performance ceiling in exchange for that portability. The right answer is workload-dependent: latency-sensitive orchestration code barely notices; a heavily-vectorized inference or encoding workload might care a lot. The point of templates is that this is now a choice you make on purpose, at bake time, with your eyes open — rather than a crash you discover in production the first time the scheduler lands a snapshot on the wrong host.

Why a sub-second restore platform cares

PandaStack's entire model is snapshot-restore on every create: there's no warm pool of idle VMs, and a create lands at a 179ms p50 (roughly 203ms p99) by restoring a baked template snapshot rather than paying the ~3s of a first cold boot. That only works if the baked snapshot is portable — the global scheduler scores agents on free CPU and memory and picks whichever host is least loaded, which means a given template's snapshot has to be restorable on any agent in the fleet. The instant that fleet spans more than one CPU model, CPU templates are what make that portability real: bake under a template that fits the lowest common denominator of your hosts, and the scheduler is free to place the restore anywhere.

The same portability requirement extends to forks. A same-host fork (roughly 400–750ms) restores the parent's frozen memory on the same box, so the CPU view already matches — no template concern. But a cross-host fork (1.2–3.5s once artifacts move over the network) restores that frozen guest on a different machine, which means the fork target's CPU has to satisfy the feature set the parent guest committed to at bake time. The CPU template the parent was baked under is precisely what guarantees the child can land elsewhere. So the same primitive that lets a snapshot restore across a heterogeneous fleet is what lets forks cross hosts at all — and it's why 'which template did we bake under' is a first-class question, not an afterthought.

PandaStack's core is open source under Apache-2.0, so you can run the control-plane API and per-host agent on your own Linux KVM hosts, bake templates under whatever CPU template your hardware mix requires, and watch a single snapshot restore across mismatched machines yourself. For the memory side of restore, start at /blog/firecracker-memory-snapshots; for the full create-vs-restore picture, /blog/snapshot-restore-boot-path.