Firecracker CPUID masking, explained

When a program wants to know what CPU it's running on, it executes one instruction: CPUID. Ask for leaf 0 and you get the vendor string ("GenuineIntel", "AuthenticAMD"). Ask for leaf 1 and you get family, model, and stepping, plus a bitfield of feature flags — does this chip have SSE4.2, AES-NI, AVX, AVX-512? Higher leaves describe cache and TLB topology, hypervisor presence, and extended features. Every language runtime, every libc, every crypto library runs CPUID at startup to decide which code path to take. It is the CPU introspecting itself, in hardware, in a single instruction.

Inside a Firecracker microVM, that self-introspection is a polite fiction. The guest executes CPUID, but the answer it gets back is whatever Firecracker decided to tell it — not necessarily what the silicon underneath actually supports. That masking is deliberate, and it's load-bearing for anything that snapshots and restores VMs across a fleet. This post is about how CPUID gets trapped, why Firecracker normalizes it, and why that's the quiet foundation under CPU templates and cross-host restore.

How KVM traps CPUID and hands the VMM the pen

On bare metal, CPUID is a non-privileged instruction — any code can run it and the hardware answers directly. Under virtualization it's different. CPUID is one of the instructions that unconditionally causes a VM exit: when the guest runs it, the CPU stops executing guest code and traps into the host. KVM catches that exit and, instead of forwarding the raw hardware result, returns a table of values the VMM (Firecracker) supplied ahead of time.

The mechanism is a per-vCPU CPUID table. Before the guest ever runs, Firecracker builds a set of (leaf, subleaf) → (eax, ebx, ecx, edx) entries and pushes them to KVM via an ioctl (the KVM_SET_CPUID2 family). From then on, every CPUID the guest executes is answered from that table. KVM traps the instruction; Firecracker owns the answer. The guest cannot tell the difference — it just sees a CPU with exactly the features the table advertises.

Why mask CPUID at all?

If the host already knows its own features, why not just pass them straight through? Because faithfully reflecting the host CPU creates three separate problems, and masking solves all of them at once:

• Security — hide host-specific details (exact model, serial-ish topology) and disable feature bits the guest shouldn't have, including ones that widen speculative-execution side-channel surface. If a feature isn't advertised, well-behaved guests won't use it.
• Snapshot portability — a VM snapshotted on one CPU must resume on another. If the guest was told it has features the destination host lacks, restore is a landmine. So you advertise a lowest-common-denominator set that every host in the fleet can honor.
• Determinism — every VM across a heterogeneous fleet sees the same advertised CPU, so runtimes make the same feature-detection decisions everywhere. No "works on the Ice Lake host, crashes on the Skylake host" surprises.

The second point is the one people underestimate, so it's worth making concrete. It has a specific, ugly failure mode.

The failure mode: the guest that was told it had AVX-512

Imagine you bake a snapshot on a host with AVX-512. At boot, glibc runs CPUID, sees the AVX-512 bit set, and wires up AVX-512 code paths in memcpy, string routines, and any math library the guest loaded. That decision is now frozen into the running process's function pointers and JIT'd code. You snapshot the VM — memory, registers, everything — and file it away.

Later you restore that snapshot onto a host without AVX-512. The guest doesn't re-run feature detection; it just resumes exactly where it left off, still holding pointers into AVX-512 code. The moment it calls one of those routines, the CPU hits a 512-bit instruction the hardware doesn't implement, raises #UD (invalid opcode), and the guest crashes. There was no chance to recover — the wrong decision was made at boot on a different machine.

This is why the masking has to happen at VM configuration time, not at restore time. By the time you're restoring, the guest's feature-detection decisions are ancient history baked into memory. The fix is to never advertise a feature that some destination host might lack. Present the intersection, and any host in the fleet can safely resume the guest.

CPU templates: a curated CPUID + MSR mask

Doing that intersection by hand for every leaf and flag would be miserable and error-prone, so Firecracker ships the answer as CPU templates. A CPU template is exactly what the name suggests: a curated set of CPUID entries (and a matching set of MSR values) that Firecracker forces onto the guest, overriding the host's real capabilities. Firecracker's built-in templates — T2 and C3 on Intel, T2A/T2CL and others across vendors — each define a stable, conservative feature set that a whole class of hosts can present identically.

The point of a named template is that a guest booted under T2 on an old host and a guest booted under T2 on a new host see the same CPU. That's what makes their snapshots interchangeable. Templates come in two flavors: static templates (a fixed, hand-picked feature set like T2/C3) and custom templates (you specify exact CPUID leaf/flag and MSR modifications yourself). The static ones are the sane default; custom ones are for when you need to pin something specific across a mixed fleet.

Conceptually, a custom CPU-template config lists the leaves and bits to rewrite. Illustratively, forcing a flag off in a leaf looks something like this (the exact schema — field names, the bitmap format, whether you match on a value or a mask — is version-specific, so treat this as the shape, not the spec, and check Firecracker's CPU templates docs for the real thing):

{
  "cpuid_modifiers": [
    {
      "leaf": "0x7",
      "subleaf": "0x0",
      "flags": 0,
      "modifiers": [
        {
          "register": "ebx",
          "bitmap": "0b_xxxx_xxxx_xxxx_xxxx_0xxx_xxxx_xxxx_xxxx"
        }
      ]
    }
  ],
  "msr_modifiers": [
    {
      "addr": "0x10a",
      "bitmap": "0b_xxxx_xxxx_xxxx_xxxx_xxxx_xxxx_xxxx_xx1x"
    }
  ]
}

Leaf 0x7 subleaf 0 is where the extended feature flags live, including the AVX-512 family bits in EBX — this is the exact register you'd clear to make our earlier crash impossible. A `0` in the bitmap forces that bit off; the `x` positions pass through unchanged. The MSR modifier alongside it is how a template also normalizes model-specific registers (for example, exposing the arch-capabilities MSR that tells the guest which speculative-execution mitigations it doesn't need to apply). A CPU template is just this, comprehensively, for every leaf and MSR that matters.

Checking what the guest actually sees

The nice thing about all this is that it's directly observable from inside the guest — the same CPUID the runtime uses is the one you can read yourself. Boot a sandbox and inspect `/proc/cpuinfo` (or run the `cpuid` tool) and you'll see the vendor and flags the template chose, which may be a strict subset of the bare-metal host's:

from pandastack import Sandbox

with Sandbox.create(template="code-interpreter", ttl_seconds=120) as sbx:
    # The flags line is CPUID leaf 1 + leaf 7, decoded by the kernel.
    flags = sbx.exec("grep -m1 '^flags' /proc/cpuinfo").stdout
    print("advertised flags:", flags)

    # Does this guest think it has AVX-512? (grep exits 1 if not found.)
    r = sbx.exec("grep -qw avx512f /proc/cpuinfo && echo yes || echo no")
    print("avx512f present?", r.stdout.strip())

    # Vendor string comes straight from CPUID leaf 0.
    print(sbx.exec("grep -m1 vendor_id /proc/cpuinfo").stdout.strip())

If the guest reports `no` for AVX-512 even though you know the host silicon has it, that's the mask working exactly as intended — the feature was deliberately withheld so the snapshot stays portable. The guest asked "do I have AVX-512?" and the hypervisor answered "no" for the guest's own long-term health.

Why this is load-bearing for PandaStack

PandaStack creates every sandbox by restoring a baked Firecracker snapshot rather than cold-booting — that's how a create is p50 179ms (p99 ~203ms) instead of ~3s. Snapshot-restore is the entire boot path, not an optimization tacked on the side. And forks push it further: a same-host fork is 400–750ms, and a cross-host fork — restoring a snapshot on a completely different physical machine — is 1.2–3.5s.

Cross-host fork only works if the guest sees a consistent CPU on the destination that it saw on the origin. The snapshot froze the guest's feature-detection decisions; if the destination host advertised a different feature set, we'd be back to the #UD crash. A normalized CPUID via a CPU template is precisely what makes a snapshot baked on one machine safely restorable on another. It's the reason we can spread load across a heterogeneous fleet (up to 16,384 microVMs per agent host on the networking side) without every restore being a gamble on which CPU generation it lands on.

So CPUID masking isn't an obscure security knob — it's the compatibility contract that makes fleet-wide snapshot-restore and cross-host fork sound. The guest gets told a slightly humbler truth about its CPU, and in exchange its running state can teleport between machines without executing an instruction that doesn't exist. That's a trade every multi-tenant microVM platform makes, whether or not it says so out loud.