Firecracker on arm64 vs x86_64: What Actually Differs

Firecracker runs on x86_64 and on aarch64, and the marketing version of that sentence is "it just works on Graviton." Which is mostly true — same REST API, same JSON, same virtio devices, same jailer. Then you try to restore a snapshot baked on an arm host onto an Intel host, or you wonder why your carefully chosen CPU template does nothing on aarch64, or your guest kernel boots on one arch and goes silent on the other, and you find out that underneath that identical API there are two genuinely different machines. This post is the diff: boot protocol, interrupt controller, CPU feature masking, timekeeping, guest kernel config, snapshot portability — and the build and ops advice for running a mixed-arch fleet without hurting yourself.

One caveat up front, because this is the kind of post where being confidently wrong is expensive. Firecracker's architecture support has moved over time, and several specifics below — ACPI table generation on x86_64, aarch64 coverage in custom CPU templates, GIC version fallback — are version-dependent. Treat this as true in shape and verify the details against the Firecracker docs and source for your version.

Start with what doesn't change

Being precise about the sameness matters, because it's why porting a Firecracker-based platform to arm is a week of work rather than a rewrite.

• The API surface. Same HTTP-over-Unix-socket API, same endpoints, same JSON bodies. Your orchestration code barely knows which arch it's talking to.
• virtio over MMIO. virtio-blk, virtio-net, virtio-vsock and virtio-rng ride the MMIO transport on both architectures, and there is no PCI bus on either. That's unusual — most arm VMMs hand you virtio-pci.
• The jailer and seccomp model. Same chroot + cgroups + namespaces + drop-privileges design, same seccomp-BPF filtering. The filter files are built per target (syscall numbers differ, and aarch64 lacks some legacy syscalls entirely — everything is openat, not open), but the posture is identical.
• Rate limiters, MMDS, the metrics and logger FIFOs, the balloon device. All implemented above the arch boundary; all behave the same.
• The snapshot API. Same create/load calls, same memory-file-plus-state-file pair, same UFFD-backed restore option. Only the contents are arch-specific.

So the delta lives in three concepts: how the machine is described to the guest at boot, how interrupts are delivered, and how the CPU and clock are modelled.

Boot: boot_params and e820 vs a device tree

This is the biggest structural difference. On x86_64, Linux has a decades-old boot protocol: the bootloader — here Firecracker itself, since there is no GRUB, BIOS or UEFI — fills in a boot_params structure, supplies an e820 memory map describing which physical ranges are usable RAM and which are holes, and jumps to the kernel entry point. Firecracker also historically writes an Intel MP table so the guest can find its CPUs and the IOAPIC without ACPI; more recent versions additionally generate a small set of ACPI tables on x86_64, so check which your version does before debugging how a guest enumerated its CPUs. On x86_64 Firecracker can also boot via the PVH protocol when the ELF image carries the PVH entry-point note, falling back to the classic 64-bit Linux boot protocol otherwise.

None of that exists on aarch64. No e820, no MP table, no PVH. The arm64 Linux boot protocol says: load the kernel Image at a defined offset into RAM, put a pointer to a Flattened Device Tree blob in register x0, and jump. Everything the guest knows about its own hardware comes from that FDT, which Firecracker builds in memory at boot — RAM ranges, CPU nodes with PSCI declared as the method for bringing up secondary CPUs, the GIC node and its register windows, the serial UART, the RTC, each virtio-mmio device with its address and interrupt, and the kernel command line in the chosen node's bootargs property.

That has a consequence people trip over. On x86_64, Firecracker announces virtio-mmio devices by appending virtio_mmio.device=<size>@<addr>:<irq> entries to the kernel command line, because there's no bus to enumerate and no device tree to read — the command line is the discovery mechanism. On aarch64 those entries are absent, because the devices are FDT nodes. Same devices, same transport, entirely different discovery path.

Memory layout differs too: on x86_64 guest RAM starts at physical zero with a sub-4GiB MMIO hole, described via e820, while on aarch64 Firecracker places guest DRAM at a high base — 2 GiB (0x8000_0000) in the versions I've read — with the MMIO region below it and the whole map expressed in the FDT. You never notice until you're writing a memory-file parser or a userfaultfd handler and the offsets you hard-coded on x86 turn out to be nonsense. The VM configuration itself, though, is arch-neutral apart from the kernel image:

// The same config shape boots on both architectures.
// Only kernel_image_path (and a couple of boot args) are arch-specific.

// PUT /boot-source
{
  "kernel_image_path": "/var/lib/pandastack/kernels/Image-5.10-aarch64",
  "boot_args": "console=ttyS0 panic=1 pci=off"
}

// PUT /drives/rootfs   -- identical on x86_64 and aarch64
{
  "drive_id": "rootfs",
  "path_on_host": "/var/lib/pandastack/vms/abc123/rootfs.ext4",
  "is_root_device": true,
  "is_read_only": false
}

// PUT /network-interfaces/eth0   -- identical
{
  "iface_id": "eth0",
  "host_dev_name": "tap0",
  "guest_mac": "06:00:AC:10:00:02"
}

// PUT /machine-config   -- identical shape, two caveats
{
  "vcpu_count": 2,
  "mem_size_mib": 1024,
  "smt": false
  // "cpu_template": "T2"  <-- static templates are x86_64-only;
  //                          smt must be false on aarch64.
}

Interrupts: APIC/IOAPIC vs the GIC

On x86_64 the guest gets the familiar legacy stack: a local APIC per vCPU plus an IOAPIC (and the vestigial 8259 PIC), created inside KVM through the in-kernel irqchip. MMIO devices raise ordinary line interrupts routed as GSIs through the IOAPIC, and the MP table — or ACPI, version depending — is what tells the guest that topology exists.

On aarch64 there is no APIC. Interrupts go through the Generic Interrupt Controller, created as a KVM device rather than via the x86 irqchip call. Firecracker prefers GICv3 — what you get on Graviton2/Graviton3 and Ampere parts — and, depending on version, falls back to GICv2 on hosts that only offer v2. virtio-mmio devices are wired to SPIs (shared peripheral interrupts), and each device's interrupt number is written into its FDT node so the guest kernel knows where to listen.

The operator-facing consequence: the GIC carries redistributor state per vCPU, and that state is part of what Firecracker saves and restores in a snapshot. It's one more reason a state file is architecture-shaped rather than generic — there is no field in an x86 snapshot to put GIC redistributor registers into.

CPU templates: CPUID masking is an x86 idea

On x86_64, a guest asks "what CPU am I on?" by executing CPUID and reading MSRs. That's a conveniently interceptable interface, and it's why Firecracker's CPU templates exist: T2, T2S, T2CL, T2A and C3 mask CPUID leaves and MSRs down to a normalized baseline so a snapshot baked on one host CPU restores on a different one without the guest tripping over a feature bit that vanished. These static templates are x86_64-only and vendor-flavoured — T2 for Intel, T2A for AMD — because the CPUID layouts they normalize are vendor-specific.

aarch64 has no CPUID instruction. A guest discovers its features by reading architectural ID registers — ID_AA64PFR0_EL1, ID_AA64ISAR0_EL1, MIDR_EL1 and that family — which the VMM and KVM can constrain through KVM's register interface. So the arm equivalent of a CPU template isn't CPUID masking, it's register modifiers. Firecracker's newer custom JSON templates are the path for expressing that on aarch64; the named static templates have no aarch64 counterpart. Availability and register coverage depend on your Firecracker version, so verify before planning a heterogeneous-arm strategy around them.

Time: the TSC vs the arm generic timer

On x86_64 the guest's fast clock is the TSC, a cycle counter whose frequency is a property of the host part. That creates a real snapshot problem: restore on a host whose TSC ticks at a different rate and the guest's notion of elapsed time is wrong. x86 KVM can paper over this with hardware TSC scaling where the CPU supports it, and Firecracker records TSC frequency in the snapshot so a restore can detect a mismatch rather than quietly producing a guest that thinks time runs 12% fast.

On aarch64 the guest reads the architectural generic timer — CNTVCT_EL0 for the virtual counter, with CNTFRQ_EL0 declaring the frequency. That frequency is fixed by hardware (commonly a tidy 25 MHz on Graviton) and isn't something the VMM dials in per guest the way TSC scaling can. The upside is boring consistency inside a host family; the downside is that if you did move a snapshot between arm hosts with different timer frequencies, there's no scaling knob to rescue you. In the guest, the clocksource name is the tell: arch_sys_counter on arm64, tsc on x86_64. What is identical on both: a restored guest wakes up believing it's whatever time the snapshot was taken, so resynchronize the guest clock on restore either way.

Snapshots: never across architectures, conditionally within one

A Firecracker snapshot is two artifacts: a memory file (the guest's physical RAM, byte for byte) and a state file (VMM device state plus per-vCPU register state). Both are architecture-specific in the most fundamental way available. The memory file holds a running kernel and userspace compiled for one instruction set. The state file holds a register dump — x86 general-purpose and segment registers, MSRs, LAPIC state on one side; arm64 core registers, system registers, GIC state on the other. There is no serialization scheme under which those are interchangeable, and Firecracker doesn't pretend otherwise: loading on the wrong architecture fails rather than producing a subtly haunted VM, which is the correct and merciful behaviour.

Within one architecture, portability is conditional. Across CPU models you need feature normalization — CPU templates on x86, register constraints on arm — or the guest may execute an instruction the new host doesn't implement. Across Firecracker versions and guest kernel versions, snapshots are pinned to what produced them: change either and you re-bake. Same rules on both architectures, different masking machinery.

Guest kernel: same minimalism, different load-bearing symbols

The philosophy carries over exactly — tiny kernel, PCI off, virtio in, no drivers for hardware that doesn't exist. But the symbols that make or break the boot differ, and most of the differences fall straight out of the FDT-versus-command-line split above. Start from the reference guest configs Firecracker publishes per architecture; these are the ones I've been burned by.

• Device tree support. aarch64 needs CONFIG_OF and the OF-based platform drivers, because that's how it finds every device. x86_64 has no use for it.
• Serial console plumbing. Firecracker describes an ns16550a-compatible UART in the aarch64 FDT rather than the PL011 most arm VMMs expose, so console=ttyS0 (not ttyAMA0) is normally right — but the guest needs the OF-platform 8250 glue to bind a device-tree-declared UART, where x86 finds the same 8250 at legacy I/O port 0x3f8. Verify the UART your version emits before assuming.
• virtio-mmio discovery. CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES matters on x86_64, where devices arrive as command-line entries, and is irrelevant on aarch64, where FDT nodes drive probing.
• Interrupt controller and timer. aarch64 needs the GICv3 driver (plus v2 if you'll run on older hosts), the arm architected timer, and PSCI firmware support — PSCI is how secondary vCPUs come up and how the guest requests reset. On x86 those roles belong to the APIC's INIT/SIPI sequence and the legacy reboot paths.
• Boot-arg x86-isms. reboot=k and the i8042.noaux/nomux/nopnp/dumbkbd flags suppress legacy PS/2 probing and route reset through the keyboard controller. None of that hardware exists on arm, so they're dead weight there. pci=off is harmless on both.
• Page size. arm64 kernels build for 4K, 16K or 64K pages, and some arm64 distros ship 64K-page kernels. Host page size changes your snapshot memory-file granularity and anything built on userfaultfd — check getconf PAGESIZE per host class instead of assuming 4096.

The build differs in the target you ask for: x86_64 Firecracker boots an uncompressed ELF vmlinux, aarch64 boots the arm64 kernel Image that make Image produces. Tooling sometimes names both artifacts confusingly alike, so go by what your build actually emitted.

# ---------- x86_64 guest kernel ----------
make ARCH=x86_64 olddefconfig
make ARCH=x86_64 vmlinux -j"$(nproc)"        # uncompressed ELF -> kernel_image_path

# boot args carry legacy-hardware suppressors AND device discovery:
#   console=ttyS0 reboot=k panic=1 pci=off \
#     i8042.noaux i8042.nomux i8042.nopnp i8042.dumbkbd
# ...plus, appended by Firecracker per device:
#   virtio_mmio.device=4K@0xd0000000:5


# ---------- aarch64 guest kernel ----------
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- olddefconfig
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- Image -j"$(nproc)"
# -> arch/arm64/boot/Image   (PE format) -> kernel_image_path

# boot args are shorter: no i8042 (no PS/2 controller exists),
# reset goes through PSCI, and virtio devices come from the FDT:
#   console=ttyS0 panic=1 pci=off

# Sanity-check what you actually built:
file arch/arm64/boot/Image   # -> Linux kernel ARM64 boot executable Image
file vmlinux                 # -> ELF 64-bit LSB executable, x86-64

The side-by-side

• Boot protocol — x86_64: Linux boot protocol with boot_params and an e820 map, plus an MP table (and ACPI tables in newer versions); PVH entry supported when the ELF carries the note. aarch64: arm64 boot protocol — load Image, DTB pointer in x0, everything described by a Flattened Device Tree Firecracker builds. No PVH.
• Device discovery — x86_64: virtio_mmio.device= entries appended to the kernel command line. aarch64: virtio-mmio nodes in the FDT; no command-line entries needed.
• Interrupt controller — x86_64: LAPIC + IOAPIC (+ legacy PIC) via KVM's in-kernel irqchip, devices raising GSI line interrupts. aarch64: GIC created as a KVM device, GICv3 preferred with GICv2 fallback depending on version, devices wired to SPIs declared in the FDT.
• CPU feature masking — x86_64: CPUID leaves and MSRs, normalized by vendor-specific static templates (T2, T2S, T2CL, T2A, C3). aarch64: no CPUID; architectural ID registers (ID_AA64*, MIDR_EL1) constrained through KVM's register interface, expressed via custom JSON templates where your version supports them.
• Timekeeping — x86_64: TSC, host-frequency-dependent, with hardware TSC scaling on capable CPUs and frequency recorded in snapshots. aarch64: architectural generic timer, CNTVCT_EL0 with a hardware-fixed CNTFRQ_EL0 and no per-guest scaling knob. Guest clocksource reads tsc vs arch_sys_counter.
• Snapshot portability — across the two: never. Different instruction set in the memory file, different register set in the state file, and the load fails outright. Within one architecture: conditional on CPU-feature normalization, and always pinned to the Firecracker and guest kernel versions that produced the snapshot.
• virtio transport — x86_64: virtio-mmio, no PCI. aarch64: virtio-mmio, no PCI. Identical, and unusual for arm.
• Jailer, seccomp, API — both: same design, same endpoints; seccomp filters built per target because syscall numbers differ and aarch64 lacks some legacy syscalls entirely.
• Host availability — x86_64: everywhere, every cloud, nested virtualization widely available for dev. aarch64: AWS Graviton, Ampere-based instances elsewhere, arm bare metal; on a Mac you need Linux-in-a-VM with nested virtualization (Lima on Apple's Virtualization.framework) to get /dev/kvm at all.

Running a mixed-arch fleet without hurting yourself

All of the above collapses into a handful of operational rules. If you take one thing from this post, take the first.

1. Make architecture a first-class key on every artifact — kernels, rootfs images, baked snapshots, memory files, seed manifests. Put it in the storage path, not just a metadata field someone might forget to read. That's cheap insurance against an x86 agent downloading an arm snapshot and failing mysteriously for two minutes.
2. Make the scheduler arch-aware and fail closed. A create that resolves to a snapshot may only be placed on an agent whose architecture matches; if none exists, return a clear error rather than falling back to close enough.
3. Build rootfs images multi-arch from one Dockerfile: buildx with --platform linux/amd64,linux/arm64 and an ARG TARGETARCH for anything you fetch as a prebuilt binary. Toolchain installers are the usual offender — they publish x64 and arm64 tarballs under different names, and a hard-coded URL yields an image that builds fine and dies at runtime on the other arch.
4. Keep separate per-arch kernel and template pipelines rather than one branchy pipeline. The build targets differ, the configs differ, and the bake step produces mutually incompatible outputs.
5. Don't develop on one arch and deploy on the other without CI on both. Code that runs aarch64 in dev and x86_64 in prod will eventually hit an arch-specific bug in the one place you have no local reproduction.
6. Preflight every host before you trust it: architecture, KVM access, page size, GIC version, clocksource. Five commands at agent boot, logged with the heartbeat.

#!/usr/bin/env bash
# Preflight a Firecracker host (Graviton, Ampere, or plain x86) before
# letting the scheduler place anything on it.
set -euo pipefail

# 1. Which architecture are we actually on?
uname -m                  # aarch64 on Graviton/Ampere; x86_64 on Intel/AMD

# 2. Can we use KVM at all? (No /dev/kvm -> nothing else matters.)
ls -l /dev/kvm
[ -w /dev/kvm ] || echo "WARN: /dev/kvm not writable by this user"

# 3. Host page size decides memory-file and userfaultfd granularity.
getconf PAGESIZE          # 4096 almost everywhere; some arm64 distros ship 65536

# 4. Which interrupt controller did the host give us?
grep -iE 'gic|apic' /proc/interrupts | head -3
dmesg 2>/dev/null | grep -i -m2 'GIC' || true   # GICv3 on Graviton2/3

# 5. Which clock will guests read, and at what frequency?
cat /sys/devices/system/clocksource/clocksource0/current_clocksource
#   arm64 -> arch_sys_counter   |   x86_64 -> tsc
dmesg 2>/dev/null | grep -i -m1 arch_timer || true
#   e.g. arch_timer: cp15 timer(s) running at 25.00MHz

# 6. Firecracker itself: the build target must match the host.
firecracker --version

How this plays out at PandaStack

I'll be honest about our setup, because it's exactly the mixed-arch shape this post describes: I develop on an Apple Silicon Mac, where Firecracker runs inside a Lima VM on Apple's Virtualization.framework with nested virtualization — so my local guests are aarch64 — while production agents run on x86_64 Linux hosts. That split is useful, because it forces the arch boundary to stay explicit, and annoying, because a snapshot I bake locally is worth nothing to a production agent and vice versa.

The rules above are the ones we actually run. Template Dockerfiles resolve prebuilt toolchain binaries through ARG TARGETARCH so one file builds for local arm64 and for the linux/amd64 CI bake; that single line has saved more debugging time than anything else in the repo. Snapshot seeds are namespaced so a cross-arch pull can't happen by accident. The performance story, meanwhile, is a property of the boot path rather than the instruction set: every sandbox create is a snapshot restore rather than a boot, landing around 179ms p50 and 203ms p99 with the restore step itself near 49ms; the once-per-template cold boot that produces that snapshot takes about 3 seconds; a same-host fork runs 400–750ms and a cross-host fork 1.2–3.5s; and each agent pre-allocates 16,384 /30 subnets so networking is never what you're waiting on. Those numbers are measured on our production x86_64 fleet — I'm not going to invent arm equivalents I haven't benchmarked.

The bottom line: porting a Firecracker platform to arm isn't hard, it's detailed. The API doesn't move. The device model doesn't move. What moves is everything that describes the machine to the guest — and the snapshot, which doesn't move at all.