Building a Minimal Firecracker Guest Kernel

Firecracker doesn't ship a kernel. It boots the one you hand it — an uncompressed vmlinux image, loaded straight into guest memory with no bootloader and no firmware in front of it. That gives you a lever most people never touch: you get to decide exactly what the guest kernel can and cannot do, because you get to decide what's compiled into it. And for a microVM that only ever sees a handful of virtio devices, the honest answer to "what should be in the kernel?" is: startlingly little. This post walks through building a minimal guest kernel for Firecracker — what to strip, what you must absolutely keep, the boot arguments Firecracker passes on the kernel command line, why PCI is switched off, and the one thing nobody warns you about: your kernel choice gets frozen into every snapshot, so changing it is not free.

Everything here is grounded in how Firecracker's own documentation and CI build a guest kernel — they ship reference configs and a recommended kernel version. Config option names and defaults drift between kernel releases, so where a specific symbol matters, verify it against the Firecracker kernel docs and the config for your kernel version rather than trusting a snippet verbatim. The shape of the exercise, though, is stable across versions, and that shape is the point.

Why a stripped-down kernel config matters

A general-purpose distro kernel is built to boot on anything — your laptop, a server with forty PCI devices, a Raspberry Pi, a machine with a sound card and three kinds of RAID controller. It carries drivers for hardware it will probably never meet, because the distro can't know in advance what it'll run on. That generality is a liability inside a Firecracker microVM, where the hardware is fixed, virtual, and known at build time. Three concrete costs come from carrying a kernel that's bigger than the machine it runs on.

• Boot speed. Every driver compiled in is code the kernel initializes and hardware it probes at boot. A microVM has no PCI bus to enumerate and no exotic controllers to poke, so all that probing is pure latency — time the guest spends looking for devices that aren't there. Strip the drivers and the init sequence has far less to grind through.
• Memory. A smaller vmlinux and a smaller resident kernel footprint mean each guest costs less RAM before it does any work. When you pack many microVMs onto a host, kernel overhead per guest is a real line item, and megabytes add up across thousands of VMs.
• Attack surface. This is the big one. Every driver you compile in is a driver an attacker can try to confuse — a code path reachable from inside the guest that has to be correct. The fastest, safest code is the code you didn't build. A filesystem driver you never mount can't have an exploitable parser; a network protocol you didn't enable can't be smuggled through. Removing a subsystem removes its entire class of bug from your guest.

The framing that keeps you honest: a minimal kernel isn't a stripped-down version of a real kernel — it's the right-sized kernel for a machine whose hardware you fully control. You're not removing features you'll miss. You're declining to compile support for a world the microVM doesn't live in.

What you strip, and what you must keep

The categories divide cleanly. On the strip side: hardware drivers for physical devices (real NICs, GPUs, sound, USB controllers, RAID/SCSI HBAs, Wi-Fi, Bluetooth), filesystems you'll never mount, most of the debugging and tracing infrastructure, and whole subsystems a sandbox has no use for. On the keep side: the virtio drivers that are the microVM's only real hardware, the serial console so you can see boot output and panics, the KVM guest paravirt bits that make the guest cooperate with the hypervisor efficiently, and just enough core kernel (the scheduler, memory management, the one filesystem your rootfs actually uses) to run userspace. Here's the map, config area by config area.

• Block I/O — Keep: virtio-blk (CONFIG_VIRTIO_BLK), the driver for your rootfs disk. Strip: SCSI, IDE/ATA, NVMe, USB storage, RAID/dm-multipath, and every physical-disk HBA — a Firecracker guest's only disk is a virtio-block device backed by a host file.
• Networking — Keep: virtio-net (CONFIG_VIRTIO_NET) plus the core TCP/IP stack. Strip: every physical NIC driver (e1000, ixgbe, r8169…), Wi-Fi, Bluetooth, and exotic L2 protocols — the guest's only interface is a virtio-net device wired to a host TAP.
• Guest-host channel — Keep: virtio-vsock (CONFIG_VIRTIO_VSOCKETS) if you use a vsock control channel to the host agent. Strip: nothing here you'd want; vsock is a small, deliberate keep.
• Console — Keep: the 8250/16550 serial UART (CONFIG_SERIAL_8250 + CONFIG_SERIAL_8250_CONSOLE) so ttyS0 works. Strip: framebuffer/VGA/DRM graphics, virtual terminals you don't need — a microVM has no display, just a serial line.
• Filesystems — Keep: exactly the one your rootfs uses (ext4 is the common choice) plus the pseudo-filesystems userspace expects (proc, sysfs, tmpfs, devtmpfs). Strip: Btrfs, XFS-in-guest, NFS, FUSE, network and cluster filesystems, and every FS you won't mount — each is an unused parser.
• Debug & tracing — Keep: a minimal amount if you're actively developing the kernel. Strip (for production): KGDB, most of ftrace/kprobes, heavy lockdep/KASAN, and debugfs surface — great for hacking, dead weight and extra surface in a shipped guest.
• PCI — Keep: nothing. Strip: the entire PCI subsystem (# CONFIG_PCI is not set). Firecracker exposes virtio over MMIO, not PCI, so there is no bus to enumerate — turning PCI off is both faster and one less subsystem to carry.
• Misc subsystems — Keep: KVM guest/paravirt support (CONFIG_KVM_GUEST, CONFIG_PARAVIRT), a virtio entropy source (CONFIG_HW_RANDOM_VIRTIO). Strip: sound, power management for physical hardware, hotplug for buses you don't have, and the long tail of platform drivers.

The Kconfig, in the parts that matter

You don't hand-write a full .config — it has thousands of symbols. You start from a reference config (Firecracker publishes recommended guest configs per architecture and kernel version) and adjust. But the handful of symbols that define a Firecracker guest are worth seeing explicitly. This is a fragment, not a complete config, and it's illustrative — confirm the exact symbol names against your kernel version, since some move between releases.

# --- Firecracker guest kernel: the load-bearing symbols (fragment) ---
# virtio transport over MMIO (Firecracker does NOT use PCI)
CONFIG_VIRTIO=y
CONFIG_VIRTIO_MMIO=y

# the microVM's only real "hardware"
CONFIG_VIRTIO_BLK=y          # rootfs disk
CONFIG_VIRTIO_NET=y          # network via host TAP
CONFIG_VIRTIO_VSOCKETS=y     # guest<->host control channel
CONFIG_HW_RANDOM_VIRTIO=y    # entropy from the host

# serial console so you can see boot + panics on ttyS0
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y

# run efficiently as a KVM guest
CONFIG_HYPERVISOR_GUEST=y
CONFIG_PARAVIRT=y
CONFIG_KVM_GUEST=y

# the one rootfs filesystem you actually mount
CONFIG_EXT4_FS=y

# --- and the things you deliberately DON'T build ---
# CONFIG_PCI is not set            # MMIO virtio only; no bus to enumerate
# CONFIG_USB is not set            # no USB controllers in a microVM
# CONFIG_DRM is not set            # no graphics / framebuffer
# CONFIG_SOUND is not set          # no audio
# CONFIG_SCSI is not set           # no SCSI/SATA/NVMe disks
# CONFIG_WLAN is not set           # no Wi-Fi
# CONFIG_MODULES is not set        # build in what you need; ship no loadable modules

That last line is a quiet but meaningful choice. Building everything in and setting `# CONFIG_MODULES is not set` means the guest ships one self-contained vmlinux with no loadable-module machinery — nothing to `insmod`, no module loader to abuse, and no need for a separate modules tree in the rootfs. For a fixed, known microVM there's rarely a reason to load a module at runtime, so compiling drivers directly into the kernel is simpler and tighter. (If your workload genuinely needs runtime modules, that's a deliberate exception, not the default.)

The build: defconfig, menuconfig, vmlinux

The workflow is ordinary kernel building with one twist: your target artifact is vmlinux (the uncompressed ELF), not the bzImage a distro would produce, because Firecracker loads the raw image. You start from a base config, layer the Firecracker-specific choices on top, optionally fine-tune interactively with menuconfig, and build. Roughly:

# 1. Get the kernel source at the version Firecracker recommends
#    (verify the exact version against the Firecracker kernel docs).
git clone --depth 1 --branch v5.10.223 \
  https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
cd linux

# 2. Start from Firecracker's published guest config for your arch,
#    or from the in-tree microvm defconfig as a base.
curl -sL https://raw.githubusercontent.com/firecracker-microvm/firecracker/main/resources/guest_configs/microvm-kernel-x86_64-5.10.config \
  -o .config
# (URL/path varies by release — check the Firecracker repo's resources/guest_configs.)

# 3. Reconcile the config against this source tree; accept defaults
#    for any new symbols this kernel version added.
make olddefconfig

# 4. OPTIONAL: fine-tune interactively. This is where you'd turn PCI off,
#    confirm the virtio + console symbols, and strip subsystems by hand.
#    Search inside menuconfig with '/', e.g. '/PCI' or '/VIRTIO_BLK'.
make menuconfig

# 5. Build the UNCOMPRESSED image. Firecracker wants vmlinux, not bzImage.
make vmlinux -j"$(nproc)"

# 6. The artifact Firecracker boots:
ls -lh vmlinux

A couple of things to internalize. `make olddefconfig` is what keeps a config portable across kernel versions — it takes your existing choices and fills in sane defaults for any symbols the new tree introduced, which is exactly the moment a config silently drifts if you're not paying attention. And `make vmlinux` (rather than the default target) is the whole point: you want the flat, uncompressed ELF that Firecracker's loader jumps into. If you hand Firecracker a bzImage, it won't boot it the way you expect — the loader is built for the raw image.

The boot args Firecracker passes

A kernel needs a command line, and since there's no bootloader, Firecracker hands it directly through the boot-source configuration in its API. The command line for a Firecracker guest is short and opinionated, and each token is there for a microVM-shaped reason. A representative line:

console=ttyS0 reboot=k panic=1 pci=off i8042.noaux i8042.nomux i8042.nopnp i8042.dumbkbd

• console=ttyS0 — send kernel messages to the first serial port. That's the UART you kept in the config; it's how you get any boot output or panic trace out of a machine with no screen.
• reboot=k — reboot via the keyboard controller. Firecracker has no ACPI/real reset hardware, so the guest uses the tiny keyboard controller path to signal a reset the VMM can act on.
• panic=1 — on a kernel panic, reboot after 1 second instead of hanging forever. For an ephemeral sandbox you want a panicked guest to die fast so the VMM can reap it, not sit wedged.
• pci=off — don't even try to probe a PCI bus. There isn't one (virtio is MMIO), and telling the kernel up front saves the probe entirely. Belt-and-suspenders with the # CONFIG_PCI is not set above.
• i8042.noaux / .nomux / .nopnp / .dumbkbd — quiet down the legacy i8042 keyboard/mouse controller. There's no real PS/2 hardware to detect; these flags stop the kernel wasting boot time probing an aux port and mouse that don't exist.

The theme running through the command line is the same as the config: tell the kernel, as early as possible, about the hardware that isn't there. `pci=off` and the `i8042.*` flags are the boot-args equivalent of not compiling a driver — they cut probes for absent devices out of the critical path so the guest reaches userspace sooner. The exact set of flags evolves; check the Firecracker docs for the current recommended command line for your kernel version.

Wiring the kernel into Firecracker

You point Firecracker at your vmlinux and set that command line through the boot-source part of its API — one PUT before the instance starts. The shape makes the two pieces explicit: a path to the uncompressed kernel on the host, and the boot args as a single string.

// PUT /boot-source  — tell Firecracker which kernel to boot and how.
{
  "kernel_image_path": "/var/lib/pandastack/kernels/vmlinux-5.10",
  "boot_args": "console=ttyS0 reboot=k panic=1 pci=off i8042.noaux i8042.nomux i8042.nopnp i8042.dumbkbd",
  "initrd_path": null
}

There's no bootloader entry, no firmware selection, no device tree to author on x86 — just the kernel path and its command line. That directness is what makes the guest kernel a first-class thing you own: the config you built and the args above are the entire contract between Firecracker and the OS inside the guest.

The catch: your kernel is pinned into every snapshot

Here's the part that bites people who treat the guest kernel as a swappable dependency. A Firecracker snapshot is a serialization of a running machine — the guest's physical RAM plus the VMM and vCPU state at a single instant. That memory image contains a running kernel: its code, its data structures, its page tables, its device state, all captured mid-execution. The snapshot is not "a disk you can boot with any kernel" — it's a specific kernel that was running, frozen. You cannot restore that memory image under a different kernel and expect it to make sense; the restored guest is the one that was snapshotted, kernel and all.

The practical consequence: the guest kernel version (and Firecracker's own version) is effectively pinned into a snapshot. Rebuild your kernel — a new version, a changed config, even a different set of compiled-in drivers — and any snapshot taken with the old kernel is now describing a machine that no longer exists in your fleet. Restoring stale snapshots across a kernel change is not supported; you re-bake. Firecracker is explicit that snapshots are tied to the versions that produced them, and a kernel swap is one of the changes that invalidates them. Verify the exact compatibility rules for your Firecracker version against its snapshot documentation before you plan an upgrade.

Why PandaStack cares

PandaStack runs every sandbox, managed Postgres database, and git-driven app on Firecracker microVMs booting a minimal guest kernel, and both halves of this post are load-bearing for how it feels to use. The lean config is part of why the once-per-template cold boot lands around 3 seconds instead of the tens of seconds a general-purpose VM takes — there's simply less kernel to initialize and no phantom hardware to probe. After that first boot, PandaStack bakes a snapshot and every subsequent create is a restore: roughly 179ms p50 and about 203ms p99, because a restore resumes a machine whose kernel is already up rather than booting one. The kernel work and the snapshot work compound — a small kernel makes the one cold boot fast, and snapshot-restore makes sure you almost never pay for it again.

The snapshot-pinning reality is exactly why PandaStack treats a kernel change as a template re-bake rather than a live swap: re-baking invalidates the old snapshot on purpose and triggers a fresh cold-boot-and-bake on the next spawn, so the fleet converges on the new kernel cleanly. The same primitive underlies forks (same-host 400–750ms, cross-host 1.2–3.5s, both restoring a kernel that's already running) and managed databases (30–90s, which blocks on Postgres bootstrap, not the kernel). Networking is per-sandbox virtio-net in its own namespace, backed by the NATID pool that tops out at 16,384 /30 subnets per agent — the guest's only NIC is the virtio-net device this kernel config keeps. The core is open source under Apache-2.0, so you can build your own guest kernel, run the control-plane API and per-host agent on your own Linux KVM hosts, and watch a stripped vmlinux boot and get baked. For the boot-vs-restore framing start with /blog/how-firecracker-boots-fast; for the device side of this same minimalism, /blog/firecracker-virtio-devices.