Debugging a Firecracker microVM That Won't Boot

You POST a machine config to Firecracker, resume the VM, and then… nothing. No SSH, no vsock, no logs, no error — just a firecracker process sitting there consuming a little CPU and giving you the silent treatment. A microVM that boots to a silent black hole is Firecracker's way of saying "you assumed the rootfs, didn't you." The kernel almost always knows exactly what went wrong; you just haven't given it a place to say so. That place is the serial console, and turning it on is the single highest-leverage move when a microVM won't boot.

I'm Ajay, I built PandaStack (a Firecracker microVM platform). This is the guide I wish existed the first fifty times a fresh template hung on boot: how to enable the serial console, how to read the kernel log it emits, the handful of failure modes that account for nearly every silent hang, and the trade-off that makes you strip the console back out again for production.

Enabling the serial console (console=ttyS0)

Firecracker emulates a legacy 16550A serial UART, and by default the guest kernel does not print to it — a fast boot is a quiet boot. To make the kernel narrate, you add `console=ttyS0` to the kernel command line. That command line lives in the boot-source config: the `boot_args` field you send to `PUT /boot-source` (or set in the JSON config file passed to `firecracker --config-file`).

{
  "boot-source": {
    "kernel_image_path": "/var/lib/pandastack/vmlinux-5.10",
    "boot_args": "console=ttyS0 reboot=k panic=1 pci=off root=/dev/vda rw init=/sbin/pandastack-init"
  },
  "machine-config": {
    "vcpu_count": 2,
    "mem_size_mib": 2048
  },
  "drives": [
    {
      "drive_id": "rootfs",
      "path_on_host": "/var/lib/pandastack/rootfs.ext4",
      "is_root_device": true,
      "is_read_only": false
    }
  ]
}

Two things have to line up for output to actually appear. First, the guest kernel needs the 8250/16550 serial driver compiled in (`CONFIG_SERIAL_8250=y` and `CONFIG_SERIAL_8250_CONSOLE=y`) — Firecracker's reference `vmlinux` builds have it, but a stripped custom kernel might not. Second, Firecracker needs somewhere to send that UART output. By default it goes to the process's stdout; the cleaner path is to point the VM's `log_path` at a file so the console (and Firecracker's own diagnostics) land somewhere you can `tail`.

Reading the kernel boot log

With `console=ttyS0` set, a healthy boot streams the familiar Linux dmesg: the kernel decompresses, brings up the CPUs, probes virtio devices, mounts the root filesystem, and finally hands control to init (PID 1). The last few lines before your guest agent comes alive are the ones that matter. A boot that gets stuck tells you where by what it printed last — and, just as usefully, what it never printed.

The most quoted failure line in all of Firecracker debugging is this one, and it means exactly what it says:

[    0.612044] VFS: Cannot open root device "vda" or unknown-block(0,0): error -6
[    0.613210] Please append a correct "root=" boot option; here are the available partitions:
[    0.614881] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

The kernel booted fine, then couldn't find a root filesystem to mount. It even lists the block devices it does see — if that list is empty, no virtio-blk device was attached; if it lists `vda` but your `root=` says `/dev/sda`, you named the wrong device. Either way the kernel dies at PID-1 handoff because there is nothing to exec. This one panic covers a surprising fraction of "my microVM won't boot" reports.

Common failure modes and what they look like

Almost every silent or panicking boot is one of a small set of problems. Match your last console line against these:

• Wrong root= device — panic "Unable to mount root fs on unknown-block(0,0)". Your `root=` names a device the kernel doesn't see. Firecracker's first drive is `/dev/vda`, not `/dev/sda` or `/dev/vdb`. Fix the `root=` to match the drive you actually attached.
• Missing virtio-blk driver — the kernel boots but the available-partitions list is empty. The guest kernel lacks `CONFIG_VIRTIO_BLK`/`CONFIG_VIRTIO_MMIO` (or they're modules with no initramfs to load them). Rebuild the kernel with virtio built in (=y), not as a module.
• init not found — "Kernel panic - not syncing: No working init found" or "Requested init /sbin/init failed (error -2)". The rootfs mounted, but the `init=` path doesn't exist or isn't executable inside it. Check the path exists in the image and has the exec bit.
• No init / empty rootfs — mounts cleanly, then panics with "Attempted to kill init!". The filesystem is there but there's no PID 1 to run. Usually a rootfs that was built empty or with the wrong layout.
• Silent hang, no panic — the console goes quiet mid-boot with no error. Often a kernel with no serial console driver (you see nothing at all), a `pci=off` mismatch, or the guest wedged waiting on a device that never arrives. If you see zero output, suspect the console setup itself before the boot.
• Network not coming up — the boot completes and init runs, but the guest is unreachable. This is not a kernel panic; the kernel is fine. The tap device, IP config, or guest-side networking failed. The serial console will show init running — you've moved past kernel debugging into guest/network debugging.

Turning on earlyprintk for the really early failures

`console=ttyS0` starts printing once the kernel's console subsystem is initialized. If your VM dies before that — a bad kernel image, a decompression failure, an early setup panic — the ttyS0 console is silent because it wasn't up yet. For those, add `earlyprintk=ttyS0` (or `earlyprintk=serial,ttyS0,115200`). It hooks the serial port far earlier in boot, before the normal console handoff, and surfaces panics that otherwise vanish.

# Broken: kernel dies silently, wrong root device, no early diagnostics.
boot_args="console=ttyS0 root=/dev/sda rw init=/sbin/init"

# Fixed: correct virtio root device (vda), early console, and
# panic=1 so a failed boot reboots (or exits) instead of hanging forever.
boot_args="console=ttyS0 earlyprintk=ttyS0 reboot=k panic=1 pci=off \
  root=/dev/vda rw init=/sbin/pandastack-init"

A few of those flags earn their place beyond just the console. `panic=1` reboots one second after a panic — with `reboot=k` (keyboard/triple-fault reboot) under Firecracker that means the microVM exits promptly instead of sitting wedged, so your control plane notices a failure instead of waiting on a timeout. `pci=off` skips PCI probing the guest doesn't need on Firecracker's minimal platform, shaving boot time and removing a class of early hangs. Keep them; they're the difference between a fast, loud failure and a slow, silent one.

Serial console vs firecracker.log vs guest-agent logs

Three log surfaces exist and they show different things — reaching for the wrong one is why boot debugging feels harder than it is. Which one you want depends on which layer is failing.

• Kernel serial console (console=ttyS0) — A: the guest kernel's own boot log (dmesg) plus early init output. B: it's where panics, root-fs mount failures, and driver probes show up. C: use it when the VM won't boot at all or hangs before your agent comes alive — this is the kernel talking directly to you.
• firecracker.log (VMM log) — A: the Firecracker process's view — API calls, snapshot load/restore, device setup, and hypervisor-level errors. B: it does not contain guest kernel messages unless you route the serial output into the same file. C: use it when the VMM itself errors (bad snapshot, config rejected, drive path missing) — i.e. the guest never even started.
• Guest-agent / app logs — A: everything after PID 1 is running — your init, your app's stdout/stderr, your guest agent's chatter. B: it assumes the boot already succeeded and the guest is reachable. C: use it for application bugs, not boot failures; if you can read these at all, the kernel and rootfs are fine.

The rule of thumb: if you get nothing from the guest agent, drop down to the serial console; if the serial console shows nothing at all, drop down to firecracker.log and check that the VMM even accepted your config and started a guest.

Why snapshot-restore hides all of this (until bake time)

Here's the twist that makes serial-console debugging a niche concern in production: a snapshot-restore platform doesn't boot the kernel at all on the hot path. On PandaStack, a create restores a baked Firecracker snapshot — the kernel already booted, mounted its rootfs, and ran init once, at bake time. Restoring that memory image is ~49ms; a p50 create is 179ms and p99 ~203ms. There is no `console=ttyS0` stream on a restore because there is no kernel boot to narrate — the guest wakes up already past all the failure modes above.

That's exactly why you don't want a chatty console in production. Every line the kernel prints to an emulated UART is synchronous, one byte at a time — a verbose console measurably slows a cold boot, and it's a small information-leak surface you'd rather not ship. Snapshot-restore sidesteps the cost entirely by skipping the boot, but even for the rare cold boot (the first spawn of a template, ~3s), a quiet console is faster.

So the workflow is: when a template won't bake — the one moment you actually cold-boot a kernel — you enable `console=ttyS0 earlyprintk=ttyS0`, watch the boot, and fix whatever the kernel complains about. Once the template boots clean and you've captured the snapshot, every subsequent microVM skips the whole sequence. You pay the debugging tax exactly once per template, not once per VM.

If you're on PandaStack and a baked template is misbehaving after it's up (not a boot failure — the guest is reachable), you read the guest side, not the kernel console. A quick exec pulls the guest's own boot and app logs:

from pandastack import Sandbox

# The guest is UP (this only works past a successful boot). For a
# won't-boot template, you'd be reading the serial console at bake
# time instead — this is the after-boot, guest-side view.
with Sandbox.create(template="base", ttl_seconds=120) as sbx:
    # Kernel ring buffer from inside the guest (post-boot dmesg).
    print(sbx.exec("dmesg | tail -n 40").stdout)

    # The guest agent / app log, not the Firecracker console.
    print(sbx.exec("cat /var/log/pandastack-app.log").stdout)

The distinction is the whole point: `dmesg` inside a running guest is the same kernel log the serial console would have shown — but you can only read it because the guest booted. When it doesn't boot, there's no guest to exec into, and the serial console is the only window you have. Wire it up at bake time and the kernel will tell you exactly what you got wrong.

For more on the boot path this all sits behind, see how Firecracker boots fast and the anatomy of microVM boot time; for building the guest kernel itself, the guest kernel config guide covers which options have to be built in.