Firecracker boot_args, argument by argument

There is a string that appears in almost every Firecracker tutorial, gets copy-pasted into every orchestrator, and is read carefully by approximately nobody: `console=ttyS0 reboot=k panic=1 pci=off`. Sometimes it grows an `i8042.noaux` or two. It is the guest kernel command line, and it is one of the few places where a microVM tells you, in plain text, exactly which parts of a computer it has decided not to have. This post goes through a realistic line token by token — what each one does, what breaks if you drop it, and which ones you should be measuring rather than cargo-culting.

I'm Ajay; I build PandaStack, which runs Firecracker microVMs as a service, so I have spent an unreasonable number of evenings staring at exactly this string. Fair warning about the genre: kernel parameters are the single easiest thing on the internet to describe confidently and wrongly. Where I'm not certain about exact semantics I'll say so and point you at the kernel's own `Documentation/admin-guide/kernel-parameters.txt`, which is the only source that is actually authoritative for your kernel version.

Where the command line comes from

In a normal machine, the kernel command line is assembled by firmware and a bootloader — GRUB reads a config, appends some things, hands the kernel a string. Firecracker has no firmware and no bootloader. You PUT a boot source over the API socket with a kernel image path and a `boot_args` string, and that string is the command line, verbatim. Nothing edits it, nothing appends to it, and there is no interactive prompt where you can fix a typo. The upside is total determinism: what you send is what the guest parses.

// PUT /boot-source -- the entire boot configuration of the guest.
// `boot_args` is the kernel command line, byte for byte.
{
  "kernel_image_path": "/var/lib/pandastack/kernels/vmlinux-5.10",
  "boot_args": "console=ttyS0 reboot=k panic=1 pci=off i8042.noaux i8042.nomux i8042.nopnp i8042.dumbkbd nomodule random.trust_cpu=on root=/dev/vda rw init=/sbin/init"
}

One structural detail worth knowing before we start on individual tokens: the kernel does not reject parameters it doesn't recognise. Per the kernel's own documentation, unknown parameters containing `=` are placed into init's environment, and unknown parameters without `=` are passed to init as arguments. So a typo does not produce an error — it produces an environment variable. `pci=of` will boot a machine that probes PCI and hands your init a mysterious `pci=of` in its environment, and you will find this out much later than you would like.

console=ttyS0 — your only window

`console=ttyS0` tells the kernel to send its messages to the first 8250/16550-style serial port. Firecracker emulates that serial device and hands the output to the VMM process, which is why you can `tail` a plain file on the host and watch a kernel boot. In a machine with no display, no framebuffer, and no BMC, this is the entire debugging surface. If a guest dies before your agent has come up and started talking over vsock, the serial log is the difference between a root cause and a shrug.

You'll often see `console=ttyS0,115200n8`. The baud rate is a formality here — there is no wire and no UART timing to match, so the divisor the kernel writes is not throttling anything. Harmless to include, pointless to tune.

Turning it off, and what that costs

Serial output is not free. Each `printk` goes through an emulated device and out to the host, synchronously, while a vCPU waits. A chatty kernel or a verbose init genuinely spends boot time talking to a port nobody is reading. So once a template is stable, people reach for quieter options: drop the `console=` token entirely so the kernel has no console to write to, and — if you want the serial device gone rather than merely unused — `8250.nr_uarts=0`, which tells the 8250 driver how many ports to register at all.

The honest tradeoff: you have just removed your only view into a machine that has no other way to speak to you. A guest that wedges before the agent starts is now indistinguishable from a guest that is slow. My rule is that production templates may be quiet, but the debug variant of every template keeps a full console, and switching between them must be a one-line change in the orchestrator rather than a heroic rebuild.

reboot=k and panic=1 — how a microVM is supposed to die

On real hardware, shutting down is a negotiation: ACPI, firmware, power management, a long chain of things that can each hang. A microVM has none of that machinery, and you would not want it if you did. `reboot=k` selects the keyboard-controller reset path — a write to the i8042 controller's reset line, which is about the crudest reset x86 offers and, crucially, one Firecracker actually implements. Firecracker treats that guest reset as "this VM is done": the VMM exits and the slot is freed. Without a working reset path, `reboot` inside the guest can leave you with a VM that stops doing anything useful but is still very much running, still holding memory, and still on your bill.

`panic=1` is the companion: on a kernel panic, reboot after one second instead of parking on a panic screen forever. On a laptop, sitting at the panic message is helpful — a human reads it. In a fleet, it is the worst possible outcome, because the VM is dead in every way that matters except the one your scheduler measures. Combined with `reboot=k`, a panic becomes a process exit that your control plane can see, count, and replace. You can set a longer delay if you want the panic text to land on a slow log sink first; `panic=0` (wait forever) is the value to avoid in automation.

In a fleet, the expensive failure is not a VM that crashes. It's a VM that crashes and then keeps existing.

This matters more than it looks, because in a microVM PID 1 exiting *is* a panic. If your guest agent is `init` and it returns, the kernel panics — and `panic=1` plus `reboot=k` is what turns that from a wedged, still-billed machine into a clean exit.

pci=off — there is no bus

Firecracker's device model is virtio over MMIO: virtio-net, virtio-blk, virtio-vsock, a serial port, an entropy device. There is no PCI host bridge, no config space, no enumeration. `pci=off` tells the kernel not to go looking. Without it, the kernel spends startup scanning for a bus that structurally does not exist, and every device it fails to find is work you paid for and got nothing from.

I'm not going to quote you a millisecond figure, because the number depends on your kernel config, your CPU, and which probes your build actually compiles in — and a made-up number would be worse than none. What I can tell you accurately is the mechanism: it removes probing work that can never succeed. Measure it on your own kernel with `initcall_debug` (below) if you want to know what it's worth to you. Note also that this is an x86 concern; on aarch64, bus discovery comes from the device tree, so `pci=off` is not part of the usual argument set. Check Firecracker's docs for your architecture rather than porting a string across.

The i8042.* family — convincing Linux it has no keyboard

Here is my favourite part of the whole string: we spend real engineering effort convincing Linux that it does not, in fact, have a keyboard. The i8042 is the ancient PC keyboard/mouse controller, and its driver's probe sequence is a museum tour of 1980s hardware quirks — checking for an auxiliary port, testing for a multiplexing controller, consulting PnP/ACPI for where the controller lives. Every one of those steps involves waiting on a device that, in a microVM, is either absent or present only as a stub that exists so `reboot=k` has something to write to.

So the flags, per the kernel's parameter documentation: `i8042.noaux` skips checking for an auxiliary (PS/2 mouse) port. `i8042.nomux` skips probing for an active multiplexing controller. `i8042.nopnp` tells the driver not to use PnP/ACPI data to discover the KBD/AUX controllers and to use hardcoded values instead. `i8042.dumbkbd` tells it to treat the controller as read-only — it can take data from the keyboard but the driver should not try to control its state (this is the one that stops it trying to blink your nonexistent LEDs). Together they collapse an elaborate probe into approximately nothing, while leaving the reset path that `reboot=k` depends on intact.

root=/dev/vda rw — the disk, and why order matters

`root=` names the block device the kernel mounts as `/`, and `rw` says mount it writable rather than the default read-only. `/dev/vda` is the first virtio-blk device; a second attached drive becomes `/dev/vdb`, a third `/dev/vdc`. The naming follows the order the devices are presented, which means your root device name is a function of how your orchestrator built the machine, not of anything intrinsic to the disk.

That's the trap. Add a scratch volume or a durable data disk to a template and, if it lands ahead of your rootfs, `root=/dev/vda` now points at the wrong thing and the guest panics with "unable to mount root fs" — a failure that reads like a corrupt image and is actually an ordering bug. On a normal distro you'd sidestep this with `root=UUID=...` and an initramfs to resolve it, or `root=PARTUUID=...` if there's a partition table. A typical microVM rootfs is a raw, partitionless filesystem image booted with no initramfs, so neither is available: the kernel can only take a device name. The practical fix is discipline — attach the root device first, unconditionally, and pin that in the code that builds the machine rather than in a comment.

random.trust_cpu=on — entropy at boot

Early boot has an entropy problem: the kernel's random number generator needs to be seeded before anything can safely ask it for bytes, and a freshly created machine has almost no unpredictable events to harvest. Historically this produced the classic hang where userspace blocks waiting for the CRNG to initialise. `random.trust_cpu=on` tells the kernel to trust the CPU's hardware RNG (RDRAND on x86) to seed the CRNG, which resolves that immediately. The tradeoff is exactly what the name says: you are choosing to trust the CPU vendor's RNG implementation. That's a defensible call and it's what most cloud guests do, but it is a trust decision, not a free optimisation. Firecracker also exposes a virtio-rng entropy device, which is the other half of this story.

init= and nomodule — the last two tokens

`init=` overrides which program the kernel execs as PID 1 after mounting the rootfs. In a microVM this is usually pointed at a small guest agent rather than a full init system, because most of what systemd does — device management, socket activation, dependency ordering across dozens of units — is solving problems a single-purpose machine with five devices does not have. If the named path doesn't exist or isn't executable, the kernel falls back to its built-in list and panics if none work; `init=/bin/sh` is the best bisecting tool in your kit for exactly that reason. The full argument for what PID 1 owes the kernel (orphan reaping, signal handling, never exiting) is a whole topic on its own.

`nomodule` disables module loading. If you build a kernel with everything you need compiled in — which is the sane choice for a fixed, known machine — then nothing should ever be loading a module, and this makes that assumption enforceable instead of aspirational. It also stops userspace burning time on `modprobe` calls that were never going to find anything.

quiet and loglevel — observability for milliseconds

`quiet` lowers the console log level so only more serious messages get printed; `loglevel=N` sets it explicitly. Both reduce how much the kernel writes to the console during boot, which — see the console section — is real synchronous work. They do not stop the kernel *recording* messages: the ring buffer still fills, so `dmesg` inside the guest still shows you everything after the fact. What you lose is the messages that would have appeared on the serial port before userspace was alive to run `dmesg`, which is precisely the window where the interesting failures live.

Quick reference

• `console=ttyS0` — What it does: routes kernel messages to the emulated serial port, which the VMM gives you as a file on the host. When to drop it: on a stable production template where boot latency matters more than visibility, optionally with `8250.nr_uarts=0` to skip registering the port entirely. Cost: a guest that fails before your agent starts now fails silently.
• `reboot=k` — What it does: selects the keyboard-controller reset path, which Firecracker implements and treats as VM shutdown. When to drop it: essentially never on x86; without a working reset, an in-guest `reboot` can leave a VM that is dead but still resident.
• `panic=1` — What it does: reboot one second after a kernel panic instead of waiting forever. When to change it: raise the delay if you need panic text to reach a slow log sink first. Avoid `panic=0` in any automated fleet.
• `pci=off` — What it does: stops the kernel probing for a PCI bus Firecracker does not emulate (devices are virtio-over-MMIO). When to drop it: on aarch64, where discovery comes from the device tree and this isn't part of the standard set.
• `i8042.noaux i8042.nomux i8042.nopnp i8042.dumbkbd` — What they do: skip auxiliary-port detection, multiplexing-controller probing, PnP/ACPI discovery, and controller state control respectively — i.e. the legacy keyboard probe dance. When to drop them: if you are debugging something genuinely i8042-related, which for most people is never.
• `nomodule` — What it does: disables module loading, which makes "everything is built in" an enforced invariant. When to drop it: if your guest legitimately loads modules at runtime.
• `random.trust_cpu=on` — What it does: seeds the kernel CRNG from the CPU's hardware RNG so early userspace doesn't block on entropy. When to drop it: if your threat model doesn't permit trusting the CPU vendor's RNG. Does nothing for snapshot-restore entropy reuse.
• `root=/dev/vda rw` — What it does: names the first virtio-blk device as root and mounts it writable. When to change it: whenever drive ordering changes — the name follows attach order, so pin the order in code, not in a comment.
• `init=/sbin/init` — What it does: chooses PID 1. When to change it: point it at a guest agent for production, or `/bin/sh` to get a root shell at the earliest possible moment while debugging.
• `quiet` / `loglevel=N` — What they do: reduce console printing during boot; the ring buffer still records everything for `dmesg`. When to drop them: any time you are actually investigating a boot problem.

How to actually measure a change

Every claim above is mechanical — "this removes probing work," not "this saves N milliseconds." That's deliberate. The value of any of these flags depends on your kernel config, your CPU, and what your build compiles in, and anyone quoting you a universal number is quoting you their machine. Here's how to get your own number, which takes about ten minutes.

1. Pick a marker that means something. "Kernel booted" is not a useful endpoint; "my agent accepted its first request" is. Time to a marker you actually care about, not to the last line of dmesg.
2. Turn on timestamps. Build with `CONFIG_PRINTK_TIME`, or pass `printk.time=1`, so every serial line carries seconds-since-boot. Without timestamps you are eyeballing a log and calling it data.
3. Add `initcall_debug` for the diagnostic run. The kernel then logs each initcall and how long it took, which is how you find out what a probe actually costs on your hardware instead of guessing.
4. Change exactly one token, rebuild, and run both configurations many times — not twice. Boot timings on a busy host are noisy; look at a distribution, not a single sample.
5. Compare against the VMM's own view too. Firecracker has a boot-timer device for measuring guest boot from outside, which sidesteps the question of whether your in-guest clock agrees with reality.

# --- Inside the guest: what did the kernel actually receive? ---
# The single most useful command when a boot argument "isn't working."
# Nine times out of ten it proves you're debugging a different machine
# than the one you configured.
cat /proc/cmdline
# console=ttyS0 reboot=k panic=1 pci=off i8042.noaux i8042.nomux \
#   i8042.nopnp i8042.dumbkbd nomodule random.trust_cpu=on \
#   root=/dev/vda rw init=/sbin/init

# Did a typo silently become an environment variable instead?
tr '\0' '\n' < /proc/1/environ

# --- What the kernel did with it ---
# Timestamped boot log (needs CONFIG_PRINTK_TIME or printk.time=1).
dmesg | head -40

# Did it try to probe things you told it not to?
dmesg | grep -iE 'i8042|pci|serial|8250|random|virtio|rootfs'

# Was the CRNG seeded early, or did something block waiting for it?
dmesg | grep -i 'random:'

# With initcall_debug, this is where the time actually went.
dmesg | grep 'initcall' | sort -t= -k2 -rn | head -20

# Which virtio-blk device really ended up as root?
findmnt -no SOURCE /
lsblk

# Seconds since the guest kernel started -- compare against your own
# marker (agent ready, port listening) rather than trusting this alone.
cat /proc/uptime

That first command deserves emphasis out of proportion to its complexity. `cat /proc/cmdline` resolves an enormous share of "my boot argument isn't working" incidents, because the usual answer is that the argument never arrived: a stale template, an orchestrator with its own default string, or a snapshot baked before your change. Read the machine, not your config file.

If you're on PandaStack, the same check is one call — useful for confirming what a template was actually baked with, which is not always what the Dockerfile next to it implies.

from pandastack import Sandbox

# A real Firecracker microVM, created by restoring a baked snapshot.
with Sandbox.create(template="base", ttl_seconds=300) as sbx:
    # The command line this guest was actually booted with -- frozen at
    # bake time, not at create time. If it doesn't match what you think
    # you configured, you need a re-bake, not a restart.
    print(sbx.exec("cat /proc/cmdline").stdout)

    # And what the kernel made of it.
    print(sbx.exec("dmesg | grep -iE 'i8042|pci|random:'").stdout)

Snapshots freeze the command line

This is the part that catches people, and it's worth being blunt about. The kernel parses its command line once, during boot, and then it's over: the string becomes decisions about which devices to probe, how the console is wired, what the CRNG trusts, which binary is PID 1. Those decisions live in kernel memory. Take a snapshot and you have captured them. Restore that snapshot and you get a machine that already made them — the guest never boots again, so it never re-reads a command line.

This inverts the usual economics of the argument list. On a platform where every create is a cold boot, shaving probe work off the boot path pays out on every single VM, forever. On a snapshot-restore platform, boot happens once — at bake time — and thereafter you're paying restore cost, not boot cost. On PandaStack a create is p50 179ms and p99 around 203ms, with the restore step itself about 49ms; the cold boot that produced the snapshot in the first place took roughly 3 seconds and happened exactly once. Trimming a probe out of that one cold boot is not where your latency budget lives.

Which does not make the arguments unimportant — it changes *why* they matter. `reboot=k` and `panic=1` govern how a VM dies, which is a fleet-health property that applies to every restored guest regardless of how it was created. `root=` governs whether it boots at all. `random.trust_cpu=on` governs whether early userspace blocks. `console=` governs whether you can see any of it. Exactly one item on the list — probe elimination — is a boot-latency optimisation, and it's the one that a snapshot makes least urgent. The rest are correctness and operability, and they're frozen into every snapshot you ship.

The short version

The canonical Firecracker command line is not magic and it is not arbitrary. It says: give me a serial console because it's my only voice; die immediately and completely when you die; don't look for a PCI bus, a keyboard, a mouse, or a multiplexer, because none of them are here; take entropy from the CPU so nothing blocks; mount this specific virtio disk writable; and run this one program. That is a fairly complete description of a microVM, written in eleven tokens.

So: read the line you're copying, keep a debug variant with a full console, pin your drive ordering in code, verify with `/proc/cmdline` rather than with hope, and check `kernel-parameters.txt` for your kernel version before trusting any blog post about parameter semantics — including this one.