Skipping DHCP: the Firecracker boot line you actually ship

There is a string that has been copied from the Firecracker getting-started guide into approximately every microVM orchestrator on earth. You know it: console, reboot, panic, pci=off, then a block of four i8042 tokens that nobody has read, then some combination of nomodule, random.trust_cpu and a root= that may or may not match the drive you actually attached. It is eleven or twelve tokens long, it is treated as a single indivisible incantation, and the number of teams who have measured what any individual token buys them is very close to zero.

I run PandaStack, which boots Firecracker microVMs as a product, so I went and looked at what we actually ship. Our production kernel command line is four tokens plus one generated argument. It is materially shorter than the thing on the internet, and the interesting part is not the tokens we dropped — it is the one we added, which almost nobody copies and which is the only token on the line with a defensible latency argument.

Where the command line lives, and how briefly it matters

A normal machine assembles its kernel command line out of firmware and a bootloader. GRUB reads a config, appends a few things, and hands the kernel a string. Firecracker has neither. You PUT a boot source over the API socket, the boot_args field of that object is the command line verbatim, and nothing edits it afterwards. No prompt, no append, no rescue entry. What you send is what the guest parses.

It is also a one-shot configuration. Boot-source calls are only legal before InstanceStart, and Firecracker rejects them outright on a snapshot-restore path — a restored guest never boots, so there is nothing to hand a command line to. Hold on to that, because it turns out to be the whole story about why these tokens matter and why they stop mattering.

# The two calls that configure a guest kernel, against the raw API socket.
# Order is not decorative: boot-source and drives must both land BEFORE
# InstanceStart, and every one of them is refused after it.
SOCK=/tmp/fc-demo.socket

curl -s --unix-socket "$SOCK" -X PUT http://localhost/boot-source \
  -H 'Content-Type: application/json' -d '{
    "kernel_image_path": "/var/lib/pandastack/kernels/vmlinux-5.10",
    "boot_args": "console=ttyS0 reboot=k panic=1 pci=off ip=10.200.0.2::10.200.0.1:255.255.255.252::eth0:off"
  }'

# root= is NOT on that line. The root device is a property of the drive,
# and the SDK/API derives the kernel argument from is_root_device. Setting
# both is how you end up with a command line that disagrees with your
# configuration, which the kernel resolves in a way you will not enjoy.
curl -s --unix-socket "$SOCK" -X PUT http://localhost/drives/rootfs \
  -H 'Content-Type: application/json' -d '{
    "drive_id": "rootfs",
    "path_on_host": "/var/lib/pandastack/vms/demo/clone.ext4",
    "is_root_device": true,
    "is_read_only": false
  }'

curl -s --unix-socket "$SOCK" -X PUT http://localhost/actions \
  -H 'Content-Type: application/json' \
  -d '{"action_type": "InstanceStart"}'

# Verify inside the guest rather than trusting the file you edited:
#   cat /proc/cmdline

One structural gotcha before we go further, because it makes every mistake in this post silent. The kernel does not reject parameters it does not recognise. Unknown parameters containing an equals sign are handed to init as environment variables; unknown parameters without one are passed as arguments. So a typo does not fail the boot — it boots a machine that quietly did not do the thing you asked, and hands PID 1 a mysterious environment variable as a souvenir. Every verification in this post is done from inside the guest for that reason.

What we actually ship

Here is the real thing, out of the agent's Firecracker driver. Four static tokens, and then a conditional network argument built from whatever address the host allocated for this particular sandbox.

// agent/internal/firecracker/driver.go -- the whole kernel command line.
cfg.KernelArgs = "console=ttyS0 reboot=k panic=1 pci=off"

// Kernel-level IP autoconfig: configures eth0 with the allocated guest
// IP/gateway BEFORE userspace runs. Avoids the need for DHCP, or for the
// guest agent to be the only path that brings the network up.
if d.spec.Network.GuestIP != "" && d.spec.Network.HostIP != "" {
    cfg.KernelArgs += fmt.Sprintf(
        " ip=%s::%s:255.255.255.252::eth0:off",
        d.spec.Network.GuestIP, d.spec.Network.HostIP,
    )
}

// Everything below this line is boot-time-only config. When restoring from
// a snapshot, Firecracker rejects boot-config calls entirely -- the SDK's
// snapshot path removes the CreateMachine/Drives/Network/KernelArgs
// handlers, because a restored guest has already made these decisions.

Things that are conspicuously absent, and the honest reason for each. No i8042 block: our guest kernel is a purpose-built config and the relevant probe cost depends entirely on whether those drivers are compiled in at all — if your kernel does not have the i8042 driver, the tokens are decoration, and if it does, you should measure the probe rather than assume it. No root= or init=, because the root device comes from is_root_device on the drive and we boot an ordinary init (more on that below). No quiet, because we want the serial console. No random.trust_cpu, which is a real decision with real consequences and one I will come back to.

The general principle I would defend: a boot line should be things you have a reason for, not a superset of every token you have ever seen. Extra tokens are not free in the way people assume — they are free at runtime and expensive at debugging time, because a line you cannot explain is a line you cannot bisect when a template stops booting.

ip= — the one token with a real latency argument

This is the part of the boot line nobody copy-pastes, and it is the part that actually earns its place. The default way a guest gets an address is DHCP: userspace comes up, a client sends a DISCOVER, waits for an OFFER, does the REQUEST/ACK round trip, and only then is there a network. On a machine that takes a minute to boot, this is invisible. On a microVM whose entire job is to be ready in under a second, a userspace round trip that happens after init has started is an enormous fraction of the budget — and the tail is worse than the median, because RFC 2131 has clients retransmit on a randomized exponential backoff starting in the region of four seconds. Lose one packet on a busy host and your sub-second boot became a multi-second one.

The kernel will do it for you instead, before userspace exists, with no server involved at all. That is what ip= is: the kernel's built-in IP autoconfiguration, wired up during boot so that when PID 1 starts, eth0 already has an address and a route.

The syntax is famously unfriendly — a colon-separated positional list where the empty fields carry meaning. Ours reads ip=10.200.0.2::10.200.0.1:255.255.255.252::eth0:off, which decomposes as:

1. client-ip — the guest's own address, from the host's allocation for this sandbox.
2. server-ip — empty. This is the NFS/boot-server field and means nothing to us.
3. gw-ip — the gateway, which is the host side of the veth pair for this sandbox.
4. netmask — 255.255.255.252, i.e. a /30: two usable addresses, one for the host end and one for the guest. There is no room in that subnet for a neighbour, which is exactly the point.
5. hostname — empty; we set it later through the guest agent, since it is per-sandbox and the command line is baked.
6. device — eth0, named explicitly rather than left to the kernel to pick, because 'the only interface' is an assumption and not a guarantee.
7. autoconf — off. This is the load-bearing token: it tells the kernel to use exactly what it was given and to make no attempt at DHCP, BOOTP or RARP. Leave it out and you can end up doing the discovery you were trying to avoid.

The /30 is not an aesthetic choice. Each sandbox on a host gets its own network namespace with its own veth pair and its own two-address subnet, carved out of a pool of 16,384 pre-allocated /30s per agent. Pre-allocating them is a boot-latency decision — creating a namespace, a veth pair, a TAP device and the matching firewall rules from cold is slow enough to dominate a sub-second create — and the ip= argument is the guest-side half of that same idea. The host has already decided what this guest's address is, so making the guest ask a server for it would be a round trip to learn a fact both ends already knew.

The console tradeoff, stated honestly

console=ttyS0 is the single easiest boot-time win on the list, and I am not going to recommend you take it. Serial output is genuinely slow — an emulated 8250 UART writing every early kernel message out one line at a time is real work, and turning it off, or turning it down with quiet and a lower loglevel, measurably shortens a cold boot. Every performance-oriented Firecracker writeup will tell you this and every one of them is correct.

It is also the only channel a guest has before it has a network, an SSH daemon, or a working init. A microVM with no console that fails to boot is a process that exits with nothing to say. You will be reduced to bisecting the rootfs, and you will spend more engineering hours on that one incident than the console cost you across every boot you ever ran.

The reconciliation is that these are not the same machine. Keep two lines, and make the difference an explicit configuration rather than an edit someone makes under pressure. Ours captures the serial console to a per-sandbox file on the host while also teeing it to the agent's own stdout, which is the compromise I would defend: you pay the UART cost, and in exchange every VM that ever misbehaves has a log you did not have to predict needing.

• Console — Debug line: console=ttyS0 with no quiet, plus ignore_loglevel and earlyprintk so you see messages emitted before the console is formally registered. Production line: console=ttyS0 kept but captured to a file, or dropped entirely to console= with quiet loglevel=3 if you have decided the boot is hot enough to matter.
• Boot tracing — Debug line: initcall_debug, which prints a timestamped line per initcall and turns 'boot is slow' into a ranked list. Production line: absent. It is very chatty and it is chatty over the same serial port you are trying to make cheap.
• Network — Debug line: ip= with autoconf off, same as production, because networking differences are the last thing you want between your debug and production paths. Production line: identical. Resist the temptation to make them differ.
• Failure behaviour — Debug line: panic=0 so a panicking guest sits there with the trace on the console and you can read it. Production line: panic=1 and reboot=k, so a panic is a fast, complete death that the supervisor observes rather than a wedged VM occupying memory until someone notices.
• Probe elimination — Debug line: leave the probes in; you want to see what the kernel found. Production line: pci=off and the i8042 family if and only if your kernel actually has those drivers compiled in, verified with initcall_debug rather than assumed.
• Entropy — Debug line: leave random.trust_cpu off and watch for 'random: crng init done' in dmesg so you can see how long the pool actually took. Production line: a decision, not a default — see the snapshot section below, because the correct answer changes when guests are clones.
• Verification — Debug line: cat /proc/cmdline as the first thing your smoke test does. Production line: same. It costs nothing and it is the only thing that proves the string you edited is the string the guest parsed.

How to attribute boot time instead of guessing

The reason everyone cargo-cults this line is that measuring it looks harder than copying it. It is not. The kernel is unusually cooperative about telling you where its time went, and two flags plus one command will get you a ranked list in about ten minutes.

Start with dmesg timestamps, which are seconds since the kernel started, not since the VMM started. That distinction matters: the gap between fork/exec of the Firecracker process and the kernel's first message is real time that dmesg will never show you, and on a fast microVM it is not a small fraction. Measure the outer number from the host and the inner numbers from dmesg, and be clear which one you are quoting.

Then add initcall_debug. It makes the kernel print a line when each initcall starts and another when it returns, with the duration in microseconds. Sort by that duration and you have an actual answer to 'what is my boot spending time on', which is nearly always different from what the boot-args folklore predicts. This is the flag that tells you whether pci=off is buying you anything on your kernel, rather than on the kernel of whoever wrote the blog post you copied.

# 1. Boot ONE guest with the debug line. The two tokens that matter here
#    are initcall_debug (per-initcall timing) and ignore_loglevel (so the
#    printk level does not silently swallow half of it).
#
#    boot_args: "console=ttyS0 reboot=k panic=0 initcall_debug ignore_loglevel \
#                ip=10.200.0.2::10.200.0.1:255.255.255.252::eth0:off"

# 2. Rank the initcalls by wall time. This is the whole measurement.
dmesg | awk '
  /initcall .* returned .* after [0-9]+ usecs/ {
    for (i = 1; i <= NF; i++) if ($i == "after") us = $(i+1)
    gsub(/\+.*/, "", $2)
    printf "%8d us  %s\n", us, $2
  }' | sort -rn | head -25

# 3. The three lines that answer most boot questions on their own.
dmesg | grep -E 'Command line|crng init|Freeing unused kernel'
#   Command line: ...        -> what the kernel PARSED, not what you sent
#   random: crng init done   -> when entropy stopped being a blocker
#   Freeing unused kernel    -> the kernel is done; everything after is you

# 4. Compare against the outer number. Kernel time is a SUBSET of boot time:
#    the VMM process start, the memory setup and the device configuration
#    all happen before the kernel says anything at all.
/usr/bin/time -f 'vmm-to-ssh: %e s' ./boot-and-probe.sh

Do this once per kernel config and once per template, write the numbers down, and then make changes against them. The single most common outcome — I would bet on it in advance — is that the tokens people argue about are not in the top ten, and the top ten is dominated by one or two initcalls for a subsystem the template does not need, which is a kernel config problem rather than a command-line problem. The command line can only skip work that the kernel was compiled to be capable of doing.

Then snapshots make most of it irrelevant

Everything above is about boot, and here is the twist that reorganises the whole subject: on a snapshot-restore platform, boot happens approximately never. On PandaStack every create restores a baked template snapshot. The full cold boot that produced that snapshot takes on the order of three seconds and happens once, at bake time; from then on a create is p50 179ms and p99 around 203ms, with the snapshot-load step itself around 49 milliseconds. Shaving probe work out of a cold boot you perform once per template release is not where any of your latency lives.

The kernel parses its command line exactly once and then it is over. The string becomes decisions — which devices got probed, how the console is wired, what the CRNG trusts, what the routing table says — and those decisions live in kernel memory. Snapshot that memory and you have captured the decisions. Restore it and you get a machine that has already made them, permanently, for every clone.

On a booting platform the command line is a latency knob. On a restoring platform it is a contract you froze, and every guest you will ever create has already signed it.

This flips which tokens deserve your attention. Probe elimination — the one thing everybody optimises — becomes the least interesting item on the line, because it only ever pays out during the bake. Meanwhile three tokens get considerably more important than they were:

• panic=1 and reboot=k. How a guest dies is a fleet-health property that applies identically to every restored clone. A guest that hangs on panic instead of dying holds its memory reservation until a human intervenes; a thousand restores of a snapshot with the wrong death semantics is a thousand potential wedges.
• The network configuration. The address in your ip= was baked in at snapshot time, so every restore comes up believing it is that address. The platform has to reconcile the frozen identity with the slot this particular clone actually landed in — for us that means patching the TAP MAC and routes on the host to match what was baked, and pushing the real per-sandbox identity into the guest afterwards. Whatever the mechanism, 'the guest's network identity is a snapshot constant' is a design fact you have to plan around, not an implementation detail.
• Entropy. This is the genuinely dangerous one. random.trust_cpu=on stops early userspace blocking on the CRNG, which on a booting machine is an unambiguous win. On a restoring machine, the entropy pool is part of the memory you snapshotted, so N clones of one snapshot resume with byte-identical random state. Two sandboxes generating a 'random' key at the same offset after restore get the same key. The kernel-side answer to this is a generation counter the hypervisor bumps on restore so the CRNG knows to reseed; the practical answer is that you must verify your kernel and VMM versions actually implement it rather than assuming, and never let a guest generate long-lived secrets from the pool it woke up with.

The clock deserves its own warning for the same reason. The guest's notion of time is frozen in the snapshot along with everything else, so a clone restored hours after its bake wakes up believing it is hours ago. That is not a cosmetic problem: it breaks TLS, which checks certificate validity windows against the local clock, and the failure presents as an inscrutable handshake error rather than as anything mentioning time. We learned this the direct way and now force a clock sync on restore, resume and wake. If you are snapshotting guests, put a clock resync in the resume path before you ship, not after your first incident.

init=, and the guest agent that is not PID 1

The maximalist version of microVM boot optimisation is init=/usr/local/bin/my-agent: skip the init system entirely and run a single purpose-built binary as PID 1. It is genuinely fast, it is what several serverless platforms do, and it is the right answer if your guest's job is to run one workload and exit.

We do not do it, and I would rather say so than let the architecture diagram imply otherwise. Our guests boot an ordinary init and our guest agent runs as a one-shot unit under it, ordered after the network and before the workload. The reason is that a general-purpose sandbox has to run software that expects a normal Linux: an SSH daemon, a Postgres that ships service units, a customer's app that assumes a service manager exists. Being PID 1 means inheriting PID 1's actual responsibilities — reaping orphaned children, handling signals correctly, ordering shutdown — and a guest agent that does that badly produces zombie processes and hung shutdowns, which is a worse problem than the milliseconds you saved.

The snapshot argument also cuts against it. If your create path is restore, the init system's startup cost is paid once at bake time, which means the strongest argument for init= evaporates precisely on the platforms most likely to be interested in it. Purpose-built PID 1 for a single-purpose function runtime: yes. For a sandbox that has to look like a computer: the milliseconds are not worth the semantics.

The short version

1. Read the line you are copying. Every token should have a reason you can state out loud, and a token whose reason is 'it was in the tutorial' is a token you cannot bisect later.
2. Add ip= with autoconf off. It is the one token on the line with an unambiguous latency argument, because it removes a userspace round trip whose tail is measured in seconds, and it needs no server. Check CONFIG_IP_PNP first.
3. Do not set root= by hand if your API or SDK derives it from the root drive. Two sources of truth for the same fact is a bug waiting for a drive-ordering change.
4. Keep the console, capture it to a file, and keep a separate debug line with initcall_debug and panic=0. The debug line is cheap insurance; the incident it prevents is not.
5. Measure with initcall_debug before you optimise. If the tokens you are arguing about are not in the top ten initcalls, you are tuning a kernel config problem from the command line, which does not work.
6. Verify from inside the guest with cat /proc/cmdline. Unknown parameters do not error, they become environment variables, so the only proof is what the kernel says it parsed.
7. If you restore snapshots, re-read the line as a correctness contract rather than a latency knob. Death semantics, network identity, entropy and the clock are the tokens that follow every clone; probe elimination is the one that stops mattering the moment your create path is a restore.

The honest summary of a Firecracker kernel command line is that it is a short, unusually candid description of what your machine has decided not to be — and that on a modern snapshot-based platform, most of that description is set once, by you, at bake time, and then inherited unread by every guest you ever create. Which is a good argument for writing four tokens you understand instead of twelve you inherited.