Firecracker Boot Sources: initrd vs a Root Block Device

A Firecracker microVM boots in a way that would look broken to anyone who learned computers on real hardware. No BIOS, no UEFI, no bootloader, no GRUB menu, no five-second countdown you can interrupt. The VMM opens a raw uncompressed kernel image, copies it into guest memory at the address the kernel's own header asks for, sets up boot parameters, and jumps to the entry point. That is the entire firmware story. It fits in a sentence because there is nothing in it.

So the interesting decision is not how the kernel starts — it's where it finds a userland. Two mechanisms, one hybrid. Path A: an initrd, a cpio archive unpacked into a tmpfs that lives in guest RAM. Path B: a root block device, a filesystem image on virtio-blk that the kernel mounts. Path C: a small initrd that does setup and then pivots onto the block device. I'm Ajay; I build PandaStack, which runs Firecracker microVMs as a service, and I think most people choose between these on the wrong axis. They argue about boot milliseconds. What bites them is RAM, image size, and whether the result can be copy-on-write cloned.

What Firecracker's boot source actually is

Firecracker's `/boot-source` endpoint takes three fields and only three: a kernel image path, a `boot_args` string, and optionally an `initrd_path`. That is the whole configuration surface for booting. The kernel must be a raw uncompressed image — not a self-extracting `bzImage` — because there is no firmware to run the decompression stub. Firecracker is the loader, and it wants something it can place in memory and jump to.

The presence or absence of `initrd_path` is the fork in the road. Include it and the kernel unpacks a cpio archive into an in-memory filesystem and execs a program out of that. Omit it and the kernel has no userland at all until it probes virtio-blk, finds the device your `root=` names, mounts a filesystem, and execs from there. Both end at a PID 1 in some root filesystem, but the paths differ, and so do their failure modes.

SOCK=/run/firecracker.sock
api() { curl -s --unix-socket "$SOCK" -X PUT "http://localhost$1" \
          -H 'Content-Type: application/json' -d "$2"; }

# =================================================================
# A) Boot source WITHOUT an initrd. The kernel is expected to find a
#    root filesystem on a block device and mount it itself.
# =================================================================
api /boot-source '{
  "kernel_image_path": "/var/lib/pandastack/kernels/vmlinux-5.10",
  "boot_args": "console=ttyS0 reboot=k panic=1 pci=off root=/dev/vda rw init=/usr/bin/pandastack-init"
}'

# The rootfs is a raw ext4 image on virtio-blk. The FIRST drive you
# present becomes /dev/vda -- so root= is a function of the order your
# orchestrator attaches drives, not of anything intrinsic to the disk.
# Attach the root device first, unconditionally, in code.
api /drives/rootfs '{
  "drive_id": "rootfs",
  "path_on_host": "/var/lib/pandastack/vms/abc123/clone.ext4",
  "is_root_device": true,
  "is_read_only": false
}'

# =================================================================
# B) Boot source WITH an initrd. One extra field, and the whole shape
#    of the boot changes: no root=, no /drives call, no block device
#    in the critical path. Userland arrives in RAM with the kernel.
# =================================================================
api /boot-source '{
  "kernel_image_path": "/var/lib/pandastack/kernels/vmlinux-5.10",
  "initrd_path": "/var/lib/pandastack/initramfs.cpio",
  "boot_args": "console=ttyS0 reboot=k panic=1 pci=off rdinit=/init"
}'

# Note rdinit= and not init=. init= names the program the kernel execs
# after mounting a real root filesystem; rdinit= names the one it execs
# out of the initramfs. Swap them and you get a panic whose text --
# "No working init found" -- is completely accurate and completely
# unhelpful, because the binary you are staring at does exist.

curl -s --unix-socket "$SOCK" -X PUT http://localhost/actions \
  -H 'Content-Type: application/json' \
  -d '{"action_type":"InstanceStart"}'

Everything else in this post is downstream of those two blocks. Same VMM, same kernel, one field of difference, two very different machines.

Path A: initrd, or userland that lives in RAM

Strictly, modern Linux uses an initramfs, not the old initrd. The distinction is mechanical: an old-style initrd was a filesystem image mounted as a RAM disk through the block layer; an initramfs is a cpio archive the kernel extracts straight into a tmpfs, with no block device and no filesystem driver involved. Firecracker's field is still called `initrd_path` for historical continuity, and what you hand it is a cpio archive. Everyone says initrd and means initramfs, including me.

The boot is beautifully self-contained. The kernel comes up, extracts the archive into tmpfs, makes that the root, and execs `/init`. No virtio-blk probe. No mount that can fail. No `root=` device name to get wrong when someone adds a second drive and shifts the ordering underneath you. The machine either has a working userland or the archive was malformed, and those two states are easy to tell apart. For an appliance that runs one static binary — a function runtime, a firewall, a unikernel-shaped thing — this is close to ideal.

#!/usr/bin/env bash
# Build a minimal initramfs: a cpio archive the kernel unpacks into a
# tmpfs. Every byte you put in here becomes RESIDENT GUEST RAM, in
# every VM you boot from it, for that VM's entire life. Budget it like
# memory, because it is memory.
set -euo pipefail
ROOT=$(mktemp -d)
mkdir -p "$ROOT"/{bin,sbin,etc,proc,sys,dev,tmp,run}

# 1. A userland. Static busybox is the usual choice -- one binary,
#    a directory of symlinks, and you have a recognisable shell.
cp /usr/bin/busybox "$ROOT/bin/busybox"
for applet in sh ls cat mount umount mkdir ip switch_root; do
  ln -s busybox "$ROOT/bin/$applet"
done

# 2. Your init, at the path rdinit= names. "Static" is load-bearing:
#    there is no ld.so in this archive unless you put one there, and a
#    dynamically linked binary fails in a way the kernel reports as a
#    missing init rather than a missing dynamic linker. That error
#    message has cost the industry more hours than it should have.
cp ./pandastack-init "$ROOT/init"
chmod +x "$ROOT/init"
file "$ROOT/init" | grep -q 'statically linked' \
  || echo "WARNING: init is dynamic -- ship its libs and its loader too"

# 3. Pack it. cpio 'newc' is the only format the kernel's extractor
#    understands. Paths must be relative -- note the leading ./ from
#    find -- or the kernel unpacks nothing and says very little.
( cd "$ROOT" && find . -print0 | cpio --null --create --format=newc ) \
  > initramfs.cpio

# 4. Compression is optional and it is a genuine tradeoff: a smaller
#    file for the host to read, paid for in guest CPU decompressing it
#    before a single line of userland runs. When the file is local and
#    on fast storage, uncompressed is frequently the faster boot.
#    Measure it on your kernel; do not take my word or anyone else's.
gzip -9 -c initramfs.cpio > initramfs.cpio.gz

ls -lh initramfs.cpio initramfs.cpio.gz
# The UNCOMPRESSED number is the one that matters. That is how much
# guest RAM each VM holds before it has done any useful work at all.

The costs are not subtle. The archive's uncompressed contents occupy guest RAM permanently: a 200 MB initramfs in a 512 MB guest has spent forty percent of that machine's memory on files before your workload allocates a byte, and a hundred VMs means a hundred copies. Worse, tmpfs pages are dirty anonymous memory to the host — neither reclaimable page cache nor shared between guests the way a read-only file mapping could be. It is the most expensive available way to store a filesystem.

The second cost shows up later: updates mean rebuilding and reshipping the whole archive. No layers, no deltas, no replacing one file. Change one line of your init and the artifact every host must pull is a new archive. Fine for a 5 MB appliance; miserable for anything with a package manager in it.

Path B: a root block device

The classic path. Build an ext4 image, attach it as a virtio-blk drive, set `root=/dev/vda rw`. The kernel probes virtio-blk, finds the device, mounts the filesystem, and execs whatever `init=` names. You pay a probe and a mount in the boot path — real work, though on a device model this small it is not the dominant term.

What that probe buys is everything the initramfs cannot give. The filesystem can be large without being expensive: it lives on disk, and its pages enter guest memory only when something reads them. It is writable, it persists, and its host-side page cache can be shared between guests reading the same blocks. And — the part that matters most — a filesystem image is a file, and files can be copy-on-write cloned. An XFS reflink or a dm-snapshot gives each VM a writable disk in O(metadata) time regardless of size: a 4 GB rootfs clones about as fast as a 400 MB one, sharing blocks with the template until written.

#!/usr/bin/env bash
# Build a raw ext4 rootfs from a container image. This is the trick
# nearly every microVM platform uses: Docker as the build system,
# the resulting filesystem as a block device. The container runtime is
# never involved at run time -- only its output is.
set -euo pipefail
IMAGE=pandastack/base:latest
OUT=rootfs.ext4
SIZE_MB=4096

docker build --platform linux/amd64 -t "$IMAGE" .

# `create` + `export` flattens the merged layers into one tar. Do NOT
# reach for `docker save` -- that gives you the layered image archive
# with its manifests, which is not a filesystem.
CID=$(docker create --platform linux/amd64 "$IMAGE")
docker export "$CID" > rootfs.tar
docker rm "$CID" > /dev/null

# A sparse file: `ls -l` reports 4G, `du -h` reports what is actually
# allocated. Only the allocated part costs you disk, and none of it
# costs you guest RAM -- which is the entire argument of this section.
truncate -s "${SIZE_MB}M" "$OUT"
mkfs.ext4 -F -q -L rootfs "$OUT"

MNT=$(mktemp -d)
sudo mount -o loop "$OUT" "$MNT"
sudo tar -xf rootfs.tar -C "$MNT"

# Things a container runtime normally provides at run time, which the
# kernel will now simply not do for you:
sudo mkdir -p "$MNT"/{proc,sys,dev,run}          # mount points must exist
echo 'nameserver 1.1.1.1' | sudo tee "$MNT/etc/resolv.conf" > /dev/null
sudo test -x "$MNT/usr/bin/pandastack-init" \
  || echo "FATAL: nothing executable at the path init= names"

sudo umount "$MNT"
e2fsck -fp "$OUT" || true
resize2fs -M "$OUT"   # shrink to minimum size; grow inside the guest

# Boot it with root=/dev/vda rw and NO initrd_path. Then reflink-clone
# this file per VM: cp --reflink=always rootfs.ext4 clone.ext4

The costs are honest ones. There is a block device in the critical path — a probe, a mount, and a failure class the initramfs path does not have: `root=` naming the wrong device. `/dev/vda` is whichever virtio-blk drive was presented first, so adding a scratch volume can silently reorder things and produce a guest that panics with "unable to mount root fs" — a message that reads like a corrupt image and is actually an ordering bug in your orchestrator. A normal distro dodges this with `root=UUID=...`, but resolving a UUID needs userspace, which needs an initramfs, which is the thing you were avoiding. Attach the root device first, in code, and pin it with a test.

Path C: the hybrid, and why it exists

General-purpose distros ship an initramfs not for speed but because mounting the real root sometimes requires running code first: LUKS decryption, dm-verity setup, assembling LVM or RAID, bringing up a network to fetch a layer, resolving a device by UUID. None of that fits inside the kernel's `root=` handling, so a small initramfs does the work and hands off.

The handoff is `switch_root`: it moves the new filesystem to `/`, deletes the initramfs contents to free the RAM they occupied, and execs the real init. That last detail catches people. It execs rather than forks, so whatever runs next is still PID 1 and inherits the full contract — reap orphans, install signal handlers, never return. If it returns, the kernel panics, and the panic names a process you thought was a normal daemon.

#!/bin/sh
# /init inside the initramfs. This is PID 1 -- briefly.
set -e
mount -t proc  proc  /proc
mount -t sysfs sysfs /sys
mount -t devtmpfs dev /dev

# The work that justifies having an initramfs at all. Pick your reason:
#   cryptsetup luksOpen /dev/vdb root      # decrypt
#   veritysetup open /dev/vdb root ...     # integrity-check
#   ip link set eth0 up && fetch-layer     # network-assembled rootfs
mkdir -p /newroot
mount -o ro /dev/vda /newroot

# Hand off. switch_root does three things: moves /newroot to /, frees
# the initramfs tmpfs (this is how you get the RAM back -- skip it and
# you pay for the archive forever), and EXECs the real init.
#
# It execs, it does not fork. Whatever runs next is still PID 1 and
# owes the kernel the same contract: reap orphans, handle signals,
# never exit. A clean `return 0` from that process is a kernel panic.
exec switch_root /newroot /usr/bin/pandastack-init

# Unreachable. If you get here, switch_root failed, and the next thing
# that happens is a panic. Leaving a debug shell here is the difference
# between a root cause and a shrug:
#   exec /bin/sh

The hybrid is right when you genuinely need pre-mount logic and over-engineering when you don't. If your rootfs is a plain unencrypted ext4 image on `/dev/vda`, an initramfs whose only job is to mount it and pivot has bought you a stage, an artifact, and a failure mode in exchange for nothing.

The kernel cmdline knobs that actually matter

The command line is the only configuration channel you have, so a few tokens are worth knowing. I'll describe mechanisms rather than quote millisecond savings — those depend on your kernel config and CPU, and anyone quoting a universal number is quoting their own machine. Full token-by-token treatment in /blog/firecracker-kernel-boot-args-explained; here's the boot-source-relevant subset.

• `init=` vs `rdinit=` — `init=` names the program execed after mounting the root filesystem; `rdinit=` names the one execed out of the initramfs. The most common initrd mistake, and its symptom is a panic about a missing init for a binary that plainly exists.
• `root=/dev/vda rw` — required on the block-device path, meaningless on the pure-initramfs path. `rw` mounts writable; the default is read-only, which gives you a userland that boots and then fails at its first write.
• `console=` — in development keep `console=ttyS0`; it's your only window into a machine with no display. Dropping it in production stops every kernel `printk` making a synchronous trip through an emulated serial device while a vCPU waits. Keep a debug variant of each template that has it back, switchable in one line.
• `quiet` / `loglevel=N` — reduce what reaches that console. The kernel still records messages, so `dmesg` in the guest has everything; what you lose is the window before userspace exists to run `dmesg`, which is where the interesting failures live.
• `reboot=k panic=1` — how a microVM is supposed to die. `reboot=k` uses the keyboard-controller reset path, which Firecracker implements and reads as "this VM is done"; `panic=1` reboots a second after a panic instead of parking forever. Together they turn a dead guest into a process exit your control plane can see, not a wedged VM you keep paying for.
• `pci=off` and the `i8042.*` family — suppress probes for hardware that structurally does not exist, removing work that could never have succeeded. These are x86 concerns; on aarch64, discovery comes from the device tree and the usual argument set differs.

Snapshots reframe the entire question

Here's the part that should change how you decide. Everything above is about booting. If your platform snapshots a booted machine and restores that snapshot on every create, then you boot approximately never, and the boot path's contribution to steady-state latency is approximately zero.

That's how PandaStack works. A template's first spawn is a real cold boot — roughly 3 seconds — and then the agent freezes the running machine's memory and device state. Every create after that restores it: p50 179ms end to end, p99 around 203ms, with the restore step itself around 49ms. The guest does not run init, probe virtio-blk, or unpack a cpio archive. It resumes mid-instruction in a machine that was already up, and how that machine originally found its userland is ancient history. Mechanics in /blog/how-firecracker-boots-fast.

Once you snapshot-restore, arguing about initrd boot time is arguing about a cost you pay once, in a build job, that nobody is timing. The costs that survive the snapshot are RAM and cloning — and those you pay on every single VM.

So flip the question. Not "which boots faster" but "which is cheaper to have ten thousand copies of." On that axis the initramfs looks worse than its boot-time reputation suggests: its contents are resident anonymous guest memory, so they sit inside `vm.mem`, the snapshot's memory file. A fat initramfs makes every snapshot bigger, every restore's working set larger, and every host's memory budget tighter — permanently, on every VM.

The block device goes the other way. The rootfs is deliberately a local file, because copy-on-write cloning needs one: a reflink gives each VM a private writable disk in constant time, blocks shared until written. Its pages enter guest memory only on demand, as reclaimable page cache. Ten sandboxes from one template share the template's blocks on disk and, for anything they only read, largely share the memory too. That is what density looks like, and an initramfs cannot participate — you cannot reflink RAM.

Side by side

• Where userland lives — initrd: a cpio archive unpacked into tmpfs, entirely in guest RAM. Root block device: an ext4 image on virtio-blk, read into page cache on demand.
• Guest RAM cost — initrd: the uncompressed archive size, resident for the VM's whole life, times every VM you run. Root block device: only the pages actually read, as reclaimable page cache.
• Boot-path work — initrd: no probe, no mount; extract and exec `rdinit=`. Root block device: a virtio-blk probe plus a mount, then `init=`. Real work, but small on a device model this minimal — measure it on your own kernel.
• Practical size ceiling — initrd: tens of megabytes before the RAM cost gets embarrassing. Root block device: gigabytes, sparse on disk, costing nothing until read.
• Writability — initrd: writable, but it's tmpfs, so writes consume more RAM and vanish with the VM. Root block device: a normal writable filesystem with real persistence and the option of separate durable volumes.
• Copy-on-write cloning — initrd: none. You cannot reflink RAM; every VM holds a full copy. Root block device: the whole point — reflink or dm-snapshot gives a per-VM writable disk in O(metadata) time, blocks shared until written.
• Updating it — initrd: rebuild and redistribute the entire archive for a one-line change. Root block device: rebuild or mutate the image, or publish a new template generation and let hosts sync it.
• Failure modes — initrd: malformed cpio, a dynamically linked init with no loader present, `init=` where `rdinit=` belonged. Root block device: `root=` naming the wrong device after a drive-ordering change, a missing mount point, an unmountable filesystem.
• Best fit — initrd: single-binary appliances, recovery images, kernel debugging, the setup stage of a hybrid boot. Root block device: anything with a package manager, a language runtime, user code, or a density target.

Which should I pick

Concretely, in the order I'd actually reason about it.

1. Running general-purpose or untrusted code — a code interpreter, a build, an agent's shell, anything with apt or pip in it: root block device, no initrd. You need gigabytes of userland, writable, clonable per VM without copying. It should take a lot to move you off this default.
2. A single static binary appliance under about 20 MB: initrd. Self-contained boot, no device ordering to get wrong, and at that size the RAM cost is noise. The simplicity buys real reliability.
3. Work is required before the real root can be mounted — LUKS, dm-verity, LVM, network-assembled storage, root by UUID: the hybrid. Keep the initramfs small enough that `switch_root` frees something worth freeing, and leave a debug shell after the `exec` so failure is diagnosable rather than merely fatal.
4. Debugging a kernel, a driver, or a rootfs that won't mount: a throwaway busybox initrd. Booting to a shell with no block device in the picture is the cleanest bisect available — if the machine is healthy there, the problem is your disk, not your kernel.
5. Optimizing steady-state create latency on a snapshot-restore platform: neither; stop here and bake a snapshot. That beats every boot-path optimization in this post combined, and it turns the choice above into a question about memory and cloning rather than milliseconds.

PandaStack takes the block-device path for every first-party template, not because block devices are elegant but because a create is a reflink of an ext4 image plus a snapshot restore, and both halves need a file on disk to be cheap. An initramfs would move gigabytes of template into every guest's RAM and into every snapshot's memory file, and delete the copy-on-write story that makes density work at all.

from pandastack import Sandbox

# None of the above is your problem here: no kernel image, no cpio
# archive, no root= ordering, no /boot-source PUT. The template's
# userland is an ext4 image, reflink-cloned per sandbox, and this
# create is a snapshot restore rather than a boot -- p50 179ms.
sbx = Sandbox.create(template="base", ttl_seconds=600)

# Worth confirming once on your own platform rather than believing a
# blog post. On a block-device boot, / is a real filesystem on a real
# virtio device, and the kernel's own command line tells you how it
# got there.
print(sbx.exec("findmnt -no SOURCE,FSTYPE /", timeout_seconds=30).stdout)
# -> /dev/vda ext4      (a rootfs boot; tmpfs here would mean initramfs)

print(sbx.exec("cat /proc/cmdline", timeout_seconds=30).stdout)
# -> ... root=/dev/vda rw init=/usr/bin/pandastack-init

# The size that would have been guest RAM under an initramfs, and is
# instead demand-paged page cache over a copy-on-write clone:
du = sbx.exec("df -h / | tail -1", timeout_seconds=30)
print(du.stdout, du.exit_code)

The summary worth leaving with: Firecracker's boot source is three fields, and the third — `initrd_path` — quietly decides whether your userland is memory or storage. Memory is simple, self-contained, and costs the same bytes in every VM you will ever run. Storage costs a probe and a mount and buys copy-on-write cloning, demand paging, and room to be large. Once you stop booting altogether, that second column is the whole comparison. Related: /blog/how-firecracker-boots-fast, /blog/firecracker-kernel-boot-args-explained, and /blog/firecracker-guest-init-pid1-explained for what the program at the end of either path owes the kernel.