The PVH Boot Protocol: How Firecracker Skips Firmware

A conventional x86 virtual machine boots the way a physical PC has booted since the IBM 5150 in 1981. Power on, the CPU comes up in 16-bit real mode, and it starts executing firmware — a BIOS or, more modern, a UEFI implementation. That firmware initializes chipset registers, runs option ROMs for emulated devices, enumerates a PCI bus, sets up ACPI tables, POSTs the (virtual) hardware, and eventually hands control to a bootloader like GRUB. GRUB reads a config, maybe draws a menu, loads a compressed kernel and an initrd off a disk, then jumps into the kernel's real-mode entry stub, which laboriously walks the CPU up through real mode → protected mode → long mode before the kernel proper even begins. It works. It is also a museum tour, and you pay for the whole thing in wall-clock time every single boot.

Firecracker declines the nostalgia. There is no BIOS, no UEFI, no option ROMs, no GRUB, no real-mode stub, no PCI enumeration. The VMM (the Firecracker process itself) plays the role that firmware and bootloader would have played on real hardware — but it does the absolute minimum: it loads the kernel image into guest RAM, builds a couple of small data structures the kernel expects to find, puts the virtual CPU directly into 64-bit long mode with page tables already set up, and jumps straight to the kernel's entry point. The guest's first instruction is kernel code, running in the mode the kernel wants to run in. This is the mechanism behind the phrase "Firecracker boots in milliseconds," and this post is about how it actually works — the boot protocols, the ELF note, and the exact structures involved.

What all that legacy boot machinery actually does

It helps to be concrete about what Firecracker is skipping, because each stage is a real chunk of time and code. On a legacy x86 boot the sequence is roughly: reset vector at 0xFFFFFFF0 in real mode → firmware (BIOS/UEFI) initializes memory controller, runs option ROMs, builds ACPI/SMBIOS tables, probes buses → firmware locates a boot device and loads a bootloader → the bootloader (GRUB) loads the kernel and initrd into memory → control transfers to the kernel's boot entry, still in a constrained mode → the kernel's early setup code discovers its environment, switches the CPU to protected then long mode, sets up its own page tables, and finally calls into the C entry point. Every arrow there is work, and most of it exists to cope with the fact that real firmware doesn't know what kernel it's about to boot and the kernel doesn't know what machine it's on.

Inside a microVM, both of those unknowns evaporate. The VMM knows exactly which kernel it is loading, because you handed it the file. The kernel is going to run on a machine with a fixed, minimal, virtio-only device model — no surprise hardware. So the entire negotiation phase is unnecessary. You don't need firmware to discover memory when the VMM allocated the guest's memory and can just tell the kernel how much there is. You don't need a bootloader to find a kernel on a disk when the kernel is already a file the VMM has open. Direct kernel boot is what you get when you delete every step that only existed to handle ignorance the microVM doesn't have.

Firmware and bootloaders exist because real hardware doesn't know what it's booting and the kernel doesn't know what it's running on. In a microVM both parties already know everything. So you delete the introductions and shake hands directly.

Path one: the Linux/x86 64-bit boot protocol

The Linux kernel has long defined a documented boot protocol — the contract between a bootloader and the kernel — in Documentation/x86/boot.rst. Historically a bootloader dropped a compressed kernel (a bzImage) into memory, filled in a structure called the boot_params (informally, the "zero page" because it traditionally sat at the start of a page), and jumped to the kernel's real-mode entry point. Over time the protocol grew a 32-bit and then a 64-bit entry point so that a loader that has already put the CPU into protected or long mode can skip the mode-switching dance entirely.

That 64-bit entry point is what a VMM uses. The bzImage carries a small setup header near its front; the loader reads fields from it (where the protected-mode kernel code lives, the boot protocol version, flags), copies the kernel body into guest RAM, populates a boot_params structure, and jumps to the 64-bit entry with the CPU already in long mode and a pointer to boot_params in a register. The kernel skips its real-mode and protected-mode bring-up code and starts almost immediately in the 64-bit world. No firmware ran; the loader (here, the VMM) supplied everything the kernel needs to know.

The boot_params structure is how the VMM answers all the questions firmware would normally answer. The load-bearing fields for a microVM are things like:

• The memory map (an e820 table) — how much RAM the guest has and which ranges are usable. The VMM knows this exactly because it allocated the guest's memory, so it just writes the map in rather than making the kernel probe for it.
• The kernel command line pointer — the boot args string (console=ttyS0 reboot=k panic=1 pci=off …). Firecracker passes this through its boot-source API and the loader points boot_params at it; there is no bootloader config file involved.
• The initrd location and size, if you use one (Firecracker often doesn't — it boots straight to a rootfs on a virtio-block device).
• The setup header fields copied from the bzImage itself (protocol version, load flags), which tell the loader how to hand off correctly.

Path two: the PVH boot protocol (hvm_start_info)

The second path is cleaner still, and it's the one worth understanding because it comes from an unexpected place: Xen. PVH stands for Para-Virtualized Hardware. It began as a Xen guest boot mode designed for a world where the guest runs on real virtualized CPU features (hardware virtualization, real page tables) but boots through a lightweight paravirtual entry point instead of emulated firmware. Crucially, the PVH boot ABI was generalized so that any VMM — not just Xen — can use it to boot a kernel directly. Firecracker (via the rust-vmm ecosystem it shares code with) supports booting a kernel through the PVH entry point, and it's a beautifully minimal contract.

Here's the elegant part. Instead of the loader hunting for a magic offset, a PVH-capable kernel advertises its entry point in its own ELF headers. An uncompressed vmlinux is an ELF file, and ELF files can carry ELF notes — small typed metadata blobs. A kernel built with PVH support emits a note named XEN_ELFNOTE_PHYS32_ENTRY. That note contains a single 32-bit physical address: the address the VMM should jump to, in 32-bit protected mode with paging disabled, to start the kernel. The VMM reads the ELF, finds that note, and now knows exactly where to enter. The kernel published its own front door.

When the VMM jumps to that PHYS32_ENTRY address, it doesn't pass a boot_params zero page. It passes a pointer (in the %ebx register, per the PVH ABI) to a different, smaller structure: hvm_start_info. This is the PVH equivalent of the zero page, and it's deliberately lean — it describes the memory map, the command line, any modules (like an initrd), and an RSDP pointer if ACPI is present. The kernel has a dedicated PVH entry stub (pvh_start_xen in the Linux source) that reads hvm_start_info, translates it into the boot_params the rest of the kernel expects internally, and then joins the normal boot flow. So PVH is a thin, self-describing front door that funnels into the same kernel.

Checking a kernel for the PVH ELF note

Because the entry point lives in an ELF note, you can inspect any uncompressed vmlinux and see whether it supports PVH — and even read off the physical entry address the VMM will jump to. readelf dumps notes with -n; the Xen notes appear under a "Xen" owner. If you see PHYS32_ENTRY, the kernel is PVH-bootable.

# Dump the ELF notes in an uncompressed vmlinux and look for the Xen/PVH note.
# A PVH-capable kernel advertises its 32-bit entry via XEN_ELFNOTE_PHYS32_ENTRY.
$ readelf -n vmlinux | grep -A1 -i xen

# Typical output on a PVH-capable kernel (address will differ):
# Displaying notes found in: .notes
#   Owner                Data size   Description
#   Xen                  0x00000008  XEN_ELFNOTE_PHYS32_ENTRY
#     entry: 0x1000000

# No output at all? The kernel has no PVH entry point — it was built
# without CONFIG_PVH, and a VMM must use the Linux 64-bit boot protocol
# (bzImage / boot_params) instead.

# You can also see the note section itself via objdump:
$ objdump -j .notes -s vmlinux | head

The address printed under the note is the literal target the VMM's PVH loader will set the instruction pointer to. That's the whole handoff: read the note, set up the CPU state PVH mandates, drop an hvm_start_info in guest memory, point %ebx at it, and set %eip to that address. Firmware never entered the picture.

bzImage vs vmlinux: which boot path you're on

The two paths correspond to two kernel artifact shapes, and it's worth being precise about the difference because they're easy to conflate. A bzImage is the compressed, self-decompressing kernel a distro ships — it has a setup header and is built for the Linux boot protocol. A vmlinux is the raw, uncompressed ELF image, and it's the one that can carry the PVH ELF note. Firecracker's own guidance and reference configs lean toward booting an uncompressed vmlinux, which is exactly the artifact that makes PVH possible. The comparison, laid out:

• Artifact format — bzImage: compressed, self-decompressing, has a setup header. vmlinux (PVH): uncompressed ELF, carries ELF notes including the PVH entry.
• How the entry point is found — bzImage: loader reads the setup header and uses the documented 64-bit entry offset. vmlinux (PVH): loader reads XEN_ELFNOTE_PHYS32_ENTRY from the ELF and jumps to the advertised address.
• Boot structure passed — bzImage: boot_params (the zero page), pointer in a register. vmlinux (PVH): hvm_start_info, pointer in %ebx.
• CPU mode at entry — bzImage 64-bit protocol: enter in long mode. PVH: enter in 32-bit protected mode, paging off, and the kernel's PVH stub takes it the rest of the way.
• Origin of the protocol — bzImage: the native Linux/x86 boot protocol. PVH: inherited from Xen, generalized so any VMM can use it.
• What both skip — both: BIOS/UEFI firmware, option ROMs, GRUB, the real-mode boot stub, and PCI/ACPI discovery of hardware the microVM doesn't have.

The practical upshot: whichever path you're on, the shape is identical — the VMM loads the kernel, fills in a small structure that answers the questions firmware would have answered, sets the CPU into the mode the entry point expects, and jumps. PVH is the tidier of the two because the kernel self-describes its entry point and the start_info struct is smaller, but neither pays the firmware tax.

What the VMM sets up in guest memory before the jump

Jumping into a kernel isn't just "set %eip and go." The CPU expects a coherent execution environment, and normally firmware would have established it. In a direct-boot VMM, the VMM builds that environment by writing a handful of structures into guest RAM and initializing the virtual CPU's registers through the KVM API before it ever unpauses the vCPU. For the 64-bit boot protocol path, that includes:

• Initial page tables. To enter the kernel in long mode, paging must be on, so the VMM lays down a minimal set of page tables (typically an identity map of low guest memory: PML4 → PDPT → PD) in guest RAM and points the CR3 register at them. The kernel replaces these with its own almost immediately, but it needs valid tables to be running under at entry.
• A GDT (Global Descriptor Table). The VMM writes a small GDT with flat 64-bit code and data segments and loads the segment registers and GDTR to match, so the CPU is in a sane flat 64-bit segmentation state.
• The boot_params / hvm_start_info structure. Populated with the e820 memory map, the command-line pointer, and initrd info as described above, and placed at a known guest-physical address.
• vCPU registers. Via KVM_SET_SREGS / KVM_SET_REGS, the VMM sets CR0 (protected mode + paging bits), CR3 (page table root), CR4, the segment selectors, %rip to the kernel entry, and the register carrying the pointer to the boot structure. Then it resumes the vCPU and the very first guest instruction executed is kernel code.

For the PVH path the CPU-state contract is slightly different — the PVH ABI specifies entry in 32-bit protected mode with paging disabled, a flat GDT, %ebx pointing at hvm_start_info, and specific values in other registers — but the principle is the same: the VMM hand-crafts exactly the machine state the entry point documents, and nothing more. There's no firmware to establish it and no bootloader to negotiate it. This is the entire reason a microVM reaches its kernel's C code in a blink: the "boot" is a few kilobytes of struct-writing and a register load, not a POST sequence.

From the operator's side this whole handoff is configured with almost nothing: you point Firecracker at a kernel image and give it a command line through its boot-source API, and the VMM figures out which protocol to use and does the struct-writing for you. There is no bootloader entry, no firmware selection, no device tree to author on x86 — just a path and a string.

// PUT /boot-source — the entire "bootloader config" for a Firecracker guest.
// Point at the kernel image; Firecracker loads it directly and jumps in.
// An uncompressed vmlinux carrying the PVH ELF note boots via the PVH path;
// otherwise Firecracker uses the Linux 64-bit boot protocol.
{
  "kernel_image_path": "/var/lib/pandastack/kernels/vmlinux-5.10",
  "boot_args": "console=ttyS0 reboot=k panic=1 pci=off i8042.noaux i8042.nomux i8042.nopnp i8042.dumbkbd",
  "initrd_path": null
}

Why this is why microVM boot is milliseconds

Stack up what's been removed. No firmware phase (no BIOS/UEFI init, no option ROMs, no ACPI/SMBIOS table construction to sit through). No bootloader (no GRUB reading a config, drawing a menu, loading files off a disk). No real-mode stub and no mode-switching climb inside the kernel's early boot — the VMM already put the CPU where the kernel wants it. No PCI bus enumeration and no probing for hardware that isn't there, because the device model is fixed and virtio-only. What remains is: load the kernel image, write a few small structures, set registers, jump. The kernel then does its own (already lean, if you built a minimal config) initialization and comes up. That's the whole boot, and it's why direct kernel boot lands in the milliseconds range where a firmware-plus-bootloader VM spends seconds.

It's worth being disciplined about the qualitative claim here: "milliseconds" is the well-established character of a Firecracker-style direct boot relative to a firmware VM that takes seconds. The exact figure depends on the kernel config, the workload, and the host. The point of this post is the mechanism, not a stopwatch — direct boot is fast because of everything it structurally refuses to do, not because of a magic number.

And then you can skip even the boot

Direct kernel boot is the fast floor for actually booting a kernel. But there's a tier below it: don't boot at all. A Firecracker snapshot serializes a running microVM — the guest's entire physical RAM plus the VMM and vCPU state — at a single instant. Restoring that snapshot doesn't re-run the PVH handoff, doesn't rebuild page tables, doesn't jump to an entry point. It maps the frozen memory image (copy-on-write) and resumes the vCPUs mid-instruction, exactly where the snapshot froze. The kernel was already up, past its own init, with a warm page cache. Restore is closer to "resume a paused computer" than to "start one."

This is the path PandaStack runs in steady state. The very first time a template is spawned on a host, PandaStack does a real direct kernel boot — the PVH/boot-protocol handoff described above, plus the guest's userspace coming up — which takes on the order of 3 seconds end to end. Then it bakes a snapshot of that booted machine. Every create after that is a restore: the snapshot load itself is roughly 49ms, and the full create pipeline (network slot, rootfs reflink, VMM launch, snapshot load, resume, readiness probe) lands at about a 179ms median and ~203ms p99. The direct-boot protocol makes the once-per-template cold boot cheap; snapshot-restore makes sure you almost never pay even that. The mechanics of the restore path are covered in /blog/how-firecracker-boots-fast and the guest-kernel side in /blog/firecracker-guest-kernel-config.

So there are really three speeds of "boot" worth holding in your head: a legacy firmware VM (seconds, most of it firmware and bootloader theater), a Firecracker direct kernel boot via the Linux 64-bit or PVH protocol (milliseconds — the subject of this post — because all that theater is deleted), and a snapshot restore (faster still, because it doesn't boot at all, it resumes). PandaStack's substrate — every sandbox, managed Postgres database, and git-driven app — sits on the third, and the core is open source under Apache-2.0, so you can build a PVH-capable vmlinux, watch it boot directly, get baked into a snapshot, and then get restored in tens of milliseconds on your own Linux KVM hosts.