How Firecracker uses the Linux KVM API: an internals explainer

Most explanations of Firecracker stop at "it's a lightweight VMM built on KVM." True, but it hides the interesting part. Firecracker doesn't contain a hypervisor — the hypervisor is the Linux kernel. Firecracker is an ordinary userspace process that assembles a virtual machine by making a small set of ioctl calls against a single device file, /dev/kvm, and then sits in a loop running the guest and servicing the handful of things the guest can't do on its own. This post walks that interface: opening /dev/kvm, creating the VM and its vCPUs, wiring guest memory, and the KVM_RUN loop where every VM-exit lands. Once you've seen the actual surface, the security argument — why a microVM exposes far less to the host than a container does — stops being marketing and becomes something you can count.

KVM is not a library — it's an ioctl API on a device file

KVM (Kernel-based Virtual Machine) is a Linux kernel module that turns the CPU's hardware virtualization extensions (Intel VT-x, AMD-V/SVM, Arm's EL2) into something userspace can drive. It exposes exactly one entry point: the character device /dev/kvm. There is no shared library, no daemon, no RPC — you open the device and issue ioctls. Critically, KVM provides CPU and memory virtualization and nothing else. It has no device model: no disk, no NIC, no timer, no BIOS. Those are the VMM's job. This division is the whole reason a VMM like Firecracker can be small — KVM hands you the hard, hardware-enforced part (a CPU that runs guest code natively and traps when it strays), and you supply only the thin device emulation on top.

There are three levels of file descriptor in the API, each with its own set of ioctls, and they nest. The /dev/kvm fd is the system handle (query the API version, create a VM). A VM fd comes from creating a VM on the system fd (set up memory, create vCPUs, configure interrupt routing). A vCPU fd comes from creating a vCPU on the VM fd (run the guest, read and write its registers). Everything Firecracker does against KVM is an ioctl on one of these three fds.

KVM_CREATE_VM and KVM_CREATE_VCPU

The first real call is ioctl(kvm_fd, KVM_CREATE_VM, 0), which returns the VM fd — an empty machine with no memory and no CPUs yet. From there, KVM_CREATE_VCPU on the VM fd creates each virtual CPU, one ioctl per vCPU, each returning its own vCPU fd. This is where Firecracker's threading model is set: it runs one host thread per vCPU, and each thread owns its vCPU fd and drives that vCPU's run loop independently. A 2-vCPU microVM is two host threads, each entering the guest on its own core. The host kernel schedules those threads like any other, which is why guest vCPUs are just host threads from the outside — you can see them, pin them, and cgroup-cap them with ordinary Linux tools.

Before a vCPU can run meaningfully, its architectural state has to be initialized: general-purpose and segment registers set with KVM_SET_REGS / KVM_SET_SREGS, and on x86 the advertised CPU features fixed with KVM_SET_CPUID2 (this is where Firecracker's CPU templates mask features so a snapshot taken on one host restores safely on another). These are the fiddly, architecture-specific details this explainer glosses — the shape that matters is: create the vCPU, set its initial state, then run it.

Guest memory: KVM_SET_USER_MEMORY_REGION and why it's the foundation for snapshots

This is the most important ioctl to understand, because it's the hinge that makes memory snapshots and copy-on-write possible. A guest needs physical RAM. In KVM, you don't allocate special hypervisor memory — you allocate ordinary host userspace memory (an mmap region in the VMM's own address space) and then tell KVM: "map this range of host virtual addresses to appear as this range of guest physical addresses." That declaration is KVM_SET_USER_MEMORY_REGION on the VM fd. It hands KVM a struct describing a guest-physical base address, a size, and the host virtual address of the backing memory.

From then on, when the guest touches guest-physical address X, the CPU's second-level address translation (Intel EPT / AMD NPT — nested page tables the guest cannot see or modify) resolves it to the corresponding host virtual address inside that mmap, and from there to real host physical memory. The guest's own page tables translate guest-virtual to guest-physical; KVM's nested tables translate guest-physical to host-physical. Two independent layers, and the guest controls only the first. It can never form a pointer into host memory, because the address space it can express simply doesn't include the host's.

Now the payoff. Because guest RAM is just an ordinary host mmap that the VMM owns, everything Linux can do to a memory mapping applies to guest memory for free. Snapshotting a VM's memory is, at bottom, writing that mmap out to a file. Restoring is mapping the file back in. Copy-on-write forking is mapping the memory MAP_PRIVATE so the kernel shares pages until the guest writes one and faults. Streaming memory in lazily on restore is registering that region with userfaultfd so a page fault becomes a fetch. None of these are KVM features — they're properties of the host memory mapping that KVM_SET_USER_MEMORY_REGION let you choose. That's why the ioctl is load-bearing far beyond boot.

The KVM_RUN loop and the kvm_run shared struct

With memory mapped and vCPUs initialized, the guest runs. Each vCPU has a shared communication struct, kvm_run, that the VMM obtains by mmap-ing the vCPU fd. This struct is how the kernel and the VMM talk without a syscall on every interaction: KVM writes the exit reason and any associated data (the I/O port and value, the MMIO address and bytes) into it, and the VMM reads them after each run. It's a mailbox that lives in memory both sides can see.

The vCPU thread then calls ioctl(vcpu_fd, KVM_RUN, 0). This ioctl blocks in the kernel while the physical CPU enters guest mode (VMX non-root / SVM guest mode) and executes the guest's instructions natively — arithmetic, its own memory accesses, its own kernel and userspace — with zero VMM involvement and at near-native speed. KVM_RUN returns only when the guest does something that traps: a VM-exit. The VMM reads kvm_run->exit_reason to find out why, handles it, and calls KVM_RUN again to re-enter the guest exactly where it left off. The guest never perceives the pause. This trap–handle–resume cycle is the entire runtime of a virtual machine.

/* ILLUSTRATIVE sketch of the KVM control flow a VMM runs.
 * This is NOT Firecracker source — it is a simplified pseudo-C model.
 * Error handling, register/CPUID setup, and IRQ config are omitted.
 * Verify real details against Documentation/virt/kvm/api.rst. */

int kvm = open("/dev/kvm", O_RDWR);

/* 1. Create the VM (system fd -> VM fd). Empty: no RAM, no CPUs yet. */
int vmfd = ioctl(kvm, KVM_CREATE_VM, 0);

/* 2. Back guest physical RAM with an ordinary host mmap, then map it in.
 *    This host mapping is what makes snapshot / CoW / UFFD possible later. */
void *guest_ram = mmap(NULL, ram_size, PROT_READ | PROT_WRITE,
                       MAP_SHARED | MAP_ANONYMOUS, -1, 0);
struct kvm_userspace_memory_region region = {
    .slot            = 0,
    .guest_phys_addr = 0x0,                    /* guest sees this as phys 0 */
    .memory_size     = ram_size,
    .userspace_addr  = (uint64_t)guest_ram,    /* host virtual address */
};
ioctl(vmfd, KVM_SET_USER_MEMORY_REGION, &region);

/* 3. Create a vCPU (one ioctl per vCPU; Firecracker runs one host
 *    thread per vCPU). Then set its initial register/CPUID state. */
int vcpufd = ioctl(vmfd, KVM_CREATE_VCPU, 0);
int runsz  = ioctl(kvm, KVM_GET_VCPU_MMAP_SIZE, 0);
struct kvm_run *run = mmap(NULL, runsz, PROT_READ | PROT_WRITE,
                           MAP_SHARED, vcpufd, 0);  /* the shared mailbox */

/* 4. Run the guest. KVM_RUN blocks in-kernel while the CPU executes
 *    guest code natively in guest-mode; it returns on a VM-exit. */
for (;;) {
    ioctl(vcpufd, KVM_RUN, 0);
    switch (run->exit_reason) {
    case KVM_EXIT_IO:    /* guest touched a PIO port  -> VMM emulates */
        /* run->io tells you port, direction, size, data offset */
        break;
    case KVM_EXIT_MMIO:  /* guest hit a device register -> VMM emulates */
        /* run->mmio tells you phys_addr, data, len, is_write */
        break;
    case KVM_EXIT_HLT:   /* guest executed HLT (idle / shutdown) */
        return 0;
    /* A handful of other reasons exist; a minimal VMM handles few. */
    }
    /* loop back into KVM_RUN to resume the guest where it stopped */
}

That switch statement is the entire host-facing behavior of the machine. Every way a guest can reach outside itself funnels through a bounded set of exit reasons, and the VMM decides — in host code the guest cannot see or influence — what each one means.

The VM-exit is where virtio device emulation happens

So what does the guest actually trap on? When guest code reads a disk block, sends a network frame, or writes to the serial console, it's really poking a virtio device — a paravirtualized device whose "registers" live at MMIO addresses the VMM chose. The guest's write to one of those addresses isn't backed by real hardware, so the CPU faults: KVM_EXIT_MMIO, back to userspace. The VMM reads run->mmio, recognizes the address as belonging to (say) its virtio-block device, and does the real work — reads the requested block off the host-side backing file, DMAs it into the guest's memory through the virtqueue, signals completion — then resumes the guest. To the guest, a disk read happened. In reality, a memory access trapped and a few thousand lines of host code serviced it.

This is the crucial reframe: the guest never makes a request to the host. It has no syscall into the host kernel, no call gate, no shared API. Its only channel to the outside is to do something that traps, and the CPU — not the guest — decides that trap is now the host's problem. The VMM emulates the device and hands control back. Every disk read, every packet, every console byte is a trap the VMM chose to service. There is no other door.

A container asks the host kernel for things through hundreds of syscalls. A KVM guest can't ask for anything — it can only trap, and a bounded set of VM-exits is the complete list of things it's even able to trap on.

Why this makes Firecracker small and secure

Firecracker's design choice follows directly from the shape of the API: implement as few exit handlers and as few emulated devices as possible. It handles the essential exits and emulates a minimal set of virtio devices (block, net, vsock, balloon, rng) plus a serial console and a one-button controller for clean shutdown — no BIOS, no PCI bus, no USB, no legacy hardware zoo. Historically, VM-escape CVEs cluster in device emulation code, because that's the host code parsing attacker-controlled input. Fewer devices means fewer exit paths means fewer lines of attacker-reachable host code means fewer places for the memory-corruption bug an escape needs. You can't have a vulnerability in a device you never emulate.

Set the two boundaries side by side as attack surfaces — this is the comparison that actually matters:

• What the untrusted code touches — Container: the full Linux system-call ABI (hundreds of syscall entry points) directly into the host's shared kernel. microVM: a bounded set of KVM VM-exits (a few exit reasons) into a tiny userspace VMM. No syscall into the host at all.
• Who enforces the boundary — Container: software (namespaces + cgroups) inside the one kernel everyone on the box shares. microVM: the CPU's virtualization hardware traps the guest first, before any host software runs.
• Attacker-reachable host code — Container: a general-purpose kernel with drivers, filesystems, and network stacks for every workload on the machine. microVM: a handful of virtio device models — a small, purpose-built codebase a security team can read end to end.
• Where escape CVEs historically live — Container: kernel bugs reachable through some syscall path. microVM: device-emulation bugs in the VMM — and there are far fewer devices to hold them.
• Blast radius of one bug — Container: the host kernel and every co-tenant on it. microVM: one VM (and even then, the VMM is jailed and seccomp-filtered, so a VMM compromise still isn't host root).

Neither boundary is magic — KVM and the VMM have had their own CVEs, and a serious deployment still runs the VMM through a jailer and a seccomp filter so that even a full VMM compromise lands in an unprivileged, chroot'd, syscall-restricted process rather than on the host. But the raw surface area isn't close. "Hundreds of syscalls into a kernel you share with strangers" versus "a few VM-exits into a few thousand lines of audited Rust" is the whole reason AWS built Firecracker on KVM to run Lambda and Fargate, and it's the boundary PandaStack runs every sandbox, managed database, and hosted app behind.

Where this shows up in a real platform

The KVM_SET_USER_MEMORY_REGION detail from earlier is exactly why a production platform can boot fast without keeping idle VMs warm. Because guest RAM is an ordinary host mmap, PandaStack bakes a booted microVM's memory into a snapshot file once, then every create maps that file back in and resumes the guest — no cold kernel boot on the hot path. A sandbox is live at roughly 179ms p50 (about 203ms p99), with the memory-restore step around 49ms; the ~3s cold boot happens once per template, then never again on the create path. Snapshot, copy-on-write fork, and lazy memory streaming are all just operations on that same host mapping the ioctl set up. The security surface and the speed come from the same small interface — you're not trading one for the other.

If you want to see the loop for yourself, this is the layer to read. Start with /blog/kvm-explained-for-developers for the hardware model underneath the API, and /blog/firecracker-vmm-security-model for the jailer and seccomp layers that wrap the VMM process. PandaStack's core is open source under Apache-2.0, so you can run the agent on your own KVM hosts, watch the VM-exits, and confirm every claim here against real /dev/kvm ioctls — and against Documentation/virt/kvm/api.rst and the Firecracker source, which are the authoritative references this explainer simplifies.