VM exits: the actual currency of virtualization overhead

Somebody will eventually tell you that "virtualization costs about 2-3%." It is one of those numbers that sounds authoritative and means nothing, because the overhead of running under a hypervisor is not a percentage tax applied evenly to your instructions. It's a per-event charge, and the event is called a VM exit. A guest doing tight numerical work can run for billions of instructions without triggering a single one. A guest hammering a virtual disk can trigger a flood of them. Same hypervisor, same host, wildly different "overhead" — because overhead was never a property of the hypervisor. It's a property of your workload.

This post is the working engineer's version of a topic that is otherwise only documented in a CPU vendor manual you do not want to read. What a VM exit is, what the round trip through KVM (and sometimes further, out to the userspace VMM) actually involves, which exits a microVM realistically hits, why virtio looks the way it does once you see exits as the cost, why demand-paged snapshot restore is the most expensive corner of the whole thing, and how to measure all of it yourself. I'm Ajay — I build PandaStack, which runs Firecracker microVMs as a service, so this is a subject I have had to care about in production. If you want the layer underneath first, /blog/kvm-explained-for-developers covers what KVM is; this post is about what it costs.

What a VM exit actually is

When a VM is running, the guest's instructions execute directly on the physical CPU. Not emulated, not interpreted — the real silicon, at real speed. The CPU is in a special mode (Intel calls it VMX non-root; you'll see it written as "guest mode") that behaves almost exactly like normal execution, with one difference: a configurable set of events and instructions are marked as things the guest is not allowed to complete on its own.

When the guest hits one of those, the CPU stops it mid-flight, saves the guest's state, flips out of guest mode into host mode, and jumps to the hypervisor's handler with a recorded reason code explaining why. That transition is the VM exit. Once the host has dealt with it, the reverse transition — a VM entry — restores guest state and resumes the guest exactly where it was, with no idea anything happened.

The important thing is what does not cause an exit. Arithmetic doesn't. Branches don't. Reading and writing the guest's own memory doesn't, once the second-level page tables (EPT on Intel, NPT on AMD, stage-2 on ARM) have a mapping for it. And — this trips up almost everyone the first time — a system call inside the guest does not cause a VM exit. The guest's userspace calls into the guest's own kernel, which is right there in guest mode. The host never hears about it. That's why a syscall-heavy but self-contained workload can still be nearly free under virtualization.

The round trip: KVM, and sometimes further out

On Linux the handler on the other side of the trap is KVM, inside the kernel. KVM looks at the exit reason and makes a decision that matters enormously for performance: can I deal with this myself, or does userspace have to get involved?

A large fraction of exits, KVM handles alone. Populating a missing second-level page mapping, emulating a model-specific register read, servicing the virtual interrupt controller, dealing with a halted vCPU — these never leave the kernel. The vCPU thread is sitting inside a blocking `ioctl(vcpufd, KVM_RUN, 0)`; KVM does the work and re-enters the guest without that ioctl ever returning. From the VMM's perspective, nothing happened at all.

The other kind is the one that costs you. When the guest touches something only the userspace VMM knows how to emulate — a virtio device register, a serial port — KVM gives up and returns from `KVM_RUN` with an exit reason in a shared memory structure. Now you've paid the hardware transition plus a return to userspace, plus the VMM's device emulation logic, plus the trip back down through the ioctl to re-enter the guest. Same hardware event, considerably more work.

/* The VMM's entire life, compressed. Error handling omitted. */
for (;;) {
    /* VM entry. The vCPU now runs guest code natively in guest mode.
     * Plenty of VM exits happen INSIDE this ioctl -- KVM handles them
     * in the kernel and re-enters the guest without ever returning.
     * It returns only when KVM decides userspace must be involved. */
    ioctl(vcpufd, KVM_RUN, 0);

    switch (run->exit_reason) {

    case KVM_EXIT_MMIO:
        /* Guest touched an address with no memory behind it -- i.e. a
         * device register the VMM is responsible for emulating. This is
         * the expensive class: kernel -> userspace -> emulate -> back. */
        emulate_mmio(run->mmio.phys_addr, run->mmio.data,
                     run->mmio.len, run->mmio.is_write);
        break;

    case KVM_EXIT_IO:
        /* x86 port I/O (in/out): serial console and other legacy bits. */
        emulate_pio(&run->io);
        break;

    case KVM_EXIT_HLT:
        /* Only reaches userspace when there is no in-kernel irqchip.
         * With one (the normal case) KVM handles the halt itself:
         * it briefly polls, then deschedules the vCPU thread. */
        wait_for_interrupt();
        break;

    case KVM_EXIT_SHUTDOWN:
    case KVM_EXIT_FAIL_ENTRY:
    case KVM_EXIT_INTERNAL_ERROR:
        return;
    }
    /* Loop back into KVM_RUN -- that is a VM entry. The guest never
     * noticed it was paused. That is the entire illusion. */
}

That loop is every VMM ever written: Firecracker, QEMU, Cloud Hypervisor, the 200-line toy one in a blog post. The interesting engineering is entirely in how rarely you can make the `switch` execute. For how those vCPU threads map onto host cores, see /blog/firecracker-vcpu-scheduling-model.

The cost hierarchy (no numbers, on purpose)

I am not going to give you cycle counts, and you should be suspicious of anyone who does without naming their CPU, microcode revision, kernel version, and mitigation settings. What's stable and worth memorizing is the ordering. There are four tiers, and the gaps between them are large.

1. No exit at all. Guest arithmetic, guest branches, guest memory access to already-mapped pages, guest system calls into the guest kernel. This is free — the CPU is just executing instructions. Optimizing here is ordinary optimization, not virtualization work.
2. An exit KVM services entirely in the kernel. You pay the hardware world-switch (saving and restoring architectural state, plus the knock-on effects on caches, TLBs, and branch predictors) and a short kernel handler. Cheap, but not free, and it adds up when it happens millions of times a second.
3. An exit that returns to the userspace VMM. Everything in tier 2, plus exiting the kernel, plus the VMM's device emulation, plus re-entering. This is the expensive tier and it is the one worth designing around.
4. An exit that returns to userspace and then blocks on something slow — a disk, a network round trip, another process that has to produce a page for you. At this point the exit machinery itself is rounding error next to the wait, but the wait is real and it stalls a vCPU.

The ratios between those tiers shift with hardware generation and with the speculative-execution mitigations your kernel has enabled, since several of them add explicit work on every VM entry or exit. That is precisely why relative cost is the durable knowledge and absolute cost is not. Learn the ordering; measure the magnitudes on the machine you actually deploy on.

What actually causes exits in a microVM

A microVM has a deliberately tiny device model — no PCI bus, no BIOS, no graphics, no USB — which already eliminates whole categories of exit that a general-purpose VM pays. Here's what's left, with what typically handles each:

• MMIO access to a virtio device register — the guest driver reads or writes the device's configuration or status registers. There's no RAM behind that address, so the CPU exits. Usually goes all the way out to the userspace VMM, which emulates the register semantics and hands back a result. Tier 3: the expensive kind.
• Virtio notification kick (the "doorbell") — the driver has already written descriptors into the shared ring and now writes the queue-notify register to say "there's work." VMMs wire this address to a KVM ioeventfd so KVM converts the write into an eventfd signal and re-enters the guest immediately, letting an I/O thread pick up the work. That demotes a tier-3 exit to roughly tier 2, which is why the mechanism exists.
• EPT / NPT violation — the guest touches a guest-physical page that has no second-level mapping yet. KVM normally resolves this in-kernel: find or allocate the host page, install the mapping, resume. Individually modest, but every page of guest memory pays it at least once, so it dominates startup and shows up as "the VM is slow for the first second."
• Port I/O (x86 in/out instructions) — mostly the serial console in a microVM. Returns to userspace as KVM_EXIT_IO unless the address is registered with an ioeventfd. Chatty guest console output is a genuine, and genuinely avoidable, source of exits.
• External interrupt while the guest is running — a physical device interrupt arrives for the host. The CPU exits so the host can service it, then re-enters the guest. Nothing to do with your guest; it's the host's hardware being busy.
• Virtual interrupt injection — delivering an interrupt to a vCPU that is currently running generally means kicking it out of guest mode first, unless the hardware supports posted interrupts / APIC virtualization, in which case much of this is offloaded to silicon. This is the completion side of I/O, and it's why interrupt coalescing matters as much as notification batching.
• hlt — the guest has nothing to do and halts. With an in-kernel interrupt controller KVM handles this itself: it briefly halt-polls (there's a tunable for the poll window) on the theory that work is about to arrive, then deschedules the vCPU thread. A mostly-idle guest with a chatty periodic timer can exit constantly while accomplishing nothing.
• Timer-related exits — the guest programs its local APIC timer, or the hypervisor's own preemption timer fires. Handled in-kernel with an in-kernel LAPIC, but still one exit per tick per vCPU, which is the entire reason tickless guest kernel configurations exist.
• Privileged and sensitive instructions — CPUID, MSR reads and writes, certain control-register writes, explicit hypercalls. Almost always handled by KVM in-kernel. Harmless in normal code; a genuine problem if some library is calling CPUID inside a hot loop, which happens more often than you'd hope.

Why virtio is shaped the way it is

Once you internalize the cost hierarchy, virtio stops looking like a weirdly indirect way to do I/O and starts looking like exactly what it is: a design whose entire purpose is amortizing exits.

The rings — a descriptor table, an available ring, a used ring — live in guest memory that the VMM can read directly. No exit is needed for the guest to describe work, and none is needed for the VMM to read those descriptions. The only thing that requires an exit is the notification: the moment where the guest says "stop whatever you're doing and look." So the driver queues up as many buffers as it can and then rings the doorbell once. One exit, many operations. That ratio is the whole product.

The guest kernel thinks it wrote to a device register. What it actually did was file a support ticket with the hypervisor and block until someone in userspace got round to reading it.

The same logic explains the rest of the ecosystem's greatest hits. Notification suppression (each side can tell the other "don't bother me until you reach this index") turns a burst of small operations into a single kick and a single interrupt. Interrupt coalescing does it on the completion side. `ioeventfd` and `irqfd` let the kick and the injection be handled in-kernel rather than round-tripping to userspace. `vhost-net` goes further and moves the network datapath into a kernel thread entirely, so a kick never reaches userspace at all. And polling — a driver that never sleeps needs no doorbell — removes notification exits completely, at the price of a core burning continuously. That's the DPDK/SPDK bargain: trade CPU you have for exits you can't afford.

Worth noting that Firecracker deliberately does not take the vhost route. It keeps virtio-net and virtio-blk emulation in its own userspace process, accepting some exit cost in exchange for keeping device emulation out of the host kernel where a bug would be much worse. That's a security decision that shows up on your flame graph, and it's the right one for multi-tenant untrusted workloads.

The expensive corner: demand-paged snapshot restore

Here's where exits stop being an abstract performance topic and start being the thing that determines whether your product feels instant. Restoring a VM from a snapshot means getting a memory image back into guest-physical memory. You can load the whole thing up front — simple, but you wait for every page including the ones the guest will never touch — or you can map it and let the guest fault pages in as it needs them.

Demand paging is obviously the right call, and the mechanism is a stack of traps. The guest touches a page. No second-level mapping exists, so: VM exit. KVM goes to resolve it, and the host page isn't resident either, so: a host page fault. If that memory region is registered with `userfaultfd`, the kernel does not quietly allocate a page — it parks the faulting thread and sends a message to a userspace handler process. The handler produces the page contents (from a local file, from a peer host, from object storage over the network), copies it in with `UFFDIO_COPY`, and only then does the faulting thread wake, the mapping get installed, and the vCPU re-enter the guest.

So one guest memory access became: a VM exit, a host page fault, a wake-up of a different userspace process, possibly a network fetch, a copy, and a VM entry. Tier 4, comfortably. And you pay it per page.

Every optimization in this area is an attack on that per-page cost, and they're all variations on the same three ideas. Don't fault for pages you can predict: record which pages the guest touches immediately after restore and replay that set in the background, so the guest's faults land on already-resident memory. Don't fault for pages that contain nothing: track which regions of the memory image are entirely zeroes and satisfy those locally instead of fetching them. And make each fault cover more ground: back guest memory with 2 MiB pages instead of 4 KiB ones, and a single fault now resolves what would otherwise have been 512 separate trips through that whole stack.

That last one has a sharp edge worth knowing: with Firecracker, hugepage-backed guest memory can only be restored through the userfaultfd path, not by pointing at a memory file. Hugepage-ness is a property of the snapshot, not of the restore, so it's decided when you bake — and existing snapshots don't inherit it retroactively. On PandaStack this machinery is what lets a create be p50 179ms and p99 around 203ms, of which the restore step itself is roughly 49ms, against about 3 seconds for a first-ever cold boot with no snapshot to restore from. Same-host forks land in 400-750ms because the pages are already local; cross-host forks take 1.2-3.5s because they aren't. That gap is not a mystery — it's the fault handler's answer distance, one page at a time.

How to actually measure exits

Everything above is a model. Models are for generating hypotheses, not for concluding things. Linux gives you several ways to count exits directly, and you should use them before changing anything. Note the asymmetry that catches people out: a guest cannot observe its own exits — invisibility is the entire point — so all of this runs on the host.

# ---------------------------------------------------------------
# 1. Live, top-style view of exits grouped by reason. Needs root.
#    Most complete on x86; arm64 has the tracepoints but encodes
#    exit reasons differently, so output differs.
# ---------------------------------------------------------------
sudo perf kvm stat live

# Record then report, so you can diff two runs of the same workload.
sudo perf kvm stat record -a -- sleep 30
sudo perf kvm stat report --event=vmexit

# ---------------------------------------------------------------
# 2. Raw tracepoints. Every exit fires kvm:kvm_exit and every entry
#    fires kvm:kvm_entry. Counting them is the crudest useful signal
#    and often the only one you need to settle an argument.
# ---------------------------------------------------------------
sudo perf stat -e 'kvm:kvm_exit,kvm:kvm_entry' -a -- sleep 10

# Scope it to one microVM rather than the whole host:
sudo perf stat -e 'kvm:kvm_exit' -p "$(pgrep -n firecracker)" -- sleep 10

# Who is exiting, and why? The reason code is in the raw event.
sudo perf record -e kvm:kvm_exit -a -- sleep 5
sudo perf script | head -40

# ---------------------------------------------------------------
# 3. trace-cmd, if you prefer ftrace's output to perf's.
# ---------------------------------------------------------------
sudo trace-cmd record -e kvm:kvm_exit -e kvm:kvm_entry -- sleep 5
trace-cmd report | head -40

# ---------------------------------------------------------------
# 4. Cumulative counters straight from the kernel: one directory
#    per live VM, one subdirectory per vCPU.
# ---------------------------------------------------------------
sudo mount -t debugfs none /sys/kernel/debug 2>/dev/null || true
sudo ls /sys/kernel/debug/kvm/
sudo grep -r . /sys/kernel/debug/kvm/*/ 2>/dev/null | head -40

# Or kvm_stat, shipped in the kernel tree (tools/kvm/kvm_stat):
sudo kvm_stat -1     # one-shot snapshot of the counters
sudo kvm_stat        # live, curses-style

# Rule of the house: run YOUR workload and read YOUR numbers. Exit
# cost varies with CPU generation, microcode, speculative-execution
# mitigations, and kernel version. Any absolute figure you read in a
# blog post -- this one included -- is a measurement of someone
# else's machine on someone else's day.

Read the output as a distribution, not a total. "Four million exits in thirty seconds" tells you nothing on its own; "87% of them are MMIO to one virtio queue" tells you exactly what to go fix. If the top reason is MMIO into userspace-emulated devices, look at batching and notification suppression. If it's second-level page faults concentrated at startup, that's cold memory and the answer is prefetching, larger pages, or pre-touching. If it's `hlt` plus timer churn on an idle guest, look at the guest's tick configuration. If it's CPUID in steady state, that's a library, and it's your problem rather than the hypervisor's.

Two workloads, one microVM

The cheapest way to feel the difference is to run two deliberately opposite workloads in the same guest and watch the exit counter while they run. Here it is with PandaStack's Python SDK, so you don't have to build a kernel image and a tap device to try it — but the point holds on any KVM host you can get root on.

from pandastack import Sandbox

# Two workloads in the same microVM. One barely exits; the other exits
# on essentially every operation. Time them in the guest, and count the
# exits on the HOST with `perf kvm stat live` while this runs -- the
# guest cannot see its own exits, which is the whole point.

with Sandbox.create(template="base", ttl_seconds=600) as sbx:
    # A) CPU-bound. Once its pages are faulted in, this is arithmetic in
    #    the guest's own RAM: native speed, essentially no exits.
    sbx.filesystem.write(
        "/work/cpu_bound.py",
        b"import time\n"
        b"t = time.perf_counter()\n"
        b"x = 0\n"
        b"for i in range(50_000_000):\n"
        b"    x += i * i\n"
        b"print('cpu_bound', round(time.perf_counter() - t, 3))\n",
    )

    # B) I/O-bound, deliberately worst-case: O_DSYNC pushes every 4 KiB
    #    write out to the virtio-blk device instead of letting it sit in
    #    page cache. Each one means a doorbell and a completion IRQ.
    sbx.filesystem.write(
        "/work/io_bound.py",
        b"import os, time\n"
        b"t = time.perf_counter()\n"
        b"fd = os.open('/work/scratch.bin',\n"
        b"             os.O_CREAT | os.O_WRONLY | os.O_DSYNC, 0o644)\n"
        b"for _ in range(20_000):\n"
        b"    os.write(fd, b'x' * 4096)\n"
        b"os.close(fd)\n"
        b"print('io_bound', round(time.perf_counter() - t, 3))\n",
    )

    for name in ("cpu_bound", "io_bound"):
        run = sbx.exec(f"python3 /work/{name}.py", timeout_seconds=600)
        print(run.stdout.strip(), "| exit", run.exit_code)
# Sandbox destroyed on block exit. Now rerun it with YOUR workload
# shape. The relationship between those two timings, on your hardware,
# is the only "virtualization overhead" number that means anything.

You can make part B arbitrarily worse by shrinking the write size or adding an fsync per operation, and arbitrarily better by buffering and letting the ring batch. That tuning knob — how many operations ride along on each exit — is the same one virtio's designers were turning, and the same one you turn when you replace ten thousand small writes with one large one.

The takeaway

Virtualization overhead is not a constant, and asking "how much slower is a VM" without naming a workload is like asking how long a piece of string is while refusing to specify the string. What exists is a per-event cost, the event is a VM exit, and the number of them your workload generates spans several orders of magnitude depending on what it does. Compute-bound: nearly none, and near-native speed follows directly. Syscall-heavy but self-contained: still nearly none, because guest syscalls stay inside the guest. I/O-heavy, interrupt-heavy, or touching a lot of cold memory: exits everywhere, and that's where the honest overhead conversation lives.

Which is also why the microVM approach works at all. Deleting devices deletes exit sources. Batching in the ring means one exit does the work of many. Snapshot-restore skips the guest's entire boot-time storm of first-touch faults and device probing, and demand paging with prefetching, zero-elision, and 2 MiB pages attacks what's left of it. None of that is a trick — it's just what you build once you accept that exits are the currency and start counting them instead of quoting percentages. If you want to see it yourself, the PandaStack core is open source under Apache-2.0: run the agent on your own KVM host, start `perf kvm stat live` in another terminal, and go make some exits.