Interrupts, IRQs, and Where microVM Tail Latency Comes From

Median latency is a statement about whether your code works. The 99th percentile is a statement about how the machine underneath it is built. A p50 of 2ms and a p99 of 180ms is not a slow system with occasional hiccups — it is a system where the fast path and the slow path are structurally different paths, and you have been averaging them together and calling the result performance.

I'm Ajay, and I run PandaStack, a Firecracker microVM platform. Sandboxes are created by restoring a baked snapshot, which lands around 179ms p50, and I spend a genuinely irritating amount of my time on the difference between that number and the tail behind it. This post is about the mechanism that produces most of the tail inside a guest: the interrupt path. What a virtio interrupt actually does, how many separate queues it waits in, which of those queues get longer as a host fills up, and — the part almost nobody separates properly — the one tail-latency source that looks exactly like interrupt latency and isn't.

One interrupt, seven handoffs

Take the simplest interesting case: a process inside the guest calls read() on a socket, a packet arrives on the host, and the process wakes up with data. In a bare-metal world this is a NIC, a DMA write, an interrupt, a softirq, a wakeup. In a microVM it is all of that plus an entire second machine wedged into the middle, and the wedge is where the tail comes from.

Here is the sequence, host to guest, on a Firecracker VM with a virtio-net device:

1. The host receives the packet on the TAP device that Firecracker owns. On PandaStack this TAP lives inside a per-sandbox network namespace, one of a pool of pre-allocated /30 slots, so the packet has already crossed a veth pair and a NAT rule before it reaches the VMM at all.
2. Firecracker's device thread — a host thread, not the vCPU thread — picks the buffer up, copies it into a descriptor the guest published in the virtqueue's available ring, and marks that descriptor used.
3. The device thread signals an eventfd wired to KVM as an irqfd. This is the crucial optimisation of the modern virtio stack: the VMM does not have to enter a syscall on the vCPU thread to inject an interrupt. It writes to a file descriptor and the kernel does the injection.
4. KVM makes the interrupt pending for the target vCPU. If that vCPU is currently executing guest code on a physical core, the kernel sends an IPI to that core to force a VM exit so the interrupt can be delivered. If the vCPU had executed a halt instruction because the guest had nothing to do, the vCPU thread is instead marked runnable and must be scheduled.
5. The guest takes the interrupt. Its ISR runs, acknowledges the device, and does almost nothing else — it schedules the bottom half and returns.
6. The guest's softirq layer runs NET_RX. The virtio-net driver polls the used ring under NAPI, pulls frames out, runs them up the network stack, and eventually marks the socket readable and wakes the blocked task.
7. The woken task is now runnable inside the guest — which means it waits for the guest scheduler to pick it, on a vCPU, which is a host thread, which waits for the host scheduler to pick it.

Seven steps, and at least four of them are queues. Steps 4, 6 and 7 are all "something is now runnable and will run when a scheduler gets around to it." On an idle host every one of those queues is empty and the whole chain completes in tens of microseconds. That is your p50. The p99 is what happens when even one of them is not empty.

A vCPU is not a CPU. It is a host thread with a job title. Everything that surprises people about microVM tail latency follows from taking that sentence seriously.

The vCPU is a host thread, and it is not alone

Firecracker runs one host thread per guest vCPU, plus an API thread and per-device threads. You can see them by name: the vCPU threads are called fc_vcpu 0, fc_vcpu 1, and so on. We match on exactly that prefix in our own CPU-pinning code, because those are the threads whose scheduling behaviour a guest can feel.

Now count the threads on a busy host. Ten sandboxes at 8 vCPUs each is 80 vCPU threads, plus device threads, plus the agent, plus whatever the host itself is doing, all competing for a core count that is nowhere near 80. This is not a misconfiguration. It is the entire economic premise of a dense sandbox fleet: guests are idle most of the time, so selling burst capacity rather than reserved cores is the only way the arithmetic works.

Our own numbers, since I'd rather be concrete than coy. Every first-party PandaStack template bakes 8 vCPUs, and that 8 is a burst ceiling, not a reservation. Each Firecracker process gets its own cgroup v2 subtree with cpu.weight set to 100 times its vCPU count, so an 8-vCPU sandbox carries a weight of 800. On an idle host, weights bind nothing and a single sandbox can burst across every physical core. Under contention, the weights divide the cores proportionally rather than by thread-count lottery. CPU admission at the scheduler is disabled by default, deliberately — subtracting 8 reserved vCPUs per guest from a core budget double-counts the very thing burst is supposed to share, and when we did have that gate on with a 4x factor it capped an 8-core host at four concurrent sandboxes while half its RAM sat unused.

The honest consequence of that design is right there in the mechanism: when the host's run queue is long, your vCPU thread waits, and it waits at step 4 and step 7 of the sequence above. Nothing is broken. The interrupt was injected on time. The vCPU simply had not been scheduled yet when it arrived, and the woken task had to queue again behind other guests' vCPU threads before it could run. That is CPU steal, and to a guest it is indistinguishable from the machine mysteriously freezing for two milliseconds.

VM exits, kicks, and the cost of asking politely

The other direction has its own tax. When the guest driver wants to tell the device it has queued work, it writes to the virtqueue notify register. That write is to an MMIO address, and MMIO writes trap: the CPU leaves guest mode. A VM exit is on the order of a few thousand cycles on modern hardware — not catastrophic, but not free, and very much not free if you do one per packet.

The mitigation is ioeventfd. KVM can be told that writes to a specific MMIO address should simply signal an eventfd rather than being bounced out to the VMM's userspace handler, which turns "exit to the VMM, decode the instruction, dispatch to a device model" into "exit to the kernel, write a counter, re-enter." The device thread wakes on the other side. It is the mirror image of irqfd, and between the two of them the common path stays out of the VMM's userspace almost entirely.

Firecracker's guests boot with pci=off in the kernel command line — I can point at the exact line in our driver, it is console=ttyS0 reboot=k panic=1 pci=off — which means virtio is transported over MMIO rather than PCI. That has a consequence people trip over when they go looking for tuning knobs: there is no MSI-X vector table to spread across queues in the way a PCI virtio-net device would offer. It is a simpler, smaller device model, which is the whole point of Firecracker, and it means several of the interrupt-tuning tricks you might remember from QEMU are not there to be turned.

The guest's own bottom half is a jitter source

Even with an idle host and a perfectly delivered interrupt, the guest can add tail on its own. Linux runs softirqs either at the end of the interrupt handler or, when there is too much work, by deferring the rest to the ksoftirqd kernel thread — which is an ordinary schedulable thread that gets in line with everything else on the guest's run queue. Under load the network stack ping-pongs across that boundary, and the difference between "processed in the interrupt tail" and "processed after ksoftirqd gets scheduled" is a large multiple, not a small delta.

NAPI's budget is the other half of this. The poll loop drains a bounded number of packets per pass, then yields so the CPU is not monopolised. If your request's packet is the one just past the budget, it waits for the next pass. None of this is specific to virtualisation — it is how Linux networking has worked for two decades — but inside a microVM it stacks on top of every scheduling queue described above, and the stacking is what produces a p99 that looks nothing like the p50.

Measure the thing, and measure it as a distribution

The single most common mistake I see when someone brings me a latency complaint is that they measured the mean. A mean over a bimodal distribution describes neither mode. It moves when the ratio between the modes shifts and tells you nothing about why. If your fast path is 200µs and one request in a hundred takes 40ms, the mean is about 600µs, which is a number that has never happened and never will.

Record every iteration. Sort. Print percentiles and the maximum. Here is a harness small enough to paste into a sandbox, which measures a round trip that actually exercises the interrupt path rather than spinning on the CPU:

# latprobe.py -- per-iteration latency, reported as a distribution.
# The work here is a loopback socket round trip: it crosses the guest's
# network stack, which means it exercises softirq processing and a task
# wakeup. Swap in whatever your real request path is; the reporting is
# the part worth copying.
import socket, statistics, threading, time

N = 20000
WARMUP = 2000

def echo_server(sock):
    conn, _ = sock.accept()
    with conn:
        while True:
            b = conn.recv(64)
            if not b:
                return
            conn.sendall(b)

srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
srv.bind(("127.0.0.1", 0))
srv.listen(1)
threading.Thread(target=echo_server, args=(srv,), daemon=True).start()

cli = socket.create_connection(srv.getsockname())
cli.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)

payload = b"x" * 32
samples = []
for i in range(N + WARMUP):
    t0 = time.perf_counter_ns()
    cli.sendall(payload)
    cli.recv(64)
    dt = time.perf_counter_ns() - t0
    if i >= WARMUP:          # discard warmup: see the UFFD section below
        samples.append(dt)

samples.sort()
def pct(p):
    return samples[min(len(samples) - 1, int(len(samples) * p))] / 1000.0

print(f"n       {len(samples)}")
print(f"mean    {statistics.mean(samples)/1000:9.1f} us   <- the lying number")
print(f"p50     {pct(0.50):9.1f} us")
print(f"p90     {pct(0.90):9.1f} us")
print(f"p99     {pct(0.99):9.1f} us")
print(f"p99.9   {pct(0.999):9.1f} us")
print(f"max     {samples[-1]/1000:9.1f} us")

# The shape of the gap is the diagnosis:
#   p50 ~= p99, high max        -> rare external stall, not structural
#   p99 = 5-50x p50             -> scheduling: run queue, softirq deferral
#   p99.9 alone explodes        -> something periodic; look for a timer
#   whole distribution shifts   -> you are simply out of CPU

Run that from a sandbox and you get a distribution rather than a vibe. Driving it through the SDK, so the measurement runs where the workload runs:

import os
from pandastack import Sandbox

# PANDASTACK_API_KEY in the environment
sbx = Sandbox.create(template="base", ttl_seconds=600)

sbx.filesystem.write("/tmp/latprobe.py", open("latprobe.py").read())

# Run it twice. The FIRST run of anything on a freshly restored guest is
# measuring a different thing (see below); the second run is measuring
# the interrupt and scheduling path you actually care about.
for label in ("cold", "warm"):
    r = sbx.exec("python3 /tmp/latprobe.py")
    print(f"--- {label} ---")
    print(r.stdout)

# Guest-side counters, before and after, tell you whether interrupts are
# even the story. If irq and softirq deltas are flat while latency is
# spiky, your problem is upstream of the device.
print(sbx.exec("grep -E 'virtio|NET_RX' /proc/interrupts /proc/softirqs").stdout)

sbx.kill()

Those last two files are underrated. /proc/interrupts gives you a per-IRQ, per-vCPU count; /proc/softirqs gives you NET_RX, NET_TX, TIMER, TASKLET and friends. Sample them either side of a slow window. If NET_RX barely moved while your latency doubled, the interrupt path is innocent and you should go look at the run queue instead.

The impostor: first-touch page faults on a streamed guest

This is the part I most want people to internalise, because it burned me and it presents identically to interrupt latency: a request stalls for tens or hundreds of milliseconds, in the kernel, with no CPU burned and nothing obviously wrong.

PandaStack restores snapshots with userfaultfd. Rather than downloading a multi-gigabyte memory image before the guest can run, we register the guest's memory with a userfaultfd handler and let the VM resume immediately. When the guest touches a page that has not been materialised yet, the kernel raises a page-fault event, our handler resolves which 4 MiB chunk of the snapshot that page lives in, fetches it from object storage with an HTTP Range GET, and installs it with UFFDIO_COPY. Chunks that the bake process recorded as entirely zero skip the fetch completely and get the shared kernel zero page via UFFDIO_ZEROPAGE instead.

The guest thread that touched the page is blocked for the whole of that. Not descheduled and retried — blocked, inside a fault, until the handler installs the page. If the chunk is already in the per-host shared cache, that is a local disk read. If it is the first restore of that seed generation on that host, it is a network round trip to object storage. We mitigate it hard — a prefetch trace baked at snapshot time replays the hot chunk set in the background so most faults become cache hits, eight fault-servicing goroutines run concurrently, duplicate chunk fetches are single-flighted, and a failed resolve retries with backoff for up to two minutes rather than killing the VM over one blip. It is still, mechanically, a different animal from interrupt latency.

How to tell them apart, because the fix is completely different:

• First-touch faults decay. Each page is paid for once per VM. If the tail shrinks as a sandbox ages and vanishes after a warmup pass, it was memory, not interrupts.
• Interrupt and scheduling tail does not decay. It tracks host load, so it gets worse when the box gets busy and better at 3am, and a warmup pass does nothing for it.
• The counters disagree. Our agent exports pandastack_uffd_page_faults_total and pandastack_uffd_chunk_fetches_total alongside pandastack_sandbox_boot_duration_seconds; a fault storm shows up there and nowhere in /proc/softirqs.
• The stall shape differs. A fault stall is one long block on a single instruction. A scheduling stall is a runnable thread that is not running, which perf and /proc/pressure/cpu can see and a fault stall cannot.
• Only one of them is fixable by you. Warm your working set before you measure and the fault tail goes away. No amount of application tuning shortens a host run queue.

The practical rule that falls out: always run a warmup pass and discard it. That is what the WARMUP constant in the harness above is for. Benchmarks that fail to do this are measuring their platform's memory restore strategy while believing they are measuring their own code, and they publish the resulting number as though it meant something.

What actually helps, ranked by how much it helps

Keep the run queue short

This dwarfs everything else and it is unglamorous. Every scheduling queue in the chain gets longer as a host approaches saturation, and they do not get longer linearly — queueing theory is merciless about the last 20% of utilisation. Our scheduler scores agents on free capacity, roughly 0.6 times free CPU plus 0.3 times free memory in gigabytes, precisely so that new work spreads toward the emptier hosts rather than piling onto the one that answered first. If you operate your own fleet, the most valuable tail-latency work available to you is having one more host than you strictly need.

Weight, so contention is fair rather than random

cgroup v2's cpu.weight does not make anything faster; it makes contention predictable. Without it, a guest's share of the CPU under load is decided by how many runnable threads it happens to have, which rewards the badly behaved. With weight proportional to entitlement, a sandbox that is paying for more gets more when it matters and everyone bursts freely when the host is quiet. We reconcile these weights on a 15-second loop across every live Firecracker process rather than wiring them into each boot path, because create, fork, wake and restore all end up in the same place and one loop catches all of them, including after an agent restart.

Pinning, with an honest caveat

Pinning vCPU threads to fixed cores kills a specific jitter source: the host scheduler bouncing a vCPU thread between cores, invalidating L1/L2 cache and TLB entries each time it lands somewhere new. We have this implemented — a core pool from an env var, round-robin assignment per sandbox, affinity applied to the fc_vcpu threads specifically — and it is off in production. Here is the honest reason: pinning without isolcpus and nohz_full only stops the bouncing, it does not stop other work being scheduled onto those same cores, so you get part of the benefit and lose the flexibility of letting a burst spread wide. For a latency-critical single-tenant deployment I would turn it on with proper core isolation. For a dense multi-tenant fleet where bursting is the product, the tradeoff genuinely runs the other way.

Do less crossing

Every guest-host boundary crossing is a chance to queue. Batching at the application level is therefore worth more inside a microVM than on bare metal: one 64 KiB write instead of sixty-four 1 KiB writes is not just fewer syscalls, it is fewer notifications, fewer interrupts, fewer softirq passes, fewer wakeups. The same logic argues for connection reuse over per-request connections, and for vsock over TCP when you are talking to the host and do not need the network stack in between.

Watch the right host signals

# On the host: is the run queue the story?
# /proc/pressure/cpu "some avg10" is the fraction of time at least one task
# was runnable but waiting for CPU. This is the number that correlates with
# guest-visible interrupt and wakeup latency.
cat /proc/pressure/cpu

# Per-VM CPU actually burned, from the cgroup the agent puts each
# firecracker process into. usage_usec is exact, unlike sampling ps.
for d in /sys/fs/cgroup/**/vm-*/; do
  printf '%s ' "$(basename "$d")"
  awk '/usage_usec/ {print $2}' "$d/cpu.stat"
done

# Agent-side view: boot duration histogram plus the UFFD fault counters.
# If the fault counters are climbing during your slow window, you are
# looking at memory streaming, not interrupts.
curl -s http://localhost:9100/metrics | grep -E 'pandastack_(sandbox_boot|uffd)'

# Inside a guest: which IRQ lines and which softirqs are actually busy.
# Sample twice around a slow window and diff, absolute counts are useless.
grep -E 'virtio' /proc/interrupts
grep -E 'NET_RX|NET_TX|TIMER|SCHED' /proc/softirqs

Things that feel like they should help and don't

Raising the vCPU count. If the host is contended, more vCPU threads per guest means more threads competing, not more CPU. Our templates all bake 8 vCPUs and that is a ceiling for bursting, not a promise of eight cores; adding vCPUs to a guest that is waiting on a run queue makes the run queue longer.

Hunting for interrupt tuning inside the guest. You do not own the host kernel on a managed platform, and the guest-side knobs that exist — NAPI budget, RPS, IRQ affinity within the guest — are operating on a virtual topology whose mapping to physical cores is not yours to control. I have watched people spend a week on guest IRQ affinity for a problem that was one host's run queue.

Averaging more. Adding samples to a mean does not reveal a bimodal distribution, it hides it better. If you take one thing from this post, take histograms.

The summary

A virtio interrupt in a Firecracker guest passes through a host device thread, an eventfd, KVM's injection machinery, possibly an IPI to force a VM exit, the guest's ISR, the guest's softirq layer, and finally the guest scheduler — and behind all of that, a vCPU that is a host thread waiting its turn like any other. The p50 measures the case where every one of those queues was empty. The p99 measures what happens when one of them wasn't.

Most of what you can do about it is not interrupt tuning. It is fleet sizing so the run queue stays short, proportional weights so contention is fair instead of arbitrary, batching so you cross the boundary less often, and measuring the distribution instead of its mean so you can tell which mode you are looking at. And before you believe any of your numbers, warm the guest — because on a snapshot-restored, memory-streamed microVM the first touch of a page is a network round trip wearing an interrupt's clothes, and it will happily corrupt an entire benchmark while you go looking for an IRQ to blame.