Firecracker's io_uring Block Backend Explained

A Firecracker guest thinks it has a disk. It doesn't — it has a virtio-block device, and behind that device is an ordinary file on the host. Every block the guest reads or writes has to become real I/O against that file, and how the VMM does that translation turns out to matter a lot for the workloads PandaStack runs: reading a rootfs on boot, paging in copy-on-write blocks as a fork diverges, and running database VMs that push a steady stream of small writes. This post walks the virtio-block request path from guest to host, then explains the two engines Firecracker can use to service it — the original synchronous backend that did one blocking read or write per request, and the async io_uring engine that batches submissions and completions through shared kernel rings. The difference is fewer syscalls and more overlap, which is exactly what you want when a lot of tiny disk operations are on the critical path.

The guest has a file, not a disk

When a Firecracker microVM boots, it sees a block device and mounts it as its root filesystem. On PandaStack that device is backed by a rootfs.ext4 image — Ubuntu 24.04 plus whatever the template baked in — living as a regular file on the host. The guest's own filesystem (ext4, formatted inside the image) issues block reads and writes against what it believes is a physical disk. There is no disk. There is a file, and a device model whose entire job is to make that file behave like one.

That device is virtio-block: a paravirtualized device the guest knows is virtual, cooperating with the host through shared-memory ring buffers rather than emulated hardware registers. (We cover the virtio model in general in /blog/firecracker-virtio-devices.) The important consequence here is that virtio-block is a thin translator. It doesn't implement storage; it shuttles bytes between a virtqueue and a host file. A guest read descriptor becomes a read from the backing file at some offset; a guest write descriptor becomes a write. The interesting question is how that host-side read or write is actually performed — and that's where the FileEngine comes in.

How a block request crosses the boundary

Before we get to the engines, trace a single read. The guest's block driver wants to read a 4 KiB block into a page of guest memory. It builds a descriptor chain in the device's virtqueue: a header descriptor saying "this is a read, at sector N," a descriptor pointing at the guest-memory buffer the data should land in, and a status byte descriptor for the result. It publishes the head of that chain into the available ring and kicks a notification register.

1. Guest builds a virtio-block request (type, sector, buffer pointers) as a descriptor chain and posts it to the virtqueue's available ring.
2. The guest writes the queue-notify register, which traps out of the guest into Firecracker.
3. Firecracker's block device thread reads the available ring, follows the descriptor chain, and validates every guest-supplied address and length against guest memory (this is the security-critical parsing — the guest controls those descriptors).
4. The device translates the request into a host file operation at the corresponding byte offset — and hands it to the FileEngine.
5. When the I/O completes, the device writes the result into the guest buffer, marks the status byte, publishes the descriptor index into the used ring, and raises an interrupt so the guest's driver knows the read is done.

Step 4 is the pivot. Once the device has decided "read 4 KiB from the backing file at offset X into this buffer," something has to actually do that read against the host file. That something is the FileEngine, and Firecracker has two implementations of it. Everything else in the path — virtqueue parsing, descriptor validation, filling the used ring — is identical between them. The engines differ only in how they talk to the host kernel.

The Sync engine: one blocking syscall per request

The original FileEngine is synchronous. When the block device needs to service a request, it calls a blocking read or write (a pread/pwrite-style operation at the file offset) directly, on the device's own thread. The thread issues the syscall, the kernel does the I/O, and the syscall returns when the data is in the buffer (or written). Then the device fills the used ring and moves to the next request.

This is simple and correct, and for a long time it was the only option. But it has two structural costs. First, it's one syscall per request: a queue full of pending block operations becomes a queue full of sequential read/write syscalls, each with its own user-to-kernel transition. Second, it's blocking: the device thread can only have one I/O in flight at a time. While the kernel services a read, the thread is parked in the syscall, not draining the rest of the queue. For a workload that submits many small requests — a database VM doing lots of little writes, or a boot that reads thousands of scattered rootfs blocks — that per-request overhead and lack of overlap adds up.

The Async engine: io_uring rings

The async FileEngine is built on io_uring, the Linux asynchronous I/O interface introduced in kernel 5.1. The idea behind io_uring is to replace the syscall-per-operation model with two shared ring buffers that live in memory mapped between the application and the kernel: a submission queue and a completion queue. Instead of calling a syscall for each I/O and waiting, the application writes requests into the rings and the kernel reads them out — batching many operations across a single (or zero) syscall boundary.

• Submission Queue (SQ) — the application (Firecracker) fills submission queue entries (SQEs), each describing one I/O: read or write, which file, what offset, which buffer. Multiple SQEs can be queued before the kernel is told about any of them.
• Completion Queue (CQ) — the kernel posts completion queue entries (CQEs) here as each I/O finishes, carrying the result (bytes transferred, or an error) and a token identifying which request it was. The application reaps completions from this ring without a syscall per result.
• io_uring_enter — a single syscall submits a batch of queued SQEs and can optionally wait for completions. One syscall can push many operations into the kernel, so the syscall-per-request cost of the sync engine collapses to roughly a syscall-per-batch.

Applied to virtio-block, the flow changes shape. When the device drains the available ring, instead of doing a blocking read for each request, it pushes an SQE for each into the submission queue and submits the batch. The kernel services them — potentially many concurrently — and posts CQEs as they complete. Firecracker reaps the completion ring, matches each CQE back to its virtio descriptor via the token it stashed, fills the used ring, and raises the guest interrupt. The device thread is no longer blocked inside a single read: it submits work, does other things, and comes back to collect finished I/O. Many operations are in flight at once, and a whole batch of requests costs on the order of one submit syscall instead of one syscall each.

The wins are the two costs of the sync engine, inverted: fewer syscalls because submissions batch, and real concurrency because the engine doesn't block waiting on each I/O. For disk-I/O-heavy guests that translates into better throughput and lower per-request latency — qualitatively, not by some fixed multiplier, because the actual gain depends entirely on the workload's I/O pattern, the host storage, and the kernel.

Choosing the engine: the drive config

You pick the engine per drive, in the drive configuration you PUT to Firecracker's API before boot. The relevant field is io_engine, which takes "Sync" or "Async". Everything else about the drive — the host file path, whether it's the root device, read-only or not — is unchanged; io_engine is the one knob that selects the FileEngine backing that virtio-block device.

// PUT /drives/rootfs — a virtio-block device backed by a host file,
// serviced by the async io_uring FileEngine.
{
  "drive_id": "rootfs",
  "path_on_host": "/var/lib/pandastack/vms/demo/rootfs.ext4",
  "is_root_device": true,
  "is_read_only": false,
  "io_engine": "Async",
  "rate_limiter": {
    "bandwidth": { "size": 104857600, "refill_time": 1000 },
    "ops":       { "size": 2000,      "refill_time": 1000 }
  }
}

That's the whole choice: swap "Async" for "Sync" and the same drive is serviced by the blocking engine instead. If you omit io_engine, Firecracker uses its default (Sync) — the conservative choice that works on any kernel. The rate_limiter block is discussed below.

It's worth knowing that io_uring's own configuration — how deep the rings are, whether a submission can carry many entries — governs how much batching you actually get. You can see the same knob at the OS level with tools like fio, which express it as an I/O depth: the number of operations kept in flight at once. Higher depth is exactly the thing the async engine exploits and the sync engine can't.

# The same idea the async engine exploits, made explicit with fio:
# keep 64 I/Os in flight at once via io_uring, instead of one at a time.

fio --name=randread --filename=/dev/vdb --rw=randread \
    --ioengine=io_uring --iodepth=64 --bs=4k --runtime=30s

# iodepth=1 (what a blocking, one-at-a-time backend effectively gives you)
# vs iodepth=64 is the crux of sync vs async: depth is throughput headroom
# only if the backend can keep that many requests outstanding.

Sync vs Async at a glance

The two engines service the exact same virtio-block requests; they differ only in how they hit the host kernel. Side by side:

• Per-request syscall — Sync: one blocking read/write syscall per request. Async: submissions batch, so a queue of requests costs roughly one submit syscall, not one each.
• Batching — Sync: none; requests are serviced one at a time on the device thread. Async: many SQEs submitted together via a single io_uring_enter.
• Concurrency — Sync: one I/O in flight per device thread (the thread blocks inside each syscall). Async: many I/Os in flight at once; the kernel services them concurrently and posts completions as they finish.
• Throughput / latency — Sync: fine for light or sequential I/O; a single-thread ceiling under heavy small-I/O load. Async: higher throughput and lower per-request latency for I/O-heavy guests — magnitude depends on workload, storage, and kernel.
• Kernel requirement — Sync: works on any supported kernel. Async: needs a host kernel with io_uring (5.1+; Firecracker documents a minimum for the async engine).
• When to use each — Sync: the safe default, older/constrained hosts, light I/O. Async: disk-I/O-heavy workloads on a modern kernel — database VMs, boot-time rootfs reads, CoW page-ins on a diverging fork.

Graceful fallback to Sync

The async engine is an optimization, not a hard dependency, and the honest position is to treat it that way. io_uring availability is a property of the host kernel, and a fleet of KVM hosts isn't always uniform — an older host, a locked-down kernel, or a build without io_uring support all mean the async path can't be used there. The right posture is to prefer Async where the host supports it and fall back to Sync where it doesn't, so a disk-I/O-heavy guest gets the fast path on capable hosts without breaking on the ones that can't provide it. The guest sees an identical virtio-block device either way — same rootfs, same behavior — just serviced by a different engine underneath. Correctness never depends on which engine ran the read.

Rate limiting the block device

Independent of the engine choice, Firecracker can rate-limit a block device, and in a multi-tenant setting you want to. Without a limit, a single guest hammering its disk can monopolize the host's I/O bandwidth and starve its neighbors — the classic noisy-neighbor problem. The virtio-block device supports a token-bucket rate limiter, configured right there in the drive config (the rate_limiter block in the JSON above).

A token bucket has two knobs, and the block device exposes both: a bandwidth limit (bytes per refill window) and an ops limit (operations per refill window). Each request spends tokens; when the bucket is empty, requests wait until it refills. Bandwidth caps how many bytes per second a guest can move; ops caps how many operations per second regardless of size, which is what protects you from a flood of tiny requests that don't move much data but still generate a lot of I/O. The limiter sits in front of the FileEngine, so it applies whether the drive runs Sync or Async — the engine determines how efficiently permitted I/O is serviced; the rate limiter determines how much I/O is permitted.

Why this matters for sandbox workloads

Bring it back to what actually runs on this device. Three PandaStack workloads lean on block I/O in ways that make the engine choice concrete. First, boot: restoring a snapshot and starting a guest reads rootfs blocks off the host file, and a create lands at 179ms p50 (p99 ~203ms, of which the restore itself is ~49ms) — a budget tight enough that the efficiency of servicing those reads is not academic. Second, copy-on-write page-ins: as a fork diverges from its parent, first writes copy blocks and the rootfs is read and written through this same virtio-block path (the CoW rootfs mechanics are in /blog/copy-on-write-rootfs). Third, database VMs: a managed Postgres microVM is a durable-volume workload doing a steady stream of small reads and writes, precisely the small-I/O pattern where batching and concurrency pay off most.

None of this changes the isolation story — the block device is still a thin, audited translator over a host file, and the security boundary is KVM plus the minimal virtio device model, not the I/O engine. What the async engine changes is efficiency: same disk, same file, same descriptors, serviced with fewer syscalls and more overlap where the host kernel supports io_uring, and serviced correctly by the sync engine where it doesn't. PandaStack's core is open source under Apache-2.0, so you can stand up the per-host agent on your own KVM hosts, set io_engine on a drive, and watch a disk-heavy guest behave differently under Sync and Async. For the broader device model, start at /blog/firecracker-virtio-devices; for the rootfs clone path, /blog/copy-on-write-rootfs.