Firecracker vhost-user-block Devices Explained

A Firecracker guest reads a disk block, and normally the VMM itself services that read: Firecracker's in-process virtio-block device parses the request, turns it into file I/O against a host file, and fills in the answer. But Firecracker also supports a different arrangement, vhost-user-block, where the VMM does almost none of that. Instead it hands the storage datapath to a separate userspace process — the vhost-user backend — over a Unix socket, and that process owns the storage. Firecracker still emulates the control plane (the guest sees a normal virtio-block device), but the actual reads and writes flow between the guest and the backend, not through the VMM. This post explains the virtio split that makes that possible, what the vhost-user protocol actually shares across the socket, why you'd deliberately move the datapath out of the VMM, the trade-offs you take on when you do, and how it compares to the built-in virtio-block + io_uring path. And where it points for a platform like PandaStack.

The virtio split: device and driver

Start with what a virtio-block device already is, because vhost-user is a variation on it rather than a replacement. A virtio device has two halves that cooperate through shared memory: a driver inside the guest and a device on the host side. The guest driver builds requests — "read sector N into this buffer," "write these bytes to sector M" — as descriptor chains and posts them into a shared-memory ring called a virtqueue. The host-side device drains that ring, does the work, and posts completions back. (We walk the full virtqueue mechanics in /blog/firecracker-virtio-devices.) The key structural fact is that the interface between the two halves is memory — the descriptor table and the available/used rings — plus a lightweight notification each way. Nobody said the host-side device has to live inside the VMM.

That's the opening vhost-user exploits. In the built-in case, the host-side device is Firecracker's own code, running in the VMM process, reaching into guest memory to follow descriptors and issuing file I/O itself. But the virtqueue is just a region of memory and a notification protocol. If some other process could see that same guest memory, read the same rings, and get poked on the same notifications, it could serve the guest's block requests just as well as Firecracker could — and Firecracker would be free of the datapath entirely. Making that possible is exactly what the vhost-user protocol is for.

What vhost-user is: the vhost protocol over a socket

vhost-user is a protocol for handing the datapath of a virtio device to an external userspace process. It descends from the older in-kernel vhost mechanism (which offloaded virtio datapaths into a kernel thread); vhost-user takes the same idea and points it at a userspace backend instead, using a Unix domain socket as the control channel between the VMM and that backend. Over that socket, the two sides negotiate features and, crucially, the VMM hands the backend everything it needs to talk to the guest directly.

• Guest memory — the VMM shares the guest's memory regions with the backend as file descriptors passed over the socket (SCM_RIGHTS). The backend maps those regions into its own address space, so when it follows a descriptor that points at a guest buffer, it's reading or writing the real guest page — no copy through the VMM.
• Virtqueues — the VMM tells the backend where each virtqueue lives (the descriptor table, available ring, and used ring addresses within the shared memory) and how big it is. From then on the backend consumes the available ring and produces into the used ring itself.
• Notifications (kick/call) — two eventfds per queue. The 'kick' fd is signalled when the guest posts new work (the guest's queue-notify is routed to it); the 'call' fd is what the backend signals to raise the guest's completion interrupt. This is how the backend gets woken and how it wakes the guest, without the VMM sitting in the datapath.
• Feature negotiation — the socket carries a handshake so the VMM and backend agree on the virtio and vhost-user feature bits they both support before any I/O flows.

Once that handoff is done, the shape of a block request changes. The guest driver still posts a descriptor chain to the virtqueue and kicks — but the kick lands on the backend's eventfd, not inside Firecracker. The backend reads the available ring, follows the descriptor into shared guest memory, does whatever it means by "storage" (a local file, a network call, a demand-paged fetch — its choice), writes results straight into the guest buffer, publishes the completion into the used ring, and signals the call fd to interrupt the guest. Firecracker set the whole thing up and then stepped out of the read/write path. It still owns the vCPUs, the memory, the other devices, and the lifecycle — but the disk bytes never pass through it.

Why you'd want the datapath outside the VMM

Moving the datapath out looks like more moving parts, so it's worth being concrete about what it buys. The core win is decoupling the storage implementation from the VMM. Firecracker's built-in virtio-block backend knows how to do one thing: I/O against a local host file. That's deliberate — it keeps the VMM small — but it means every storage capability you want has to be a local file or nothing. A vhost-user backend is a separate program you write (or adopt), so it can implement storage however it likes: a networked block store, a distributed volume, a thin-provisioned or deduplicated pool, a format with its own encryption or compression, or a backend that fetches blocks on demand from object storage. The guest just sees virtio-block; the interesting behavior lives in a process you control, evolving on its own release cadence rather than requiring a change to the VMM.

The second win is keeping the VMM minimal — which, for Firecracker specifically, is a security argument, not just an aesthetic one. Firecracker wraps the VMM process in a tight seccomp syscall allowlist (we cover that in /blog/seccomp-explained), and every capability the built-in datapath needs — the syscalls to open, read, write, and flush storage — has to be permitted in that jail. If your storage backend needs to open network sockets, talk to a distributed system, or do exotic I/O, folding that into the VMM would mean widening its syscall surface for every guest on the host. Push the datapath into a separate vhost-user process and that process carries its own capabilities and its own sandbox; the VMM's allowlist stays narrow. You've moved the complicated, capability-hungry code out of the most security-sensitive process.

There's an operational win too: a single backend process can serve storage for many VMs, so pooling, shared caches, and centralized policy (rate limiting, tiering, telemetry) live in one place instead of being duplicated per-VMM. And because the backend is decoupled, it can be developed, tested, and hardened independently of Firecracker's release cycle.

The trade-offs: another process to trust

None of this is free, and the honest framing is that vhost-user trades the built-in device's simplicity for flexibility. First, it's an extra process to run, supervise, and account for: something has to start the backend, hand it the socket, restart it if it dies, and reason about what happens to in-flight guest I/O if it does. A crash in the built-in path is a crash of the VMM; a crash in a vhost-user backend is a storage stall for every guest it was serving, and recovering cleanly is real engineering.

Second, and more pointed: the backend has the guest's memory mapped into it. That's what makes the datapath fast — no copies through the VMM — but it means the backend is inside the trust boundary. A compromised or buggy vhost-user backend can read and write guest memory directly, so it is every bit as security-critical as the VMM, and it must be secured with the same seriousness: its own seccomp jail, its own least-privilege posture, its own audit. "We moved the storage code out of the VMM" only improves your security if the process you moved it into is at least as locked down as the one it left. Otherwise you've just relocated the attack surface, not shrunk it.

Configuring a vhost-user drive

The difference shows up cleanly in the drive config you PUT to Firecracker's API before boot. A normal drive names a host file and (optionally) an io_engine; a vhost-user drive names a Unix socket instead — the endpoint where the backend is already listening. That single distinction is the whole switch: file-backed and in-process, versus socket-backed and delegated.

// A normal file-backed virtio-block drive: Firecracker emulates the
// device and services the I/O in-process against a host file.
// PUT /drives/rootfs
{
  "drive_id": "rootfs",
  "path_on_host": "/var/lib/pandastack/vms/demo/rootfs.ext4",
  "is_root_device": true,
  "is_read_only": false,
  "io_engine": "Async"
}

// A vhost-user-block drive: no host file path. Instead a Unix socket
// where an external backend is listening. Firecracker keeps the control
// plane and delegates the datapath to that backend over the socket.
// PUT /drives/data
{
  "drive_id": "data",
  "socket": "/var/run/pandastack/vhost-user-data.sock",
  "is_root_device": false,
  "is_read_only": false
}

Note what's absent from the vhost-user drive: there's no path_on_host and no io_engine, because Firecracker isn't the one doing the I/O — the backend behind the socket is, and it decides how storage works. The socket has to be ready before you configure the drive: something must already be listening there, because Firecracker connects to it during setup to run the vhost-user handshake and receive the queue and memory information. Sketching that backend side, in the shape of a launch script:

# The vhost-user backend listens on the Unix socket FIRST, then
# Firecracker connects to it when the vhost-user drive is configured.
# (Illustrative: a real backend implements the vhost-user protocol
#  and whatever storage it fronts — local file, network store, etc.)

SOCK=/var/run/pandastack/vhost-user-data.sock
rm -f "$SOCK"

# Start the backend, bound to the socket, backing some storage.
my-vhost-user-blk-backend \
    --socket-path "$SOCK" \
    --backend    demand-paged-from-object-store \
    &

# ...then start Firecracker and PUT the vhost-user drive pointing at $SOCK.
# Firecracker connects, negotiates features, and shares guest memory +
# virtqueue + eventfds with the backend over that socket.

vhost-user-block vs built-in virtio-block

Both present the guest with an identical virtio-block device; the guest driver can't tell them apart. The difference is entirely on the host side — who owns the datapath, and where the storage logic lives. Side by side:

• Where the datapath runs — built-in virtio-block: inside the Firecracker VMM process; the VMM parses descriptors and does the I/O itself. vhost-user-block: inside a separate backend process; the VMM sets it up and steps out of the read/write path.
• What backs the storage — built-in: a local host file, serviced by the Sync or Async (io_uring) FileEngine. vhost-user: whatever the backend implements — local file, networked/distributed store, demand-paged blocks, custom formats.
• Config shape — built-in: path_on_host plus an optional io_engine (Sync/Async). vhost-user: a socket path to an already-listening backend; no path_on_host, no io_engine (the backend chooses how I/O is done).
• Guest memory access — built-in: the VMM already has guest memory and reads buffers directly. vhost-user: the VMM shares guest memory regions with the backend over the socket, so the backend reads/writes guest buffers directly too — which puts it inside the trust boundary.
• Operational surface — built-in: one process (the VMM); a datapath crash is a VMM crash. vhost-user: an extra process to launch, supervise, and secure; a backend crash stalls storage for the guests it served.
• When to reach for it — built-in: the default; local disk, simplest and smallest, ideal when a host file is all you need (and it's what the CoW rootfs uses). vhost-user: when you need a storage backend the built-in device can't provide, or want storage logic decoupled from the VMM and shared across VMs.

It's worth being clear that vhost-user-block and the io_engine choice (Sync vs async io_uring) live at different layers and don't compete. io_engine is a knob on the built-in datapath: it selects how efficiently Firecracker's own in-process device services file I/O (we cover it in /blog/firecracker-io-uring-block-io). vhost-user-block replaces the built-in datapath wholesale — so it has no io_engine field, because Firecracker isn't servicing the I/O at all. One tunes the in-house backend; the other swaps in an external one.

Where this points for PandaStack

Tie it back to the substrate this runs on. PandaStack boots every sandbox on a copy-on-write rootfs — a reflink clone of a template's rootfs.ext4 (XFS reflink, O(metadata) to create), serviced by the built-in virtio-block device. That rootfs must be a local block device, on purpose: copy-on-write via reflink (and dm-snapshot) needs a real local block device to clone, so the fast path — a create landing at 179ms p50, ~203ms p99, of which the snapshot restore itself is ~49ms — depends on the disk being local. The built-in file-backed device is exactly right for that, and it's what ships today. Where PandaStack streams from object storage today is memory, not disk: the UFFD path demand-pages a snapshot's guest RAM from GCS. The rootfs stays a local file.

But vhost-user-block is precisely the mechanism you'd reach for to change that picture as a design direction. A vhost-user backend that demand-pages rootfs blocks from an object store — serving a local cache on hit, fetching a range from GCS on miss, and materializing a copy-on-write overlay locally — is the kind of thing that could let a cross-host restore (today 1.2–3.5s when the artifacts aren't already resident) start before the whole disk is present, paying for blocks as the guest touches them. That's a design shape the vhost-user architecture makes clean: the streaming, caching, and CoW logic all live in a backend process, and Firecracker's minimal VMM stays exactly as small and seccomp-tight as it is now. To be explicit, that is a direction, not a shipped feature — PandaStack does not stream the rootfs from object storage in production today; the live streaming path is for memory. But the map from "vhost-user delegates the block datapath to a process you write" to "a demand-paged rootfs backend" is short, and it's why this device type is worth understanding.

None of it changes the isolation story: the guest still sees a thin, audited virtio-block device, and whether the datapath runs in the VMM or a delegated backend, the descriptor-parsing code has to validate every guest-supplied address and length. vhost-user moves that responsibility; it doesn't dissolve it — which is why a backend has to be secured as carefully as the VMM. PandaStack's core is open source under Apache-2.0, so you can stand up the per-host agent on your own KVM hosts, wire a drive to a Unix socket, and watch an external backend serve a guest's blocks while Firecracker holds only the control plane. For the request path and the Sync/Async engines behind the built-in device, start at /blog/firecracker-io-uring-block-io; for the device model as a whole, /blog/firecracker-virtio-devices; for the CoW rootfs, /blog/copy-on-write-rootfs.