Firecracker Doesn't Use vhost-net (On Purpose)

There's a well-known trick for making virtual machine networking fast: vhost-net. It moves the virtio-net device out of the userspace hypervisor and into the host kernel, so guest packets never have to round-trip through the VMM process at all. QEMU uses it by default in most production setups, and it's a big part of how a QEMU/KVM guest reaches near line rate. Firecracker, running the same virtio-net protocol on the same KVM, pointedly does not use it. The network device stays in Firecracker's own userspace process, every packet passing through code the hypervisor controls. That's not a missing feature or a roadmap gap — it's a deliberate trade, and it's the same trade that runs through Firecracker's entire design: give up some throughput to keep the trusted boundary small and the jailed process contained. This post walks the actual packet path, explains what vhost-net does and why Firecracker declines it, and shows where that leaves you on a multi-tenant platform like PandaStack.

The data path: guest to virtqueue to userspace to TAP

Start with what actually happens when a Firecracker guest sends a packet. Inside the VM, the guest kernel has a virtio-net interface — an ordinary-looking eth0 with a MAC and a link. Its virtio driver doesn't poke hardware registers the way a driver for a real NIC would; instead it cooperates with the hypervisor through shared-memory ring buffers called virtqueues. To transmit, the guest writes the packet into a buffer, publishes a descriptor pointing at that buffer into the transmit virtqueue's available ring, and then does a single notification write that traps out of the guest into the host.

That trap lands in Firecracker. A device thread inside the Firecracker process wakes up, reads the available ring, follows the descriptor chain into guest memory (bounds-checking every address and length against the guest's memory region, because the descriptors are guest-controlled and therefore hostile), copies the frame out, and writes it to the host-side TAP device. The TAP is a virtual Layer 2 interface backed by userspace — here, Firecracker — so once the frame is on the TAP it's ordinary Linux traffic that the host stack can route, NAT, and filter. Receive is the mirror image: a frame arrives on the TAP, Firecracker's device thread reads it, copies it into a guest buffer described by the receive virtqueue, posts a completion to the used ring, and raises an interrupt into the guest.

The load-bearing detail is that middle step. Every single packet, in both directions, passes through Firecracker's userspace device model — it gets read, validated, and copied by code running in the hypervisor process. The guest kernel's virtio driver talks to a device that lives in userspace, and userspace is where the frame is bridged onto the host's TAP. Hold onto that, because vhost-net is precisely the mechanism for cutting that step out.

What vhost-net does (and why QEMU wants it)

vhost-net is a host-kernel module that implements the virtio-net device's data path in the kernel rather than in the VMM. The setup handshake still happens in userspace — the hypervisor negotiates features and hands the kernel the virtqueue memory locations and a pair of eventfds — but after that, the packet-moving work is done by a kernel thread. The guest's notification, instead of trapping all the way out to the userspace VMM, is routed (via KVM's ioeventfd) straight to the kernel's vhost worker, which reads the virtqueue, moves the frame to or from the TAP, and signals completions back to the guest through an irqfd. The userspace hypervisor is out of the per-packet loop entirely.

The payoff is throughput and latency. Cutting userspace out of the data path removes context switches between the VMM and the kernel on every batch of packets and lets the kernel move frames with fewer copies and tighter integration with the host network stack. For a general-purpose hypervisor whose job is to run any workload, including bandwidth-hungry ones, that's a clear win — which is why QEMU commonly pairs virtio-net with vhost-net in production and treats it as the default fast path. (Exact throughput numbers depend heavily on hardware, kernel version, and configuration; verify against current QEMU and kernel documentation rather than trusting any single figure.)

So vhost-net is faster, it's mature, it's widely deployed, and it speaks the identical virtio-net protocol Firecracker already implements. On paper, adopting it looks like free performance. Firecracker's decision not to is the interesting part.

Why Firecracker keeps virtio-net in userspace

The reason comes straight out of Firecracker's threat model. Firecracker exists to run untrusted, multi-tenant code with the smallest possible trusted boundary, and it enforces that boundary two ways: a minimal userspace device model, and a hard shell around the VMM process — the jailer plus a seccomp filter that restricts the process to a tight allowlist of host syscalls, run in its own namespaces with dropped privileges. The whole security argument is that even if a guest finds a bug in a device emulator, it's trapped inside a heavily constrained process that can barely touch the host.

vhost-net breaks that containment story. It moves the network data path from Firecracker's jailed userspace into a kernel component that runs outside the seccomp/jailer boundary. Now guest-influenced packet processing happens in the host kernel, in code the jailer can't constrain — and the kernel is the crown jewel, the one thing a VM escape most wants to reach. Enabling vhost-net would mean a bug in that kernel path is a bug in the most privileged possible place, not inside a locked-down userspace process. Firecracker's designers chose to keep every packet inside the constrained process rather than hand part of the job to an unconstrained kernel thread, even though it costs throughput.

There's a simplicity dividend too. Keeping virtio-net fully in userspace means the device is a small, self-contained, auditable piece of Rust that reads a virtqueue and writes a TAP — the same shape as Firecracker's block and vsock devices. No feature negotiation with a kernel module, no eventfd plumbing across the userspace/kernel seam, no dependency on the host kernel shipping and correctly configuring vhost. Fewer moving parts on the boundary is itself a security property. The cost is real — userspace virtio-net can't match kernel-offloaded throughput — but for serverless functions and sandboxed agent workloads, where per-VM bandwidth is modest and isolation is everything, it's the right side of the trade.

Userspace virtio-net vs. vhost-net offload

The two approaches run the same virtio-net protocol; they differ only in where the per-packet work happens and, therefore, in what's inside the trusted boundary.

• Userspace virtio-net (Firecracker) — every packet is read, validated, and copied by a device thread inside the VMM process. That process is jailed: seccomp-filtered syscalls, dropped privileges, its own namespaces. Smaller trusted boundary, fully auditable device code, no kernel-side dependency — at a throughput cost versus kernel offload.
• vhost-net kernel offload (common in QEMU) — the virtio-net data path runs in a host-kernel worker thread; the guest's notifications route straight to the kernel via ioeventfd/irqfd, and the userspace hypervisor is out of the per-packet loop. Near line-rate throughput and lower latency — but guest-influenced packet handling now runs in the host kernel, outside any jailer/seccomp constraint. (Verify performance specifics against current QEMU and kernel docs.)

Read that as a single knob: how much of the network stack are you willing to run outside the constrained process in exchange for speed? QEMU, being general-purpose, defaults toward speed. Firecracker, being purpose-built for untrusted isolation, defaults toward containment and keeps the whole data path in the jail.

The rate limiter: token buckets on the device

Keeping virtio-net in userspace has a side benefit that matters a lot for multi-tenant platforms: Firecracker can meter the traffic itself, right at the device, because it's already touching every packet. Firecracker's network interface supports a built-in rate limiter — token buckets on bandwidth and on operations, applied independently to the RX and TX paths. A token bucket has a size (the burst it will tolerate) and a refill rate (the sustained throughput it allows); a packet is only forwarded when the bucket has tokens, so you get a hard, per-VM cap that a noisy or hostile guest can't exceed.

You configure it on the same network-interface object you use to attach the TAP. The bandwidth bucket limits bytes; the ops bucket limits packet count. Because the device model is in userspace and already inspecting each frame, enforcing these buckets is cheap and local — no separate kernel qdisc to wire up, no policy living outside the VMM.

// PUT /network-interfaces/eth0 to the Firecracker API socket, before InstanceStart.
// virtio-net backed by a host TAP, with per-VM token-bucket rate limits.
{
  "iface_id": "eth0",
  "host_dev_name": "tap0",
  "guest_mac": "06:00:0a:c8:00:02",
  "rx_rate_limiter": {
    "bandwidth": { "size": 1048576, "refill_time": 100 },
    "ops":       { "size": 2000,    "refill_time": 100 }
  },
  "tx_rate_limiter": {
    "bandwidth": { "size": 1048576, "refill_time": 100 }
  }
}
// size = bucket capacity (bytes for bandwidth, count for ops);
// refill_time = milliseconds to refill the bucket. Tune to your policy;
// verify field semantics against the Firecracker API docs.

This is a direct consequence of the userspace design. With vhost-net, the packets you'd want to meter are being moved by a kernel thread; enforcing a per-VM cap means reaching for a separate kernel mechanism. With Firecracker's in-process device, the rate limiter is just a check the device does before forwarding — the same place all the other network policy conceptually wants to live.

Isolation: a TAP per sandbox, in its own netns

The userspace virtio-net device is only half the isolation story; the other half is where its TAP lives on the host. Firecracker connects the guest's NIC to a TAP and stops there — it implements no routing, no NAT, no cross-VM policy. All of that is the host's job, done in the Linux networking stack around the TAP. The dangerous default is to plug every sandbox's TAP into one shared bridge, which puts all your guests on a single Layer 2 segment where one can ARP for and reach another, or scan for your internal services and the cloud metadata endpoint. For untrusted multi-tenant code that's a cross-tenant leak waiting to happen.

The fix is a Linux network namespace per sandbox: each TAP lives inside its own netns — a private copy of the entire networking stack, with its own interfaces, routing table, and iptables rules — connected to the outside world by a veth pair. Guests in different namespaces literally cannot address each other's traffic. The plumbing, conceptually, is a handful of ip commands:

# ILLUSTRATIVE — the concept a platform automates per sandbox, not exact commands.

# A private network namespace for this one sandbox.
ip netns add ns-demo

# The TAP that Firecracker drives as the guest's virtio-net NIC, inside the netns.
ip netns exec ns-demo ip tuntap add tap0 mode tap
ip netns exec ns-demo ip link set tap0 up

# A veth pair as the only door out of the namespace.
ip link add vg-demo type veth peer name vh-demo
ip link set vg-demo netns ns-demo

# Address the /30 and NAT outbound so the private IP can reach the world.
ip netns exec ns-demo ip addr add 10.200.0.2/30 dev tap0
iptables -t nat -A POSTROUTING -s 10.200.0.0/30 -j MASQUERADE
# For untrusted code, prepend default-deny egress rules to this chain.

Building all of that cold — namespace, veth, TAP, iptables — costs on the order of 100ms, which is ruinous on the latency budget of every create. That's the problem PandaStack's NATID networking solves: each agent pre-allocates a pool of 16,384 /30 subnets out of 10.200.0.0/16, with each slot's namespace, veth pair, TAP, and rules standing and ready in advance. Allocating one for a new sandbox becomes a roughly 9ms patch rather than a 100ms build — a big part of why a PandaStack create lands at 179ms p50 (around 49ms for the snapshot restore itself). A /30 holds exactly two usable addresses, gateway plus one guest, so there's no room in a sandbox's own subnet for another guest to exist. We walk the full path in /blog/firecracker-networking-explained.

Why the userspace choice matters for a sandbox platform

Put it together and Firecracker's refusal to use vhost-net stops looking like a limitation and starts looking like the whole point. When you hand a fresh microVM to untrusted code on every request — an AI agent running model-written commands, a per-user code interpreter, ephemeral CI for arbitrary repos — the thing you're betting on is the boundary between guest and host. Keeping the network data path in a jailed userspace process keeps that boundary small and inspectable: guest-influenced packet handling never runs in the host kernel, the device is a small auditable piece of code, and the rate limiter that caps a noisy tenant lives right there in the same place. The throughput you give up is bandwidth per VM you almost never need for these workloads.

This is the substrate PandaStack runs on. Every sandbox, managed database, and hosted app is its own Firecracker microVM with virtio-net in userspace, its TAP in a per-sandbox network namespace, egress governed by an iptables chain the guest can't touch, and the whole VMM process wrapped in the jailer and a seccomp filter. The minimal, in-process network device is what makes it defensible to run untrusted code on every request; NATID pre-allocation is what makes it cheap. PandaStack's core is open source under Apache-2.0, so you can stand up the agent on your own KVM hosts and confirm exactly where every packet goes. For the broader device-model argument, start at /blog/firecracker-virtio-devices; for the full network path, /blog/firecracker-networking-explained.