Firecracker virtio-net RX/TX Queues Explained

A virtqueue is a ring buffer with commitment issues: one side keeps publishing things it wants done, the other side keeps promising it will get to them, and the whole arrangement only works because both parties agreed in advance never to trust each other's pointers. That is, more or less, the contract that carries every packet in and out of a Firecracker microVM. If you have ever stared at a guest's `eth0`, wondered why throughput plateaus somewhere you didn't expect, and reached instinctively for the multi-queue knob you'd use on a QEMU guest — this post is for you. Firecracker's virtio-net is a single RX/TX queue pair per network device, by design. That's not a bug to work around; it's a deliberate simplicity-and-attack-surface decision. What follows is how the queues actually work, what that one-pair design costs you, the tuning levers that genuinely exist, and how to measure instead of guessing.

The virtqueue: three shared structures and a trust boundary

A virtqueue is not a queue in the tidy computer-science sense. It's three data structures living in guest memory that the guest driver and the host device model both read and write, plus a pair of notification mechanisms to poke each other when something interesting happens. The three structures are the descriptor table, the available ring, and the used ring.

• Descriptor table — a flat array of entries, each describing one buffer: a guest-physical address, a length, some flags, and a `next` index. Flags can mark a descriptor as write-only (the device fills it) or as chained to another descriptor, so a single logical packet can span several buffers. It is a pool, not a queue: entries are addressed by index and reused.
• Available ring — the guest's outbox. When the driver wants the device to do something with a descriptor chain, it writes that chain's head index into the available ring and bumps the ring's index counter. "Available" means available to the device.
• Used ring — the device's outbox. When the device has finished with a chain, it writes the head index plus a length back into the used ring and bumps that counter. The guest reads the used ring to learn what completed and to reclaim buffers.

Everything in that list lives in guest memory, which is the load-bearing fact for security. The descriptor addresses, the lengths, the ring indices — all of it is written by the guest, and in a sandbox platform the guest is hostile by assumption. So Firecracker's device model treats every field as adversarial input: each descriptor address and length is bounds-checked against the VM's memory regions before a single byte is touched, chains are walked with loop and length limits, and a descriptor that claims to describe a buffer outside guest memory is rejected rather than followed. This is exactly the code you want running inside a jailed, seccomp-filtered userspace process rather than in the host kernel — the argument laid out in /blog/firecracker-vhost-net-explained.

The notification half of the contract is the part that costs you. When the guest publishes work, it performs a write that traps out of the VM (a VM exit) so the host can wake up and look at the ring. When the device completes work, it injects an interrupt into the guest. Both directions are expensive relative to a memcpy, which is why the entire virtio design is built around amortizing them: publish a batch, then notify once; and both sides can ask the other to suppress notifications while they're already actively draining a ring. If you internalize one thing about virtio performance, make it this — the enemy is not copying bytes, it's per-packet trips across the boundary.

The TX path: guest to host, step by step

Follow a packet leaving the guest. An application writes to a socket, the guest kernel's network stack builds an skb, and the virtio-net driver takes over. It places the frame — a small virtio-net header followed by the Ethernet frame itself — into buffers, writes descriptors for those buffers into the descriptor table, publishes the head index into the transmit virtqueue's available ring, and (if notifications aren't currently suppressed) performs the notification write that traps to the host.

On the host side, Firecracker's network device thread wakes up. It reads the available ring, walks the descriptor chain, bounds-checks each address and length against guest memory, and copies the frame out into a host buffer. It then writes that frame to the TAP file descriptor. Once the frame is on the TAP, it is ordinary Linux traffic: it enters the host's networking stack in whatever namespace the TAP lives in, subject to routing, iptables, NAT, and any qdisc on the path. Finally the device writes the descriptor head back into the used ring so the guest can reclaim the buffer, and signals the guest if the guest wants signalling.

The important structural detail: that whole sequence — read ring, validate, copy, write TAP, publish used — happens on one path, for one queue. Firecracker will happily drain many descriptors per wakeup, which is where batching earns its keep, but there is no second TX queue being drained concurrently on another core for the same device.

The RX path: host to guest, and why it needs buffers in advance

Receive runs the same machinery inverted, with one twist that trips people up. For RX, the guest publishes *empty* buffers in advance. The driver fills the receive virtqueue's available ring with descriptors marked write-only, pointing at free pages — these are the buffers the device will eventually write incoming frames into. The guest is essentially saying "here is where you may put things" before anything has arrived.

When a frame arrives on the host TAP for that guest, Firecracker's device thread reads it off the TAP fd, takes the next available descriptor chain from the RX ring, bounds-checks it, copies the frame (plus its virtio-net header) into guest memory, publishes the head index and the actual byte count into the used ring, and injects an interrupt. The guest's interrupt handler runs, the driver polls the used ring, hands the frames up to the kernel's network stack via NAPI, and — critically — refills the available ring with fresh empty buffers.

That coupling between guest CPU availability and RX capacity is why network tuning in a microVM fleet is never purely a network problem. The queue is drained by guest code running on a vCPU thread that is competing, on the host, with every other vCPU and every other Firecracker device thread on the box. See /blog/firecracker-vcpu-scheduling-model for how those threads actually get scheduled.

One queue pair per device — and what that actually costs

Modern virtio-net in the general-purpose virtualization world supports multi-queue: N receive queues and N transmit queues on one device, so packet processing can fan out across multiple host threads and multiple guest vCPUs, with the guest steering flows to queues via RSS or XPS. It's the standard way to scale a single VM's networking past what one core can push. Firecracker does not do this. A Firecracker virtio-net device presents one RX queue and one TX queue, full stop.

This is consistent with everything else in Firecracker's device model: the smallest device set that runs a real Linux guest, implemented as a small auditable amount of Rust, running inside a jailed process. Multi-queue would multiply the state machine — per-queue threads, steering logic, feature negotiation, per-queue notification suppression — and every one of those is code parsing guest-controlled ring structures. For a VMM whose entire pitch is a minimal trusted boundary for untrusted multi-tenant code, more concurrent guest-driven state machines is a bad trade. The same instinct produced the userspace-only data path and the deliberately short device list described in /blog/firecracker-virtio-devices.

The practical consequence is a per-VM throughput ceiling shaped by serialization, not by bandwidth. One queue pair means one path draining packets, so a single guest's network throughput is bounded by how fast that path can run: the host device thread doing validation and copying, plus the guest's own driver work on a vCPU. Both of those compete for host CPU. Two things follow. First, adding vCPUs to a guest does not widen the network path the way it widens compute — there's still one queue pair. Second, at high packet rates the guest's own interrupt and softirq handling starts eating the vCPU time the workload wanted, so "the network got slower" and "the app got slower" become the same phenomenon. Do not pretend to know the number where that bites on your hardware; measure it (there's a section below on exactly that).

The tuning levers you actually have

Given no multi-queue knob, here is the honest list of what you can move. Most of these are about reducing per-packet boundary crossings rather than widening the pipe — which is the right instinct anyway, since crossings are the expensive part.

Guest-side ring depth and coalescing

Inside the guest, `ethtool -g` shows the virtio-net ring sizes and `ethtool -c` shows coalescing settings. Ring depth determines how many descriptors can be outstanding — deeper rings absorb bursts and give the guest more slack before RX starvation causes drops, at the cost of memory and potential bufferbloat-style latency. Coalescing, where the driver supports adjusting it, trades interrupt count for latency: fewer, larger batches mean fewer boundary crossings per byte. Note that virtio-net's tunables are narrower than a physical NIC's; several `ethtool` knobs simply won't apply, and what's adjustable depends on your guest kernel and the negotiated feature set. Check what's actually settable before writing it into a template.

Offloads (GSO/TSO/checksum) and MTU

This is usually the biggest lever, and it works precisely because of the boundary-crossing framing. With segmentation offload negotiated, the guest hands the device one large super-frame instead of many MTU-sized packets, and segmentation happens later — so one descriptor chain, one notification, and one trip across the boundary carries far more payload. Turning offloads off multiplies your per-byte crossings, which is why the classic "I disabled TSO to debug something and everything got slow" story is so common. Verify which offload features are negotiated on your setup with `ethtool -k` in the guest rather than assuming; feature availability depends on the virtio feature bits, the guest kernel, and the TAP's own offload configuration on the host.

MTU is the same idea in cruder form: a larger MTU means fewer packets for the same bytes, and packets are the unit of overhead here. The catch is that the MTU only helps if the entire path — TAP, veth, host interfaces, and whatever is upstream — agrees. A mismatch buys you fragmentation, black-holed traffic when path-MTU discovery is blocked, and a debugging afternoon you will not enjoy. Change it deliberately, end to end, or not at all.

The rate limiter is a fairness tool, not a performance tool

Firecracker's per-interface token-bucket rate limiter — separate `rx_rate_limiter` and `tx_rate_limiter`, each with bandwidth and ops buckets — is frequently misfiled as a performance knob. It is the opposite: it can only ever make a given guest slower. Its job is to stop one tenant from consuming the host's network capacity at everyone else's expense. On a dense multi-tenant host that is enormously valuable, because aggregate fairness is what makes the box usable. But if your single VM is slower than you'd like, a rate limiter is not the thing to tune upward for speed; it's the thing to check you haven't set too tight. The full mechanics are in /blog/firecracker-rate-limiter-explained.

# Attach a virtio-net device via the Firecracker API socket, before InstanceStart.
# One RX/TX queue pair (that's the only option), with token-bucket limits.
# size = bucket capacity, refill_time = ms to refill from empty.
# Rate = size / refill_time. This bounds a noisy tenant; it does not
# make this guest faster. Verify field semantics against the API spec
# for your Firecracker version.

curl --unix-socket /run/firecracker.sock -i \
  -X PUT 'http://localhost/network-interfaces/eth0' \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "iface_id": "eth0",
    "host_dev_name": "tap0",
    "guest_mac": "06:00:0a:c8:00:02",
    "rx_rate_limiter": {
      "bandwidth": { "size": 52428800, "refill_time": 1000 },
      "ops":       { "size": 20000,    "refill_time": 1000 }
    },
    "tx_rate_limiter": {
      "bandwidth": { "size": 52428800, "refill_time": 1000 }
    }
  }'

# Need genuine parallelism? Attach a SECOND device — eth1 on its own TAP.
# Two devices = two independent queue pairs. Your application has to
# actually spread traffic across them; nothing does it for you.
curl --unix-socket /run/firecracker.sock -i \
  -X PUT 'http://localhost/network-interfaces/eth1' \
  -H 'Content-Type: application/json' \
  -d '{ "iface_id": "eth1", "host_dev_name": "tap1", "guest_mac": "06:00:0a:c8:00:06" }'

Multiple net devices when you genuinely need parallelism

Since one device is one queue pair, N devices are N queue pairs. Attaching a second (or third) virtio-net interface, each with its own host TAP, is the supported way to get concurrent network paths into a single Firecracker guest. It is not equivalent to multi-queue and you shouldn't pretend it is: there's no RSS steering the same flow set across devices, so your application or your routing has to deliberately spread traffic — different services bound to different interfaces, or connections distributed across source addresses. It also multiplies host-side setup: another TAP, another set of addresses and rules, another thing to clean up when the sandbox dies. Reach for it when a specific workload genuinely needs the parallelism, not as a default.

Host side: TAP placement, qdiscs, RPS, and where the CPU goes

Beyond the TAP file descriptor, everything is ordinary Linux networking, and it's tunable in ordinary Linux ways. The qdisc on the TAP and on the veth or upstream interface shapes and queues traffic; a default queueing discipline that's fine for a workstation may add latency or drop under a many-VM load. RPS/RFS settings influence which CPUs process softirqs for received packets, which matters a great deal when you have hundreds of TAPs on one host and don't want all their softirq work landing on the same few cores as your vCPU threads. Isolating or partitioning cores between vCPU threads, VMM device threads, and host softirq processing is a real lever on a dense box — and one that only shows its value under load, never on an idle test host.

Where the TAP lives matters too. On PandaStack every sandbox's TAP sits inside its own network namespace on a dedicated /30, connected out by a veth pair, with NAT and egress policy applied in that namespace — 16,384 such subnets are pre-allocated per agent so that allocating one is a fast patch rather than a slow build. That's an isolation design first, but it also means each guest's traffic traverses a predictable, per-sandbox path you can shape and observe independently. The layout is walked through in /blog/firecracker-networking-explained.

How to actually measure (instead of guessing)

Every number in a networking argument should come from your hardware, your kernel, and your load. Anyone quoting a microVM throughput figure without naming the host CPU, the kernel version, the MTU, and what else was running on the box is telling you a story, not a measurement. Measuring throughput on an idle host and calling it capacity planning is the most popular form of self-deception in this field: an empty machine will happily show you a number you will never see again in production.

The instrumentation you want is boring and available everywhere. Run `iperf3` inside the guest against a target whose own capacity you've verified, in both directions, and watch not just the throughput but the guest's CPU time while it runs. Check `/proc/interrupts` before and after to see how many interrupts that traffic actually cost — that's your boundary-crossing bill. Watch `softirq` time in the guest (`mpstat`, or the `si` column in `top`) because a guest spending its vCPU on softirq is a guest not spending it on your workload. On the host, `ethtool -S` on the TAP and the interface counters give you drops and errors that tell you whether the ceiling is queueing or something dumber, like an MTU mismatch.

# --- Inside the guest -------------------------------------------------

# What does the virtio-net device actually expose? Check before tuning.
ethtool -i eth0          # driver: virtio_net, and feature info
ethtool -g eth0          # ring sizes (pre-set maximums vs current)
ethtool -c eth0          # coalescing, if the driver exposes it
ethtool -k eth0          # offloads: tso, gso, gro, checksum — what's ON
ip -d link show eth0     # MTU and link details

# Interrupt cost: snapshot, run traffic, snapshot again, diff.
cat /proc/interrupts | grep -i virtio > /tmp/irq.before

# Throughput, both directions. Point at a target you've verified
# can absorb it, or you're measuring the target instead.
iperf3 -c <target> -t 30            # guest -> target (TX path)
iperf3 -c <target> -t 30 -R         # target -> guest (RX path)
iperf3 -c <target> -t 30 -P 4       # parallel streams: still ONE queue pair

cat /proc/interrupts | grep -i virtio > /tmp/irq.after
diff /tmp/irq.before /tmp/irq.after  # interrupts burned per unit of traffic

# Where did the vCPU actually go? High %soft means the network is
# eating the CPU your workload wanted.
mpstat -P ALL 1 10

# --- On the host ------------------------------------------------------

# Per-TAP counters: drops and errors on the sandbox's own interface.
ip netns exec ns-demo ethtool -S tap0
ip netns exec ns-demo ip -s link show tap0

# Queueing discipline in the path — the default may not suit a dense host.
ip netns exec ns-demo tc -s qdisc show dev tap0

# The real fleet question: aggregate, under concurrent load, not one idle VM.
sar -n DEV 1 30

Run that suite before and after any change you make, and run it while the host is doing something else — ideally while a realistic number of neighbor sandboxes are alive and busy. A tuning change that helps an idle host and hurts a loaded one is worse than no change, and you will only ever find that out by loading the host.

The density argument: aggregate throughput is a different target

Here's the framing that reorganizes everything above. If you run one big VM, peak single-VM throughput is your metric and Firecracker's single queue pair is a real constraint you should weigh honestly. If you run a fleet of hundreds of short-lived microVMs — agent sandboxes, per-request code interpreters, ephemeral CI, per-tenant jobs — your metric is aggregate throughput across the host and the CPU cost of achieving it. Those are different optimization targets, and tuning for the first can actively hurt the second.

• Optimization target — Peak single-VM: maximize one guest's Gbps; multi-queue, dedicated cores, deep rings, and generous buffers all pay off. Fleet aggregate: maximize total useful bytes across hundreds of guests per host CPU spent; per-VM ceilings are irrelevant if no single VM is near them.
• Queue design — Peak single-VM: one queue pair is a genuine ceiling, and multi-queue is the standard answer. Fleet aggregate: a queue pair per VM already means hundreds of independent queue pairs on the host, and the parallelism you wanted exists across VMs rather than within one.
• CPU accounting — Peak single-VM: spending host CPU on network processing is fine, it's what the machine is for. Fleet aggregate: every core burned on softirq and device threads is a core not running tenant workloads, so CPU-per-byte matters more than bytes-per-second.
• Buffers and rings — Peak single-VM: deeper is usually better, memory is cheap relative to one VM. Fleet aggregate: ring memory is multiplied by VM count, and density is memory-bound, so oversized rings quietly cost you sandboxes per host.
• Rate limiting — Peak single-VM: pure overhead, remove it. Fleet aggregate: essential, because one unbounded tenant degrades every neighbor and predictable per-tenant slices are the product.
• What to measure — Peak single-VM: iperf3 to a big target on a quiet host. Fleet aggregate: many concurrent guests doing realistic traffic while you watch host softirq time, per-VM drops, and whether create latency holds up under the load.

Read down that column and the single-queue-pair decision stops looking like a compromise. For a sandbox fleet, per-VM network parallelism is something you'd be buying with host CPU and per-VM memory — the two resources that directly determine how many tenants fit on a box. Firecracker spends neither, and hands you hundreds of small independent queue pairs instead of a few wide ones. That's the right shape for this workload, and it's the same reasoning that keeps the data path in a jailed userspace process instead of the kernel.

What this means on PandaStack

PandaStack runs every sandbox, managed database, and hosted app as its own Firecracker microVM: one virtio-net device, one RX/TX queue pair, its TAP in a per-sandbox network namespace on a dedicated /30, egress governed by rules the guest can't reach. Because slot setup is pre-allocated rather than built per create, a sandbox lands at 179ms p50 (p99 around 203ms), with roughly 49ms of that being the snapshot restore itself — versus about 3s for a first cold boot. Forks land in 400–750ms same-host and 1.2–3.5s cross-host. Networking is not on the critical path of any of that, which is precisely the point: the network design was chosen so it never becomes the thing you wait on.

From the SDK you don't configure queues at all — you get a sandbox with a working NIC and sane per-tenant limits. What's worth knowing is where to look when something feels slow: measure inside the guest first, check whether the ceiling is the guest's own CPU rather than the network, and only then go looking at the host.

from pandastack import PandaStack

ps = PandaStack()  # reads PANDASTACK_API_KEY

# Each sandbox is one microVM with one virtio-net queue pair, its TAP in a
# dedicated network namespace. Inspect what the guest actually negotiated
# rather than assuming what your host NIC would do.
sbx = ps.sandboxes.create(template='code-interpreter')

print(sbx.exec('ethtool -i eth0 || true').stdout)   # driver + feature info
print(sbx.exec('ethtool -k eth0 || true').stdout)   # which offloads are ON
print(sbx.exec('ip -d link show eth0').stdout)      # MTU and link state
print(sbx.exec('cat /proc/interrupts | grep -i virtio').stdout)

# Then generate real traffic and re-read the counters. The delta is the
# interrupt bill for that traffic — the number that actually explains
# where your throughput ceiling comes from.
sbx.delete()

PandaStack's core is open source under Apache-2.0, so if you want to see exactly how the TAP is wired, how the namespace is built, and where the rate limiter is attached, you can run the per-host agent on your own KVM hosts and read every line of it. And then — please — measure on your hardware, under load, with neighbors running. Anything else is a story.