Firecracker Networking: TAP vs macvtap (and netns)

Firecracker's networking is refreshingly small: the guest gets a virtio-net NIC, and on the host side that NIC is backed by a TAP device — a virtual Ethernet interface the VMM reads and writes frames through. That's the whole contract. Firecracker hands you a TAP and stops. Everything that makes that TAP useful — how it reaches the outside world, whether it's isolated from other guests, how it's torn down — is your job, and the way you do it is the single biggest lever on whether your microVM fleet is actually multi-tenant-safe or just a group chat your tenants didn't ask to join.

There are three common ways to wire that TAP up, and they optimize for different things. A TAP on a shared Linux bridge is the classic, simple approach. A TAP inside a per-VM network namespace with veth and NAT is the isolation-first approach. macvtap is the throughput-and-L2-directness approach. People search 'Firecracker TAP vs macvtap' expecting a performance shootout, but the more useful axis is isolation and teardown — because for a platform running untrusted, multi-tenant code, that's the axis that decides whether one guest can see, reach, or spoof another. I'm Ajay; I build PandaStack, a Firecracker platform, and this post is the tradeoff laid out honestly, including why we land on per-VM namespaces.

Option 1: TAP on a shared Linux bridge

The tutorial approach: create one Linux bridge on the host, give each microVM a TAP, and enslave every TAP to that bridge. The bridge acts as a software switch; NAT or a routed uplink gets the guests to the internet. It's simple, it works on your laptop, and it's exactly what you'll find in most 'Firecracker networking hello world' posts. It's also a shared broadcast domain, and that's the catch.

• It's one L2 segment for everyone. Every guest on the bridge shares a broadcast domain: ARP, DHCP, and broadcast traffic are visible across it, and without extra filtering, guests can attempt to talk to each other directly and to spoof MAC/IP. For trusted workloads that's fine; for untrusted multi-tenant code it's a lateral-movement surface you have to actively fence off.
• Isolation is bolt-on, not structural. Making a shared bridge safe means layering ebtables/iptables rules, per-port filtering, and MAC/IP anti-spoofing on top — a growing pile of rules whose correctness you have to maintain, and whose failure mode is silent (a missing rule doesn't error; it just permits).
• Teardown is fiddly. Removing a guest means detaching its TAP, cleaning its rules, and not disturbing the shared bridge everyone else is still using. Cleanup errors accumulate as leaked interfaces and stale rules on a long-lived host.
• It scales as a shared resource. One bridge with thousands of ports plus a flat iptables ruleset becomes a contention and rule-lookup concern at density — the opposite of what you want when creates are supposed to be O(1).

Option 2: TAP in a per-VM network namespace

The isolation-first approach gives each microVM its own Linux network namespace. Inside that namespace lives the guest's TAP; a veth pair bridges the namespace to the host's root namespace; and NAT plus a small, namespace-local ruleset routes the guest out. This is the model PandaStack uses (we call it NATID). The difference from the shared bridge is categorical: there is no shared L2 segment. Each guest sits in its own network world, sees only its own interfaces, and reaches the outside only through the veth-and-NAT path you defined for it.

• Isolation is the default, not a rule you might forget. A guest in its own netns can't see another guest's ARP, can't join a shared broadcast domain, and can't reach a neighbor at L2 — because there is no neighbor in its namespace. Cross-tenant reachability isn't blocked by a filter; it's absent by construction.
• Rules are O(1) and local. Each namespace has its own tiny ruleset for its own guest, so a lookup doesn't walk a giant shared table, and one guest's rules can't collide with another's. Per-guest egress allowlisting — the thing that turns 'reads its own data' into 'can't exfiltrate it' — is natural here because each guest already has its own network world to fence.
• Teardown is atomic. Destroy the namespace and everything in it — TAP, veth, routes, rules — vanishes together. No leaked interfaces, no stale rules on a shared bridge, no partial cleanup to reconcile later.
• The cost is per-guest setup, so you pre-allocate. Building a namespace cold — ip netns add, link creation, iptables — is around 100ms, which would wreck a fast create path. So PandaStack pre-allocates 16,384 /30 subnets as ready-made namespaces; allocating one to a new sandbox is around 9ms (patch a MAC and go). The isolation is structural and the setup cost is amortized away from the create.

Option 3: macvtap

macvtap is a different animal. Instead of a TAP enslaved to a bridge, macvtap attaches a virtual interface — with its own MAC address — directly onto a lower interface (usually a physical NIC, sometimes another virtual one), bypassing the Linux bridge entirely. The guest's frames go more directly to the wire. Its appeal is performance and L2 simplicity: skipping the bridge removes a hop and some overhead, so throughput can approach line rate, and each guest presents as its own MAC on the network. macvtap runs in several modes — bridge, VEPA, private, and passthru — that govern whether guests on the same lower device can talk to each other and how frames are hairpinned.

• It optimizes throughput and directness, not tenant isolation. macvtap's win is fewer layers between guest and wire; that's a performance story. Its 'private' mode does isolate guests on the same lower device from each other, but the tool is fundamentally about L2 placement and speed, not about the deny-by-default, per-guest egress control an untrusted-multi-tenant platform needs.
• The host-isolation gotcha. In the common modes, a macvtap guest typically can't talk to the host over the same lower interface — the host and its macvtap children are isolated from each other by design. That surprises people who expected host<->guest traffic to just work, and it complicates any control-plane-to-guest path that assumed the host could reach the guest directly.
• MAC explosion and promiscuity. Every macvtap guest adds a MAC to the physical NIC; at density that can hit switch MAC-table limits, and some setups need the NIC in promiscuous mode or need switch-side support (VEPA needs a hairpin-capable switch). It couples your VM density to your physical network's tolerance.
• Throughput is real but verify it. Skipping the bridge genuinely reduces overhead, but exact numbers depend entirely on your NIC, offloads, and traffic pattern — treat 'near line rate' as a claim to measure on your own hardware, not a guarantee.

The honest summary: macvtap is a great answer to 'my trusted guests need maximum throughput and clean L2 placement.' It's a weaker answer to 'I run untrusted tenant code and need each guest fenced into its own network world with deny-by-default egress,' because its isolation modes are a side effect of its L2 design rather than the primary goal, and the host-isolation and MAC-scaling behaviors fight the control-plane needs of a dense sandbox platform.

The three side by side

• Isolation — TAP+bridge: shared L2, isolation is bolt-on filtering. TAP+netns: per-guest network world, isolation is structural. macvtap: L2 modes that can isolate, but it's a throughput tool first.
• Throughput — TAP+bridge: an extra bridge hop. TAP+netns: veth + NAT hop, fine for typical sandbox traffic. macvtap: most direct to the wire, best raw throughput (verify on your NIC).
• Teardown — TAP+bridge: detach TAP + clean shared rules, error-prone. TAP+netns: destroy the namespace, everything goes atomically. macvtap: remove the macvtap link; simpler than a bridge but no private-world to collapse.
• Multi-tenant fit — TAP+bridge: needs hardening to be safe. TAP+netns: the natural fit for untrusted, per-tenant egress control. macvtap: fits trusted high-throughput guests better than untrusted multi-tenant ones.
• Setup cost — TAP+bridge: cheap per guest, one shared bridge. TAP+netns: ~100ms cold, so pre-allocate (~9ms allocated). macvtap: cheap per link, but couples to physical NIC/switch limits.

Seeing the host-side setup

The mechanical difference is visible in the host commands each model needs. A TAP in its own namespace with a veth uplink (the isolation-first shape, simplified and illustrative):

# Per-VM network namespace: the guest's TAP + veth live in a PRIVATE
# network world. No shared L2 segment, atomic teardown.
ip netns add ns-vm1
ip netns exec ns-vm1 ip tuntap add tap0 mode tap   # Firecracker's guest NIC
ip link add vh-vm1 type veth peer name vg-vm1        # veth: host <-> ns
ip link set vg-vm1 netns ns-vm1                       # guest side into the ns
# ...assign /30 addresses, enable NAT + a namespace-local egress ruleset...
# Teardown is one line and takes EVERYTHING with it:
#   ip netns del ns-vm1   # tap0, veth, routes, rules -- all gone atomically.

macvtap, by contrast, attaches straight onto a lower interface with its own MAC, no bridge and no namespace required (also illustrative):

# macvtap: a virtual NIC with its own MAC directly on a physical interface.
# Fewer layers, more direct to the wire -- but note the host-isolation and
# MAC-scaling caveats above.
ip link add link eth0 name mvt0 type macvtap mode bridge
ip link set mvt0 up
# The macvtap creates a /dev/tapN char device Firecracker's TAP-style backend
# reads/writes. `mode private` isolates guests on eth0 from each other;
# `mode bridge` lets them talk. NONE of these modes let the guest reach the
# host over eth0 by default.

Putting it together

Firecracker gives you a TAP and gets out of the way, which means the interesting decision is entirely yours. A shared bridge is the easy start and a shared broadcast domain you'll spend rules trying to fence. macvtap is the throughput-and-directness choice, excellent for trusted guests you control and awkward for the host-isolation and MAC-scaling realities of a dense multi-tenant platform. A per-VM network namespace is the isolation-first choice: no shared L2, O(1) local rules, per-guest egress allowlisting as the default, and atomic teardown — at the cost of per-guest setup you amortize by pre-allocating (16,384 /30 subnets ready to hand out at ~9ms instead of ~100ms cold). If you're running your own trusted traffic and want speed, reach for macvtap. If you're running code you didn't write for tenants who must not see each other, put every guest in its own network world and never think about a shared bridge again.