IPv6 for Firecracker MicroVM Fleets

Open any guide to microVM networking and you land in the same place: a virtio-net NIC in the guest, a TAP on the host, a private /30, an iptables MASQUERADE rule, and a guest entirely convinced it is on the internet. Good design — it is the one PandaStack ships, and /blog/firecracker-networking-explained walks the packet path. But it is an IPv4 design, and IPv4 designs have two failure modes that only surface on a real fleet: you run out of addresses, and you write a firewall that covers half your traffic.

This is the IPv6 version. Not advocacy — the mechanics of putting a v6 address on a TAP, what snapshot-restore breaks that a cold boot would not, why duplicate address detection is a live hazard for cloned VMs, and the most common IPv6 bug in sandbox platforms, which is that somebody blocked outbound traffic with `iptables` and went home. There is an honest verdict at the end, and for plenty of fleets it is "don't."

Why IPv4 gets tight in a dense fleet

The per-guest /30 is standard for a reason: four addresses, two usable, one for the host-side gateway and one for the guest, and nothing else in the subnet to discover. Isolation by address math on top of isolation by namespace. The cost is that every sandbox burns four addresses whether it needs them or not, and carving a prefix of length P into /30s yields only 2^(30-P) subnets. PandaStack pre-allocates 16,384 /30s per agent out of 10.200.0.0/16 — the full 2^14 the /16 contains, and a hard per-host ceiling. It never binds; memory and CPU run out long before sixteen thousand microVMs do. But notice the cost: an entire /16 of RFC 1918 space, per host, to buy headroom you will not use. Take a /24 from the corporate range instead and you get sixty-four slots, which one enthusiastic CI matrix eats before lunch.

The real pressure is fleet-wide. Because the addresses sit behind NAT, two hosts can reuse the same range — right up until you want VM-to-VM routing and discover every agent claims 10.200.0.0/16. Planning non-overlapping ranges is a coordination problem, a registry, and a conversation with whoever owns the corporate address plan. Then a customer peers a VPC in and turns out to have been on your prefix since 2014. RFC 1918 is not a big space; it is a small space everyone shares. /blog/microvm-ip-address-planning-subnet-exhaustion goes deeper on that arithmetic.

What IPv6 actually changes

The headline is not "more addresses." It is that address supply stops being a design constraint. A single /64 holds 2^64 addresses; a delegated /48 holds 65,536 /64s. Hand every guest a routed /64 and never think about slot arithmetic again, or hand out individual /128s out of one /64 if you prefer counting to one. The allocator stops rationing and becomes a naming scheme.

The more interesting consequence is that you can drop NAT. With a routed prefix the guest's address is the address everyone else sees: no source rewriting, no per-flow conntrack entry, no port-forward table, no hairpinning surprise when a guest resolves your own public endpoint. Debugging gets startlingly better — the address in the host log is the address in the guest log is the address the remote service saw. Anyone who has correlated a NAT'd flow across three logs at 2am will recognise the value.

• Address supply — IPv4 + NAT per guest: a fixed pool you size and reconcile; 16,384 /30s per host, and non-overlapping ranges across hosts are a coordination problem. Routed IPv6 per guest: effectively unbounded, a /64 per guest out of a delegated /48.
• Inbound reachability — IPv4 + NAT per guest: none by default; every inbound path is a DNAT rule you create, track, and remember to delete. Routed IPv6 per guest: the guest is directly addressable, so inbound is a firewall decision rather than bookkeeping — which makes default-deny load-bearing.
• Host state per flow — IPv4 + NAT per guest: a conntrack entry per connection, and stale entries that can deliver a dead VM's replies into its successor's namespace. Routed IPv6 per guest: routing only — the host forwards and forgets, and address reuse carries no translation state.
• Snapshot and clone identity — IPv4 + NAT per guest: the baked private address is identical in every clone and that is fine, because each clone lives alone in its own /30. Routed IPv6 per guest: a globally unique prefix cannot be baked, so you reconfigure at restore or translate at the host. This is the real tension.
• Firewalling — IPv4 + NAT per guest: one rule set, and NAT's implicit inbound protection covers a lot of sloppiness. Routed IPv6 per guest: a second family to cover everywhere, no implicit inbound protection, and ICMPv6 cannot be blanket-dropped the way people casually drop ICMP.

The mechanics on a TAP: link-local, SLAAC, and static

Three things are true of every IPv6-enabled interface, TAPs included. It always has a link-local address in fe80::/10, formed without asking anyone. That link-local is enough to route through — the standard pattern is a default route whose next hop is the gateway's link-local, not a global address. And unless told otherwise, the kernel listens for router advertisements and configures a global address from whatever prefix it hears.

Why static beats SLAAC for snapshot-restored guests

SLAAC is the elegant option and the wrong one here. A router advertisement is not a one-shot fact; it is a lease with timers. The prefix carries preferred and valid lifetimes, and the address moves through preferred, deprecated, and invalid as those run down. That is all state — and PandaStack creates sandboxes by restoring a baked snapshot, not by booting. The guest in that snapshot was frozen mid-lifetime, so restore it a week later and it wakes holding an address whose valid lifetime expired six days ago, with a renewal timer that fired while the VM did not exist. It is the networking flavour of /blog/firecracker-guest-clock-and-time-drift-explained: restored guests inherit frozen time and everything derived from it.

Static addressing has none of that. An address added with `ip -6 addr add` and no lifetime is `forever`, which survives being frozen fine, and a default route via a fixed link-local next hop has no timer at all. So: turn `accept_ra` off on the guest-facing interfaces, pin the gateway's link-local to something deterministic like `fe80::1` so the baked default route is still valid months later, and configure the guest statically. A create lands at 179ms p50 and 203ms p99; there was never room to wait for a router advertisement anyway.

# ---------------------------------------------------------------------------
# HOST, root netns: forward IPv6 and route a /64 at one sandbox's namespace.
# The prefix below is documentation space (2001:db8::/32). Substitute the /48
# or /56 your provider delegates -- prefix delegation differs per cloud, so
# check their docs rather than assuming this shape ports.
# ---------------------------------------------------------------------------
sysctl -qw net.ipv6.conf.all.forwarding=1

# GOTCHA: accept_ra=1 (the default) means "accept RAs *unless* this interface
# is forwarding". The line above just stopped your uplink accepting the RA
# that gave you your default route. If the uplink is RA-configured it needs
# accept_ra=2 to keep working while forwarding is on. Ask me how I know.
sysctl -qw net.ipv6.conf.eth0.accept_ra=2

# The veth pair. Pin BOTH link-locals rather than letting them derive from the
# MAC: deterministic addresses matter when the guest's default route was
# frozen into a snapshot months ago and cannot be renegotiated.
ip link add vh-demo type veth peer name vg-demo
ip link set vg-demo netns ns-demo
ip -6 addr add fe80::a/64 dev vh-demo nodad
ip link set vh-demo up

# One routed /64 per sandbox. No NAT, no port-forward table, no conntrack
# entry per flow -- just a route.
ip -6 route add 2001:db8:cafe:2a::/64 via fe80::b dev vh-demo

# ---------------------------------------------------------------------------
# INSIDE ns-demo: the guest-facing gateway lives on tap0.
# ---------------------------------------------------------------------------
ip netns exec ns-demo sysctl -qw net.ipv6.conf.all.forwarding=1
ip netns exec ns-demo sysctl -qw net.ipv6.conf.vg-demo.accept_ra=0
ip netns exec ns-demo ip -6 addr add fe80::b/64 dev vg-demo nodad
ip netns exec ns-demo ip link set vg-demo up

ip netns exec ns-demo ip tuntap add tap0 mode tap
ip netns exec ns-demo sysctl -qw net.ipv6.conf.tap0.accept_ra=0
ip netns exec ns-demo ip -6 addr add fe80::1/64 dev tap0 nodad   # stable next hop
ip netns exec ns-demo ip -6 addr add 2001:db8:cafe:2a::1/64 dev tap0 nodad
ip netns exec ns-demo ip link set tap0 up
ip netns exec ns-demo ip -6 route add default via fe80::a dev vg-demo

# ---------------------------------------------------------------------------
# GUEST, baked into the template. `nodad` because this address is unique by
# construction -- one guest per namespace -- and DAD costs a second or more on
# a boot path budgeted in tens of milliseconds.
# ---------------------------------------------------------------------------
#   ip -6 addr add 2001:db8:cafe:2a::2/64 dev eth0 nodad
#   ip -6 route add default via fe80::1 dev eth0

# Verify from the host, without entering the guest:
ip netns exec ns-demo ip -6 route show
ip netns exec ns-demo ip -6 addr show dev tap0
ip netns exec ns-demo ip -6 neigh show dev tap0

Duplicate address detection meets the cloned VM

IPv6 does something IPv4 never did: before using an address, a host actively checks whether anyone else has it, by sending a neighbor solicitation to the address's solicited-node multicast group and seeing if anybody answers. That is duplicate address detection, and it turns a fork-and-clone workflow into an interesting afternoon.

The root cause is the one behind duplicated MACs: a snapshot freezes identity, and a fork produces two guests that believe identical things about themselves. Same-host forks land in 400–750ms and cross-host forks in 1.2–3.5s, and every one is a byte-for-byte copy of a guest that already had an address. The derivation paths make collisions likely rather than unlikely. EUI-64 derives the interface identifier straight from the MAC, so a cloned MAC is a cloned address. RFC 7217 stable-privacy addressing derives it from a per-interface `stable_secret` that a baked template ships identically to every clone. RFC 4941 temporary addresses derive from the guest's RNG, which has its own post-restore problem — see /blog/snapshot-clone-randomness-problem-explained.

What makes DAD worse than the IPv4 equivalent is how decisive it is. Duplicate IPv4 addresses produce ARP confusion and intermittent, maddening loss — bad, but the machine limps. A DAD failure marks the address `dadfailed` and the host simply does not use it; with `accept_dad` set to 2, a failure on the link-local disables IPv6 on that interface entirely. Your clone does not get flaky networking, it gets none, which is at least honest of it.

Neighbor discovery and ndp proxying

IPv6 has no ARP. Mapping an address to a link-layer address is done by neighbor discovery, an ICMPv6 protocol using neighbor solicitation and advertisement messages, with results in a cache you inspect via `ip -6 neigh show`. Functionally it is ARP with better manners and a multicast group it can scope its questions to.

Which brings up the analogue of proxy ARP. Clouds frequently hand you a single on-link /64 rather than a prefix you may route. The upstream router then believes every address in that /64 is directly reachable on your host's uplink segment, so it sends a neighbor solicitation onto that segment for your guest — and the guest is not there, it is behind a veth in a namespace. Nobody answers; the address is unreachable. NDP proxying fixes it: set `net.ipv6.conf.<iface>.proxy_ndp=1` on the uplink and add `ip -6 neigh add proxy <guest-address> dev <uplink>` per address, and the host answers on the guest's behalf, then routes inward.

Treat it as a fallback rather than a design. Every proxied address is a per-guest table entry you create on sandbox create and remove on delete — exactly the port-forward bookkeeping you adopted IPv6 to escape, wearing a different hat. If your provider will delegate a prefix, take the delegation and route it, and the whole apparatus disappears. Whether they will, and at what prefix length, varies enormously between clouds; check their documentation, not anyone else's setup.

The open window: firewalls that only speak IPv4

Here is the bug. Not exotic, not clever, and in a distressing number of production sandbox platforms right now. Somebody does the right thing: reads up on exfiltration, decides untrusted code gets default-deny egress with a small allowlist, writes it carefully. `iptables -P OUTPUT DROP`, an ESTABLISHED,RELATED accept, one allowed API endpoint, a DROP on 169.254.169.254 for the metadata service. Tests it. `curl https://attacker.example` hangs. Ship it.

`iptables` and `ip6tables` are separate rule sets with separate policies. Nothing they wrote applies to a single IPv6 packet. If the guest has any working v6 path — a delegated prefix, an RA from the underlay, a dual-stack upstream — its egress policy is "allow all," and neither the test nor the review noticed, because a bare `curl` picks whichever family the resolver and Happy Eyeballs steer it toward, and that day it picked v4. It is a very good lock on the front door of a house with an open window, and `curl -6` is a burglar who reads.

Two honest fixes. Either provably disable IPv6 in the guest and on its interfaces — then verify it, because "we do not configure IPv6" is not the same as "IPv6 is off" — or write the policy in something that covers both families at once. nftables' `inet` family does exactly that: one table, one chain, one policy, both address families. That property alone is worth the migration off iptables.

#!/usr/sbin/nft -f
# Applied in the ROOT namespace, on the host side of one sandbox's veth.
# `table inet` covers IPv4 and IPv6 in a single rule set -- the single best
# defence against a policy that only speaks one address family.

table inet sandbox {
  set allowed_v6 {
    type ipv6_addr
    flags interval
    elements = { 2001:db8:feed::10, 2001:db8:feed::53 }   # one API, one resolver
  }

  chain forward {
    type filter hook forward priority filter; policy drop;

    # ICMPv6 is NOT optional. Unlike ICMP on IPv4 -- which people drop with
    # mild consequences -- dropping ICMPv6 breaks IPv6. No packet-too-big
    # means no Path MTU Discovery, and the symptom is the worst kind: small
    # requests succeed, TLS handshakes complete, large responses hang forever.
    icmpv6 type { destination-unreachable, packet-too-big, time-exceeded,
                  parameter-problem, echo-request, echo-reply } accept

    ct state established,related accept

    # Deny the ambient-credential and neighbour-scan targets FIRST. Cloud
    # metadata services have IPv6 endpoints too -- AWS documents one in
    # fd00:ec2::/32 -- so a v4-only DROP on 169.254.169.254 covers nothing
    # here. Check your own provider's docs and block theirs explicitly.
    iifname "vh-demo" ip6 daddr fd00:ec2::/32  drop
    iifname "vh-demo" ip6 daddr fc00::/7       drop   # unique-local: neighbours
    iifname "vh-demo" ip6 daddr fe80::/10      drop   # link-local: the plumbing
    iifname "vh-demo" ip  daddr 169.254.0.0/16 drop   # same idea, other family

    # Then the allowlist. Everything not named here hits the chain policy.
    iifname "vh-demo" ip6 daddr @allowed_v6       tcp dport 443 accept
    iifname "vh-demo" ip6 daddr 2001:db8:feed::53 udp dport 53  accept

    counter comment "dropped-by-default"
  }
}

The chain policy is `drop` for both families, so your IPv4 allowlist lives in the same chain and cannot drift out of sync with the v6 one. That is the entire point. DNS is allowlisted as one specific resolver rather than port 53 to the world, because DNS is a data channel regardless of address family — /blog/controlling-network-egress-untrusted-code covers why an open resolver undoes an otherwise decent policy, and /blog/firecracker-guest-dns-resolution-explained covers how the guest gets a resolver at all.

Proving it from inside a guest

Do not take my word for it, and definitely do not take your firewall config's word for it. The only claim about egress policy worth making is one you made by asking a guest to try. Here is a probe that asks the same question twice, once forced to each family, and fails loudly when the answers disagree — the shape you want in CI on every template rebuild.

from pandastack import Sandbox

# Raw string: the \n below belongs to curl's format spec, not to Python.
PROBE = r'''#!/bin/sh
echo "--- addresses the guest actually has ---"
ip -6 addr show dev eth0 | sed -n "s/^ *inet6 /inet6 /p"
ip -6 route show default

echo "--- forced IPv4 ---"
curl -4 -sS -m 8 -o /dev/null -w "v4 %{http_code}\n" https://example.org || echo "v4 BLOCKED"

echo "--- forced IPv6 ---"
curl -6 -sS -m 8 -o /dev/null -w "v6 %{http_code}\n" https://example.org || echo "v6 BLOCKED"

echo "--- what the resolver hands back ---"
getent ahosts example.org | awk "{print \$1}" | sort -u
'''

# Snapshot-restore create: ~179ms p50. `with Sandbox.create(...) as sbx:`
# works too; the explicit form here just makes the teardown obvious.
sbx = Sandbox.create(template="base", ttl_seconds=300)
try:
    sbx.filesystem.write("/work/probe.sh", PROBE)

    # Capture to a file so a mid-probe non-zero exit can't cost us the output.
    run = sbx.exec("sh /work/probe.sh > /work/out.txt 2>&1", timeout_seconds=60)
    if run.exit_code != 0:
        print("probe exited", run.exit_code, run.stderr)

    report = sbx.filesystem.read("/work/out.txt").decode()   # read() -> bytes
    print(report)

    # The assertion that matters. A policy written only in iptables produces
    # exactly this: the front door is locked and the window is open.
    if "v4 BLOCKED" in report and "v6 BLOCKED" not in report:
        raise SystemExit(
            "egress policy is IPv4-only -- the guest still has a route out over IPv6"
        )
finally:
    sbx.kill()

Dual-stack reality: nobody is v6-only

Even a v6-native fleet has to reach a v4-only internet, so in practice you run dual-stack. Clients implement Happy Eyeballs (RFC 8305): resolve both A and AAAA, start connecting over one family, race the other after a short delay rather than waiting out a timeout. The goal is that a broken v6 path costs milliseconds instead of thirty seconds — great for users, terrible for operators, because a half-broken IPv6 path does not present as an outage. It presents as "things feel a bit slow sometimes," the least actionable bug report in existence.

The listening side trips up more people than routing does. A process bound to `0.0.0.0` listens on IPv4 only; one bound to `::` gets both, because Linux's `net.ipv6.bindv6only` defaults to 0 and maps incoming v4 connections into the v6 socket. Which one your framework picks is close to a coin flip across languages and web servers, so do not reason about it — check from inside the guest with `ss -ltn` and see whether the local address column reads `0.0.0.0:8080` or `[::]:8080`. If your health check connects over v6 to an app that only bound v4, the app is fine and the platform declares it dead.

The tension: baked identity vs. a unique routed prefix

One honest wrinkle, because it stops IPv6 being a free upgrade on a snapshot-based platform. A restored guest wakes with the exact network identity it was frozen with — that is the trick, and it is why the host patches the TAP's MAC and routes to match rather than asking the guest to reconfigure. It is also why the IPv4 model works so cleanly: every guest can be 10.200.0.2 because every guest lives alone in its own namespace, and the host does the per-slot mapping underneath.

A globally routed per-guest IPv6 prefix demands the opposite: the address must be unique, so it cannot be baked. Three ways out. Reconfigure the guest at restore — cheap in principle, but it adds a step to a boot path budgeted at 179ms and makes the guest agent a dependency. Bake a namespace-local address and translate at the host with NPTv6 (RFC 6296), a stateless one-to-one prefix rewrite rather than stateful NAT — it keeps the baked identity and yields a unique routable prefix, at the cost of admitting you did not escape translation. Or keep IPv4-plus-NAT for guest-initiated egress, where per-guest addressing does not matter, and reach for routed IPv6 only where you need inbound reachability. That last one is unglamorous and, for most fleets, correct.

The honest verdict

Bother with IPv6 if any of these hold. You need guests reachable from outside without a per-sandbox port-forward or NDP-proxy table. You peer with customer networks and RFC 1918 collisions are a live risk rather than a hypothetical. You run enough hosts that fleet-wide non-overlapping v4 planning is somebody's actual job. Or your underlay is already v6 and you are the one adding translation to avoid it, which is a strange place to be.

Do not bother if the numbers do not push you there. 16,384 slots per host against a practical ceiling set by memory and CPU is enormous headroom on a constraint that is not binding, and adopting IPv6 to relieve it adds a second address family to every firewall, log query, dashboard, monitoring check, and debugging session, forever. "We might need it someday" does not cover that.

One thing everybody should do regardless of which side of that line they land on. Decide explicitly whether IPv6 is on or off inside your guests, then verify the decision from inside a guest rather than from a config file. If it is off, prove there is no v6 route and no v6 address. If it is on, write your egress policy so one rule set covers both families, and put a probe in CI that fails when they disagree. The catastrophic version of this is not running out of addresses. It is a beautifully written allowlist, reviewed and approved, that a `curl -6` walks straight around.