IP Address Planning for a MicroVM Fleet

Subnet exhaustion is the outage that gives you no warning and then all of it at once. For weeks the graph is flat, creates are fast, nobody thinks about IP addresses. Then one afternoon a create fails with something unhelpful about address allocation, and thirty seconds later every create is failing, and you are reading `ip netns list` on a production host trying to work out why there are four thousand namespaces for eleven running VMs. Nothing degraded gracefully on the way there because address space does not degrade — it is available, and then it is gone.

I'm Ajay; I built PandaStack, which runs Firecracker microVMs and hands each one its own network namespace and its own tiny subnet. This is the boring half of microVM networking: not how packets move — /blog/firecracker-networking-explained and /blog/firecracker-network-namespace-isolation-explained cover the mechanism — but how you decide which addresses to hand out, how many you actually get, and what happens when your allocator quietly loses track of them. We shipped the naive version first. It leaked. This post is what I wish I'd thought about before writing a single line of it.

The address math, exactly

Start with the unit. The standard shape for a per-VM link is a /30: four addresses, of which the first is the network address, the last is the broadcast address, and the two in the middle are usable. One goes on the host side (the gateway), one goes to the guest. That is the entire point — a /30 is a subnet with exactly enough room for a point-to-point link and no room at all for a neighbor. There is nothing for the guest to ARP for, because there is nothing else in its subnet. Isolation by subnet math, sitting on top of isolation by namespace.

Now the capacity question, which is pure arithmetic and worth doing on paper before you pick a range. Carving a prefix of length P into /30s gives you 2^(30-P) subnets. A /16 gives 2^14 = 16,384. A /24 gives 2^6 = 64. Sixty-four. If your instinct was to grab a /24 out of the corporate space because that felt like plenty, you have provisioned sixty-four concurrent microVMs per host, which a single busy CI job will eat before lunch. The gap between /24 and /16 is not a matter of taste; it is two orders of magnitude.

PandaStack carves 10.200.0.0/16 into /30s, which is where the 16,384 pre-allocated subnets per agent comes from — 10.200.0.0/30, 10.200.0.4/30, 10.200.0.8/30, all the way to 10.200.255.252/30. Mapping a slot index to its CIDR is four-times-N arithmetic, and it is worth writing the mapping down explicitly rather than letting it live implicitly in whatever loop assigns addresses:

# Slot math: a /16 carved into /30 blocks.
# 2^(30-16) = 16384 subnets. A /24 would give 2^(30-24) = 64.

python3 - <<'PY'
BASE   = (10 << 24) | (200 << 16)   # 10.200.0.0
PREFIX = 16
SLOTS  = 1 << (30 - PREFIX)         # 2**14 = 16384

def slot_to_cidr(n):
    if not 0 <= n < SLOTS:
        raise ValueError("slot %d outside the pool" % n)
    net = BASE + n * 4              # every /30 is 4 addresses wide
    o   = [(net >> s) & 0xFF for s in (24, 16, 8, 0)]
    fmt = lambda last: "%d.%d.%d.%d" % (o[0], o[1], o[2], last)
    return fmt(o[3]) + "/30", fmt(o[3] + 1), fmt(o[3] + 2)

for n in (0, 1, 63, 64, 16383):
    cidr, gw, guest = slot_to_cidr(n)
    print("slot %-6d %-18s gw %-15s guest %s" % (n, cidr, gw, guest))
PY

# slot 0      10.200.0.0/30      gw 10.200.0.1     guest 10.200.0.2
# slot 1      10.200.0.4/30      gw 10.200.0.5     guest 10.200.0.6
# slot 63     10.200.0.252/30    gw 10.200.0.253   guest 10.200.0.254
# slot 64     10.200.1.0/30      gw 10.200.1.1     guest 10.200.1.2
# slot 16383  10.200.255.252/30  gw 10.200.255.253 guest 10.200.255.254

Two things fall out of writing it this way. First, the slot index is the identity — an integer you can store, index, and reconcile, from which the addresses are derived deterministically rather than remembered. Second, the ceiling is now a number you can put on a dashboard instead of a surprise. On a PandaStack agent that ceiling is 16,384 sandboxes, and in practice it never binds: host memory and CPU run out long before the /16 does. That is the correct relationship between the two limits. If your address space is the tighter constraint, you sized it wrong.

Picking a range: 10/8 feels infinite until it isn't

RFC 1918 gives you three private ranges: 10.0.0.0/8 (16,777,216 addresses — 4,194,304 /30s), 172.16.0.0/12 (1,048,576 addresses), and 192.168.0.0/16 (65,536 addresses). The 10/8 space is enormous and everybody knows it, which is precisely the problem: everybody also uses it. 10.0.0.0/8 feels infinite right up until your customer's VPN peers into it and discovers that the /16 you picked for microVM plumbing collides with the /16 their finance department has been on since 2014. Now their route table has two claims on the same prefix and one of you loses, nondeterministically, in a way that presents as "the database is intermittently unreachable."

So the selection rule is not "pick something free on this host." It is: pick a range that is unlikely to appear in anyone else's network, carve it from the least popular corner of the space, write it down somewhere public, and never change it. The low end of 10/8 (10.0.x, 10.1.x) and the usual home-router prefixes (192.168.0/1.x) are the most contested addresses on the private internet. Something like 10.200.0.0/16 is unremarkable enough to be free almost everywhere, and specific enough that a customer can check it against their own allocations in thirty seconds — provided you publish it.

What one slot actually is

Before the failure modes, the concrete shape. A slot is not just an address — it is a network namespace, a veth pair with one end in that namespace, a TAP device for Firecracker to drive, the addressing on both ends, and the NAT rule that lets the guest out. Standing one up and tearing it down looks like this:

SLOT=42
NET=10.200.0.168; GW=10.200.0.169; GUEST=10.200.0.170   # slot 42 -> 42*4 = 168
ID=vm$SLOT

# --- create -------------------------------------------------------------
ip netns add ns-$ID
ip link add vh-$ID type veth peer name vg-$ID
ip link set vg-$ID netns ns-$ID

ip addr add $GW/30 dev vh-$ID && ip link set vh-$ID up
ip netns exec ns-$ID ip addr add $GUEST/30 dev vg-$ID
ip netns exec ns-$ID ip link set vg-$ID up
ip netns exec ns-$ID ip link set lo up

# The TAP Firecracker drives, inside the namespace.
ip netns exec ns-$ID ip tuntap add tap0 mode tap
ip netns exec ns-$ID ip link set tap0 up

# MTU: match the underlay, or large packets vanish silently.
ip link set vh-$ID mtu 1450
ip netns exec ns-$ID ip link set vg-$ID mtu 1450

iptables -t nat -A POSTROUTING -s $NET/30 -j MASQUERADE

# --- teardown (do ALL of it, in this order) ------------------------------
iptables -t nat -D POSTROUTING -s $NET/30 -j MASQUERADE
conntrack -D -s $NET/30 2>/dev/null || true    # flush before reuse
ip netns del ns-$ID                            # takes tap0, vg-, routes, ARP
ip link del vh-$ID 2>/dev/null || true         # the host half does NOT auto-die
                                               # if it was never moved in

Deleting the namespace is atomic for everything inside it, which is most of the mess. What it does not take with it is the state you created outside the namespace: the NAT rule in the root namespace's table, the conntrack entries the kernel is holding for flows from that /30, and — depending on how you built it — the host-side veth. Those are the pieces that leak. And the reason building all of this on the create path is unattractive is cost: cold, it is on the order of 100ms of netlink and syscall work. Pre-build the slot ahead of time and claiming one drops to single-digit milliseconds, which is how PandaStack keeps a full snapshot-restore create at 179ms p50 and 203ms p99 without networking dominating the budget.

The leak: how a pool drains without telling you

Here is the part I got wrong. The first PandaStack allocator was an in-memory free list: a slice of free slot indices in the agent process, pop on allocate, push on release. It is the obvious implementation, it is fast, it has no dependencies, and it is correct as long as the process lives forever and every release path runs. Neither of those is true.

The failure is quiet and cumulative. An agent restarts — a deploy, an OOM, a crash — and the free list rebuilds from whatever the process can see. If the reconstruction is optimistic, previously-claimed slots come back as free and you double-allocate; if it is pessimistic, slots belonging to VMs that died during the restart stay marked used forever. A cleanup path panics halfway through and the slot is never pushed back. A VM is force-killed and the release code never runs at all. Each of those leaks one slot. One slot a day, out of 16,384, is invisible — and then it is four months later, the number that was never on a dashboard has reached zero, and every create on that host fails at once.

What makes it genuinely nasty is that the symptom does not point at the cause. Creates fail on a host with idle CPU and free memory. The VMs that are running are fine. Nothing in the logs from four months ago is still around. You end up counting namespaces by hand and discovering thousands of orphans whose VMs stopped existing weeks ago.

The fix: a database-owned allocator with a reclaim pass

We moved the allocator into the database, and the shape that works has three parts: one row per slot with an explicit owner, a claim that is a single atomic statement, and a janitor that reconciles the table against the VMs that actually exist. The last part is the one people skip, and it is the one that makes leaks self-healing instead of permanent.

-- One row per /30. The slot index is the identity; the CIDR is derived.
CREATE TABLE allocations (
  slot        INTEGER PRIMARY KEY,        -- 0 .. 16383
  cidr        TEXT    NOT NULL,
  sandbox_id  TEXT    UNIQUE,             -- NULL = free; UNIQUE = no double-claim
  owner_agent TEXT,
  claimed_at  TIMESTAMPTZ,
  released_at TIMESTAMPTZ
);

-- CLAIM: lowest free slot, taken in one statement. No read-then-write race.
UPDATE allocations a
   SET sandbox_id = $1, owner_agent = $2, claimed_at = now(), released_at = NULL
  FROM (SELECT slot FROM allocations
         WHERE sandbox_id IS NULL
         ORDER BY slot
         FOR UPDATE SKIP LOCKED
         LIMIT 1) AS free
 WHERE a.slot = free.slot
RETURNING a.slot, a.cidr;
-- SKIP LOCKED: two concurrent creates never receive the same slot.

-- RELEASE: idempotent, safe to call twice.
UPDATE allocations
   SET sandbox_id = NULL, owner_agent = NULL, released_at = now()
 WHERE sandbox_id = $1;

-- RECLAIM: the janitor. Every 60s, free slots whose VM no longer exists.
-- $2 is the list of sandbox ids this agent can actually see running RIGHT NOW.
UPDATE allocations
   SET sandbox_id = NULL, owner_agent = NULL, released_at = now()
 WHERE owner_agent = $1
   AND sandbox_id IS NOT NULL
   AND sandbox_id <> ALL ($2::text[])
   AND claimed_at < now() - interval '5 minutes'   -- grace for in-flight creates
RETURNING slot;   -- log these: a nonzero count is a bug somewhere upstream

The grace window matters: without it the janitor will reclaim a slot out from under a create that has claimed its address but not yet started its VM. And the `RETURNING slot` is not decoration — every reclaimed slot is evidence that some release path failed, so you want that count as a metric with an alert on it, not silently swallowed by a job that quietly papers over the bug forever.

• Durability across restarts — In-memory free list: state dies with the process; the pool is reconstructed from guesses on every restart. Database-owned allocator: state outlives the agent, a restart re-reads the truth.
• Leak behavior — In-memory free list: leaks are permanent and silent; each missed release is a slot gone until the next restart, which may reintroduce it as a double-allocation instead. Database-owned allocator: leaks are visible rows and the janitor reclaims them within a minute.
• Multi-agent safety — In-memory free list: only safe if exactly one process ever touches the pool; two agents on one host will hand out the same address. Database-owned allocator: the unique constraint plus SKIP LOCKED makes concurrent claims correct by construction.
• Reclaim story — In-memory free list: no reconciliation is possible, because there's nothing durable to reconcile against. Database-owned allocator: reconcile the table against the running VM list on a timer; drift converges to zero.
• Complexity — In-memory free list: about thirty lines and a mutex. Database-owned allocator: a table, three queries, a background loop, and a hard dependency on the database being reachable at create time. Real cost, worth paying.

Reuse hazards: the address is free, the kernel disagrees

Reclaiming a slot correctly is necessary but not sufficient, because the host kernel holds state keyed to addresses that outlives your bookkeeping. The classic version: VM A at 10.200.0.170 opens a long-lived outbound connection, the VM is destroyed, the slot goes back in the pool, and forty seconds later VM B comes up on the same address. If the conntrack entries for A's flows are still live, replies to A's connection can be delivered into B's namespace. That is cross-talk between tenants, arriving through an address the kernel considers to be the same host. It is rare, it is timing-dependent, and it is exactly the class of bug you cannot reproduce on demand.

The mitigations are unglamorous and you should do all of them. Flush conntrack for the freed subnet as part of teardown rather than trusting timeouts. Delete the netns explicitly so its ARP and neighbour cache go with it, and delete the host-side veth so no stale neighbour entry maps the old address to a dead interface. Remove the NAT rule you added, matching on the same /30 you inserted with, so the root namespace's table does not accumulate thousands of dead entries — a POSTROUTING chain with four thousand stale rules is both a leak and a measurable per-packet cost. And if you can afford it, do not reuse a slot immediately: a short quarantine on freed slots costs you nothing when the pool is 16,384 deep and removes the whole timing window.

Two more that bite in production. NAT hairpinning: when a guest tries to reach a public address that resolves back to its own host, the packet leaves through MASQUERADE, comes back, and needs a translation that only exists if you configured it — the usual symptom is that VMs can reach the whole internet except your own platform's public endpoint. And MTU: if your underlay is a tunnel (VXLAN, WireGuard, a cloud overlay), the veth's default 1500 is too large, and the failure mode is that small packets work, TLS handshakes complete, and large responses hang forever. Set the MTU explicitly on both veth ends at slot build time and stop guessing. /blog/firecracker-guest-mtu-and-network-tuning-explained goes deeper on that one.

When a /16 isn't enough: /31s, more space, and IPv6

Suppose the ceiling actually binds — very dense hosts, or a design where one host serves far more than a few thousand VMs. There are three escapes, in increasing order of ambition.

The first is /31 point-to-point links, per RFC 3021. On a genuine point-to-point link there is no need for a network or broadcast address, so a /31 gives you two usable addresses in two addresses of space — exactly double the density of a /30. A /16 carved into /31s is 2^15 = 32,768 links instead of 16,384. The caveat is compatibility: /31 support is standard in the Linux kernel and iproute2, but not every tool, agent, embedded stack, or cloud appliance in the path handles a /31 gracefully, and the failures are confusing when they happen. Verify it end to end on your actual stack — guest OS, DHCP or static config, monitoring agents — before betting a fleet on it.

The second is simply more space. Nothing forces one /16 per host: you can allocate a distinct /16 per agent out of a larger, documented block, which keeps the per-host math identical while removing any fleet-wide ceiling. Since the addresses are host-local behind NAT, two agents can even use the same range as long as nothing needs to route between them — though I would not recommend leaning on that, because the day you want direct host-to-host VM routing, overlapping ranges become an expensive retrofit.

The third is IPv6, which is the honest long-term answer. A /64 per host gives you an address space that cannot be exhausted by any number of VMs you will ever run, and unique local addresses (fc00::/7, with a randomly generated /48) make accidental collisions with a customer's network essentially impossible in a way no RFC 1918 choice can match. The cost is real: dual-stack guests, IPv6-aware egress policy, tooling and monitoring that handle v6 addresses correctly, and the fact that plenty of the internet your guests want to reach is still v4-only, so you need NAT64 or dual-stack egress anyway. It solves address planning completely and adds a category of operational work — worth it at scale, overkill if a /16 per host has you covered by a factor of a hundred.

The short version

Do the arithmetic before you pick a prefix: /30s out of a /16 give you 16,384 links, out of a /24 they give you 64, and the difference between those two decisions is whether address space is a footnote or your production ceiling. Pick an unfashionable range so your customers' VPNs don't collide with it, and publish it. Make the slot index the durable identity, keep the allocator in a database with an explicit owner and an atomic claim, and run a janitor that reconciles against reality and alerts when it finds anything. Tear down all of the slot — conntrack, NAT rule, host-side veth — not just the parts that die with the namespace. Then put the free-slot count on a dashboard with an alert, because the whole reason exhaustion hurts is that nobody was watching a number that only moves in one direction.

PandaStack's agent is open source under Apache-2.0, so the NATID pool, the slot allocator, and the teardown paths are all readable — including the parts that exist only because we got this wrong the first time and had to move the free list out of process memory after it drained a pool we weren't measuring.