Conntrack Table Limits: The Shared Resource Under Your MicroVM Fleet

There is a class of outage that arrives dressed as something else. Nothing crashed. No process is missing. CPU is fine, memory is fine, disk is fine, and the hypervisor is doing exactly what it was told. But three tenants have opened tickets in ten minutes saying the same useless sentence: the network is being weird. Requests that worked an hour ago now time out, sometimes. Retries help, sometimes. A curl to the same endpoint from the host works perfectly, which is the detail that sends everyone off in the wrong direction for the next forty minutes.

Then someone runs dmesg and finds a line the kernel has been repeating, patiently and rate-limited, for a while: nf_conntrack: table full, dropping packet. And now you know two things. You know what happened, and you know that it has been happening to every tenant on the box simultaneously, because the resource that filled up was never per-tenant in the first place.

This post is about that resource. It sits underneath an otherwise very clean isolation story — separate kernels, separate namespaces, separate address space — and it is shared, finite, sized at boot from a number nobody chose deliberately, and almost never on anyone's dashboard until the first time it fills. I want to be precise about what it tracks and why, precise about how it is sized and accounted for across namespaces, and specific about the handful of things that actually help.

Why NAT requires the kernel to remember things

Start with the thing that is easy to skip. Forwarding a packet is stateless. A router looks at the destination address, consults a routing table, picks an interface, and sends it. It does not need to know anything about the packet that came before or the packet that will come after. You could reboot the router between two packets of the same TCP connection and, as far as forwarding is concerned, nothing is lost. The routing table is a function of topology, not of traffic.

Network address translation is not that. When a guest at 10.200.0.170 opens a connection to an API on the internet, the host rewrites the source address to its own public address and the source port to something it picked, and sends the packet on. The reply comes back addressed to the host's public address and that chosen port. At that moment the host has to answer a question that the packet itself cannot answer: which of the several hundred guests behind this address was that reply for? The packet contains no trace of 10.200.0.170. The only way to recover it is if the host wrote it down when the outbound packet went through.

That written-down record is a conntrack entry. It holds the original tuple, the translated tuple, the protocol state, and a timer. It is created on the first packet of a flow and it lives until the flow is finished or the timer expires. It is not an optimisation and it is not optional: it is the mechanism by which the reverse translation is possible at all. Stateful firewalling piggybacks on the same machinery — the reason a rule can say established,related accept is that conntrack already knows which flows are established — but NAT is the part that structurally cannot work without it.

One consequence worth internalising early: the cost of a flow to the table is completely divorced from the cost of that flow to anything else you measure. A connection that transfers four gigabytes and a connection that sends one SYN and is never answered occupy exactly the same amount of the table: one entry. Your bandwidth graphs, your CPU graphs and your egress bill all agree that the second connection is nothing. The conntrack table thinks they are identical. This is why the failure mode surprises people — the guest that fills the table is frequently the guest that looks idle on every other metric.

A fixed-size hash table, sized from RAM, at boot

The table has two numbers and people routinely confuse them, which matters because tuning one without the other produces a different problem rather than a fix.

nf_conntrack_buckets is the size of the hash table itself — how many hash buckets exist. Each bucket is one pointer, so the hash table's own memory is roughly the bucket count times eight bytes. It is genuinely fixed: allocated once, resized only by explicit action, and on a running system a resize is a real operation with a brief cost, not a free knob turn.

nf_conntrack_max is the ceiling on how many entries may exist at once. Entries hang off the buckets in chains, so max and buckets together determine the average chain length, which determines lookup cost. The kernel's default relationship is max equals four times buckets, and that ratio is a deliberate design point: an average chain of four is cheap to walk. If you raise max to a million and leave buckets at 65536, you have not bought a bigger table, you have bought an average chain of sixteen — every packet's lookup walks four times as far, and the cost lands in softirq CPU on the receive path. The symptom of that mistake is not drops; it is latency that gets worse as the table gets fuller, which is much harder to attribute.

Both defaults are derived at module load from the machine's RAM, clamped to a maximum. On most 64-bit hosts with more than about four gigabytes you will find 65536 buckets and a max of 262144. That is a number nobody on your team picked. It was picked by a heuristic written for general-purpose Linux boxes, and it does not know that you intend to run several hundred untrusted guests behind one masquerade rule. Read yours rather than trusting mine; the derivation has changed across kernel versions and distributions patch it.

Per-entry memory is the other half of the sizing arithmetic. The conventional figure is a few hundred bytes per entry — a struct nf_conn plus the extensions attached to it, and a NAT'd flow carries the NAT extension, so a gateway's entries are at the larger end of that range. Do not take my number for it. slabtop will tell you the real object size on your kernel, and multiplying it out is how you convert a proposed nf_conntrack_max into a number of megabytes of unswappable kernel memory that you are agreeing to potentially spend.

Reading the counters, before dmesg reads them to you

Everything you need is in /proc/sys/net/netfilter and in the conntrack tool from conntrack-tools. The important discipline is reading nf_conntrack_count rather than counting lines in /proc/net/nf_conntrack: the former is a counter the kernel maintains, the latter is a full walk of the table that takes locks and gets slower exactly as the situation gets worse. I have watched a monitoring script make an incident measurably worse by scraping the wrong one every fifteen seconds.

# ---------------------------------------------------------------------------
# 1. The two numbers that matter. Read them on the HOST, in the root netns,
#    because that is where your masquerade rule lives.
# ---------------------------------------------------------------------------
cat /proc/sys/net/netfilter/nf_conntrack_count      # entries in use, right now
cat /proc/sys/net/netfilter/nf_conntrack_max        # the ceiling
cat /proc/sys/net/netfilter/nf_conntrack_buckets    # hash buckets (see below)

# Typical 64-bit host with plenty of RAM:
#   count    18342
#   max      262144
#   buckets  65536          <- max is 4x buckets. That ratio is the default.

# Utilisation as a percentage, which is the thing you actually want on a graph.
awk 'NR==FNR{c=$1; next}{printf "conntrack %d/%d = %.1f%%\n", c, $1, c*100/$1}' \
  /proc/sys/net/netfilter/nf_conntrack_count \
  /proc/sys/net/netfilter/nf_conntrack_max

# ---------------------------------------------------------------------------
# 2. Per-CPU stats. This is where the failures are recorded, and unlike the
#    kernel log it does not get rotated away before you look at it.
# ---------------------------------------------------------------------------
conntrack -C            # same as nf_conntrack_count, one number
conntrack -S            # per-CPU counters -- read these columns:
#   drop            packets dropped because the table was full
#   early_drop      entries evicted early to make room (pressure, pre-failure)
#   insert_failed   could NOT insert -- usually NAT tuple exhaustion, not size
#   invalid         packets conntrack could not associate with a flow
#   search_restart  hash resize raced with a lookup (benign unless constant)

# DON'T do this on a busy host to count entries -- it walks the whole table:
#   wc -l /proc/net/nf_conntrack        # O(n), locks, and lies by the time it ends
# Use nf_conntrack_count. It is a counter, not a scan.

# ---------------------------------------------------------------------------
# 3. What the table is actually costing you in kernel memory.
# ---------------------------------------------------------------------------
sudo slabtop -o | grep -E 'OBJS|nf_conntrack'
#  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
# 24576  18342  74%    0.31K   1536       16      6144K nf_conntrack

# ---------------------------------------------------------------------------
# 4. Per-namespace view. Each sandbox netns has its own count; the flows that
#    leave the box are counted AGAIN in the root namespace, where NAT happens.
# ---------------------------------------------------------------------------
for ns in $(ip netns list | awk '{print $1}'); do
  n=$(ip netns exec "$ns" cat /proc/sys/net/netfilter/nf_conntrack_count 2>/dev/null || echo 0)
  [ "$n" -gt 0 ] && printf '%-28s %6d\n' "$ns" "$n"
done | sort -k2 -rn | head -20
# The top of that list is your noisy neighbour. Name it before it names itself.

# ---------------------------------------------------------------------------
# 5. Ship it. A node_exporter textfile collector is ~10 lines and turns the
#    whole thing from an incident into a dashboard.
# ---------------------------------------------------------------------------
cat > /usr/local/bin/conntrack-textfile.sh <<'EOF'
#!/bin/sh
set -eu
OUT=/var/lib/node_exporter/textfile/conntrack.prom
C=$(cat /proc/sys/net/netfilter/nf_conntrack_count)
M=$(cat /proc/sys/net/netfilter/nf_conntrack_max)
{
  echo "# TYPE host_conntrack_entries gauge"
  echo "host_conntrack_entries $C"
  echo "# TYPE host_conntrack_limit gauge"
  echo "host_conntrack_limit $M"
  conntrack -S | tr ' ' '\n' | grep -E '^(drop|insert_failed|early_drop)=' \
    | awk -F= '{s[$1]+=$2} END {for (k in s) printf "host_conntrack_%s_total %d\n", k, s[k]}'
} > "$OUT.tmp" && mv "$OUT.tmp" "$OUT"
EOF
chmod +x /usr/local/bin/conntrack-textfile.sh
# Run it from a 15s systemd timer. Alert at 70% of limit, page at 85%,
# and alert on ANY nonzero rate of drop or insert_failed -- those are not
# warnings, those are packets you already lost.

The counter to understand properly is early_drop. When an insert would exceed nf_conntrack_max, the kernel does not immediately give up: it looks in the target hash bucket for an entry that has not been marked assured — roughly, a flow that has not yet proven itself to be a real two-way conversation — and evicts it to make room. If it finds one, your packet gets through and somebody else's half-formed flow quietly dies. If it does not, you get the drop and the log line.

This means early_drop is your early warning, and it is strictly better than the log line because it fires before anything user-visible has happened. A host with a rising early_drop rate and zero drops is a host that is already failing at the edges — you are evicting real flows to make room — and it is perhaps a week away from failing loudly. Alert on it. The kernel log line is not a warning; it is a receipt for packets you have already lost.

Network namespaces do not partition this

Here is the nuance that makes the whole thing counterintuitive on a microVM host, and it is the one I would most want a reader to leave with.

Network namespaces really do give each sandbox its own conntrack table in the sense that matters for isolation: its own entries, its own count, its own view. A process in one namespace cannot list, read or interfere with another namespace's flows. That is real and it is worth having. But three things sit underneath that separation and they are all shared.

• The hash table is global. nf_conntrack_buckets is one table for the whole machine; the namespace is part of the key, not a partition of the storage. Creating a thousand namespaces does not create a thousand hash tables. It creates a thousand tenants of one hash table, and it is only writable from the initial namespace.
• The memory comes from one slab. Entries are allocated from a single kernel slab cache shared by every namespace on the box. Per-namespace limits cap counts; they do not reserve or partition kernel memory.
• Per-namespace maxima do not add up to a budget. On modern kernels nf_conntrack_max is per-namespace and each new namespace inherits the initial namespace's value. Two hundred sandbox namespaces each inheriting a max of 262144 does not mean each gets a fair share — it means you have issued two hundred independent permits, each authorising 262144 entries, against one pool that cannot serve a fraction of the sum. The limits are not a quota system. They are two hundred separate promises the machine cannot keep.

And then the part that actually decides the outcome: the NAT does not happen in the sandbox's namespace. It happens in the root namespace, on the way out of the box, because that is where the uplink and the public address are. So a guest opening a connection to the internet creates an entry in the root namespace's accounting — the shared one — regardless of how beautifully isolated its own namespace is. If your topology NATs twice, once inside the sandbox's namespace and once again on the host, a single guest flow costs two entries, and you can be surprised by a factor of two you never modelled.

KVM gives every guest its own kernel for everything except the one hop where its packets meet everyone else's. That hop is where the shared table lives.

This is the honest limit of the microVM isolation story on the network side. A container and a microVM are wildly different propositions for kernel-level isolation, and I will argue that all day. But the guest that runs a port scanner is not exhausting your host's conntrack table by exploiting a shared kernel. It is exhausting it by doing the completely legitimate thing of sending packets to the internet through the NAT you built for it. There is no escape here, no privilege boundary crossed, nothing to patch. It is a capacity question wearing a security question's clothes.

Sixteen thousand subnets, two hundred thousand entries

PandaStack makes this arithmetic concrete in a way I find clarifying. Each agent pre-allocates 16,384 /30 subnets out of a 10.200.0.0/16 pool, each with its own network namespace, veth pair and TAP device, built ahead of time so that a create does not pay for namespace setup. The address plan says this host can address sixteen thousand sandboxes. Addresses are cheap; the whole point of pre-allocation is that they are so cheap you can waste them on warm slots that nothing is using.

Now put the default conntrack ceiling next to it. At 262144 entries, and assuming a modest hundred concurrent flows per active sandbox — which is nothing; a single npm install against a registry with keepalive off will beat it, and an AI agent fanning out across a dozen APIs will beat it comfortably — the table supports about 2,600 concurrently active sandboxes. Change the assumption to a thousand flows each, which is entirely ordinary for a crawler or an integration test suite hitting a service mesh, and it is 262. The addressing plan and the flow-tracking plan are three orders of magnitude apart, and only one of them is written down anywhere.

That gap is not a bug in either plan. It is the difference between a resource that is a naming decision and a resource that is a memory allocation. But it does mean the honest sizing question for a dense host is never how many sandboxes fit — it is how many concurrently active flows fit, and how you stop one tenant from answering that question on everyone else's behalf.

There is a mitigating factor and it is a real one: scale-to-zero. A sandbox that is idle, hibernated or snapshotted holds no flows at all, because it has no packets in motion. Table pressure tracks the actively-working subset, not the fleet size, which is why a platform with aggressive TTL reaping and idle hibernation runs far below its worst case in practice. Right up until the morning that a scheduled job fans out across four hundred sandboxes at once, and the worst case is the case.

The knobs, and what each one actually costs

Sizing and timeouts are the two levers. Sizing buys you headroom at the cost of kernel memory; timeouts reduce how long a finished flow keeps occupying space, which is very often the bigger win because most of a busy table is not active connections at all.

• nf_conntrack_buckets — What it does: sets the hash table size, which fixes the average chain length for a given max. What raising it costs: eight bytes per bucket of permanently allocated memory, plus a live resize operation. Only writable from the initial namespace, and on older kernels only via the module's hashsize parameter.
• nf_conntrack_max — What it does: caps concurrent entries machine-wide, per namespace. What raising it costs: a few hundred bytes of unswappable kernel memory per entry you actually create, and — if you do not raise buckets to match — longer hash chains and more CPU on every packet. Raise both or raise neither.
• nf_conntrack_tcp_timeout_established — What it does: how long an idle established TCP flow stays tracked. Default is 432000 seconds, five days. What lowering it costs: a genuinely idle long-lived connection gets forgotten and its next packet is treated as invalid, which breaks long-poll and idle SSH. An hour is defensible for ephemeral workloads; if you need more, fix it with keepalives rather than with days.
• nf_conntrack_tcp_timeout_time_wait — What it does: how long a closed flow's tuple is held so a late duplicate cannot be misread as a new connection. What lowering it costs: a small, real correctness risk on a network that reorders and delays heavily. At high connection churn this is frequently the single largest slice of the table, and taking it from 120 seconds to 30 is the highest-yield change on this list.
• nf_conntrack_tcp_timeout_syn_sent — What it does: how long an unanswered SYN keeps an entry. What lowering it costs: almost nothing on a healthy network. This is the knob that decides how much of your table one port scan can hold, because a scan is thousands of SYNs to closed ports and every one of them books an entry for the full timeout.
• nf_conntrack_udp_timeout — What it does: how long a UDP pseudo-flow lives after the last packet. What lowering it costs: for genuinely long-lived UDP the stream timeout still applies, so the risk is low. Every DNS query is one of these, and a service-discovery loop makes DNS the second-largest slice of a sandbox host's table.
• nf_conntrack_tcp_loose — What it does: when 1 (the default), conntrack will create an entry for a TCP flow whose handshake it never observed. What turning it off costs: mid-stream pickup after a failover stops working. On a gateway that sees every packet of every flow you do not need it, and leaving it on means spoofed or stray packets can create entries.
• nf_conntrack_acct — What it does: adds per-entry byte and packet counters. What turning it on costs: memory on every entry, forever. Worth it if you are metering per-tenant egress from conntrack; pure cost if you are not.

# /etc/sysctl.d/60-conntrack-microvm-host.conf
#
# Sizing and timeouts for a host that NATs a few hundred short-lived guests.
# Every value here is a trade. The comments are the trade.

# --- Sizing -----------------------------------------------------------------
# Buckets first, max second. Keep max at ~4x buckets so the average hash chain
# stays short; raising max alone just makes lookups walk longer chains, which
# shows up as softirq CPU, not as a drop counter.
#
#   buckets memory = buckets * 8 bytes           (one pointer per bucket)
#   entry   memory ~= 300-500 bytes per flow     (nf_conn + NAT extension)
#
# 262144 buckets  ->    2 MiB of hash table
# 1048576 entries -> ~320-500 MiB of slab, IF you actually fill it
net.netfilter.nf_conntrack_buckets = 262144
net.netfilter.nf_conntrack_max     = 1048576

# --- Timeouts ---------------------------------------------------------------
# The default established timeout is 432000 seconds. That is five days. It is
# there so that an idle SSH session survives a long weekend. Sandboxes do not
# have long weekends; they have a TTL measured in minutes.
net.netfilter.nf_conntrack_tcp_timeout_established = 3600    # was 432000 (5d)

# TIME_WAIT entries are pure overhang: the flow is over, the kernel is holding
# the tuple so a late duplicate cannot be mistaken for a new connection. At high
# churn this is the single largest slice of the table.
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 30        # was 120

# Half-open and half-closed states. A port scanner lives almost entirely in
# SYN_SENT; 120s per unanswered SYN is how one nmap fills a table.
net.netfilter.nf_conntrack_tcp_timeout_syn_sent   = 30       # was 120
net.netfilter.nf_conntrack_tcp_timeout_syn_recv   = 20       # was 60
net.netfilter.nf_conntrack_tcp_timeout_fin_wait   = 30       # was 120
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 30       # was 60
net.netfilter.nf_conntrack_tcp_timeout_last_ack   = 20       # was 30

# UDP: one DNS query = one entry, held for 30s after the answer arrives. An
# agent doing service discovery in a loop makes this the second-largest slice.
net.netfilter.nf_conntrack_udp_timeout        = 15           # was 30
net.netfilter.nf_conntrack_udp_timeout_stream = 60           # was 120

# Everything that is not TCP/UDP/ICMP -- GRE, ESP, whatever a guest invents.
net.netfilter.nf_conntrack_generic_timeout = 120             # was 600

# --- Behaviour --------------------------------------------------------------
# 0 = do not create an entry for a TCP flow whose handshake we never saw. On a
# gateway that sees every packet of every flow, mid-stream pickup is not a
# feature you need, and it is a free way for spoofed traffic to make entries.
net.netfilter.nf_conntrack_tcp_loose = 0

# Byte/packet accounting per entry. Genuinely useful for per-tenant egress
# billing; it also grows every entry. Turn on deliberately, not by accident.
net.netfilter.nf_conntrack_acct = 0

# Apply, then verify -- a typo here fails silently and you find out in dmesg.
#   sudo sysctl --system
#   sudo sysctl -a --pattern 'nf_conntrack_(max|buckets|tcp_timeout_established)'
#
# Note: buckets is only writable from the initial network namespace. On older
# kernels it is not a sysctl at all and you resize the live table with:
#   echo 262144 | sudo tee /sys/module/nf_conntrack/parameters/hashsize

The other exhaustion nobody expects: tuples, not entries

There is a second, sneakier limit that lives in the same subsystem and produces a completely different symptom, and if you only know about nf_conntrack_max you will misdiagnose it.

When the host masquerades a guest's connection, it must choose a source port on the host address such that the resulting tuple — source address, source port, destination address, destination port, protocol — is unique. For any single destination address and port, there are only about 64,000 possible source ports. So if four hundred sandboxes are all hammering the same package registry, or the same model API endpoint, from one host address, the ceiling on concurrent connections to that one destination is the port space, and it can be reached while the conntrack table as a whole is at fifteen percent.

The symptom is bizarre if you have not seen it: connections to one specific host fail or hang while everything else on the box is perfectly healthy. Conntrack's counters tell you exactly what happened — insert_failed goes up, drop does not — but nothing in dmesg mentions a table being full, because it is not. Nothing is full except the tuple space for one destination.

• Diagnosing it — Look for: insert_failed climbing in conntrack -S while nf_conntrack_count sits well below max, and failures concentrated on one destination. Confirm with: conntrack -L -d <that address> | wc -l, and compare against 64k.
• Fixing it — More host source addresses is the direct fix: SNAT across a small pool rather than masquerading to one address, which multiplies the tuple space by the number of addresses. Also widen net.ipv4.ip_local_port_range, enable the fully-random port allocation flag on the SNAT rule, and turn on connection reuse in the clients so they stop opening a fresh socket per request.
• Not fixing it — Raising nf_conntrack_max does nothing here, which is the trap: the knob everyone reaches for is the one knob that cannot possibly help, and the fact that it does not help is often read as evidence that conntrack was not the problem.

Mitigations, in the order I would apply them

The instinct is to raise the limit, and raising the limit is fine, but it should be third on the list. The first two changes are the ones that reduce demand rather than adding supply, and they are the ones that keep working when the fleet doubles.

Tighten the timeouts first, because the cheapest entry is the one that expired ten minutes ago. On a host churning through short-lived sandboxes, an enormous fraction of the table is flows that are over. Nothing is using them. They are being retained against a scenario — a delayed duplicate packet arriving days later — that is a real concern for a long-lived router and close to irrelevant for a workload whose entire lifetime is measured in minutes.

Then stop tracking what you never translate. Conntrack is engaged on a path because something on that path asked for it: a NAT rule, a stateful match, a helper. If a flow is neither NAT'd nor stateful-filtered, an explicit NOTRACK rule in the raw table keeps it out of the table entirely. On a sandbox host the obvious candidates are the host-to-guest control paths — the SSH or vsock bridge the agent uses to exec into guests, health probes, metrics scrapes. These are high-count, entirely internal, and need no translation whatsoever. Be careful and specific: NOTRACK on a path that is being NAT'd or stateful-filtered breaks that path immediately and completely, and the breakage will look like a routing problem.

Then limit per guest. The /30-per-sandbox layout is a gift here, because one guest is exactly one source address and a per-source-IP limit is precisely a per-sandbox limit with no extra bookkeeping. Two ceilings are worth having: concurrent tracked connections per guest, and new-connection rate per guest. You need both, because they catch different pathologies — the crawler holding thousands of sockets open trips the first, and the scanner opening and abandoning connections faster than they expire trips the second while staying well under the first.

#!/usr/bin/env bash
# Two mitigations that attack the problem from opposite ends: stop creating
# entries you do not need, and stop any one guest from creating all of them.

set -euo pipefail
POOL=10.200.0.0/16          # the whole per-sandbox /30 pool on this agent
UPLINK=eth0

# ---------------------------------------------------------------------------
# A. NOTRACK -- do not track what you do not translate.
#
# conntrack is engaged because something asked for it. If a flow is neither
# NAT'd nor stateful-filtered, an explicit notrack in the raw table keeps it out
# of the table entirely. The classic candidates on a sandbox host are the
# host<->guest control paths: the SSH/vsock bridge, health probes, the metrics
# scrape. They are high-count, low-value, and they need no translation.
# ---------------------------------------------------------------------------
nft -f - <<'EOF'
table ip pandastack_raw {
  chain prerouting {
    type filter hook prerouting priority raw; policy accept;
    # Agent -> guest control plane on the /30 link. Purely local, never NAT'd.
    ip saddr 10.200.0.0/16 ip daddr 10.200.0.0/16 notrack
  }
  chain output {
    type filter hook output priority raw; policy accept;
    ip daddr 10.200.0.0/16 tcp dport { 22, 8080 } notrack
  }
}
EOF
# Verify it is doing something: entry count should drop, not just stop growing.
#   conntrack -C; sleep 30; conntrack -C

# ---------------------------------------------------------------------------
# B. Per-guest connection ceiling. One misbehaving sandbox should hit a wall
#    long before the host does. A /30 means one guest IP, so a per-source-IP
#    limit is exactly a per-sandbox limit -- the addressing does the grouping
#    for you.
# ---------------------------------------------------------------------------
nft -f - <<'EOF'
table inet pandastack_limits {
  chain forward {
    type filter hook forward priority filter; policy accept;

    # Concurrent tracked connections per guest IP. 512 is generous for a build
    # or an agent; a crawler will notice, which is the point.
    ct state new meter per_guest_conns { ip saddr ct count over 512 } \
      counter drop

    # New-connection RATE per guest. Catches the scanner that opens and closes
    # fast enough to stay under the concurrency cap while still churning the
    # table at thousands of inserts a second.
    ct state new meter per_guest_rate { ip saddr limit rate over 200/second burst 400 packets } \
      counter drop
  }
}
EOF

# iptables equivalents, if you have not migrated:
#   iptables -A FORWARD -m conntrack --ctstate NEW \
#     -m connlimit --connlimit-above 512 --connlimit-mask 32 -j DROP
#   iptables -A FORWARD -m conntrack --ctstate NEW -m hashlimit \
#     --hashlimit-name guestrate --hashlimit-mode srcip \
#     --hashlimit-above 200/sec --hashlimit-burst 400 -j DROP

# Watch the counters rather than guessing at the threshold:
#   nft list meter inet pandastack_limits per_guest_conns
#   nft list chain inet pandastack_limits forward   # the counter on each rule

# ---------------------------------------------------------------------------
# C. Teardown. When a /30 goes back in the pool, its entries must go with it,
#    or the next tenant on that address inherits a stranger's flow state.
# ---------------------------------------------------------------------------
release_slot() {          # $1 = the /30, e.g. 10.200.0.168/30
  nft delete rule ip nat postrouting handle "$(nft -a list chain ip nat postrouting \
    | awk -v n="$1" '$0 ~ n {print $NF}')" 2>/dev/null || true
  conntrack -D -s "$1" 2>/dev/null || true    # <- do not skip this line
}

It is worth being clear about what a hypervisor-level rate limiter does and does not cover here, because it is a reasonable thing to assume covers you. Firecracker's network rate limiter is a token bucket on bandwidth and operations for a TAP device — bytes per second and packets per second. It is genuinely useful, it stops a guest from saturating the host NIC, and it does nothing at all about this problem. A port scan is minuscule in bytes and packets and lethal in entries. If your only per-guest network control is a bandwidth cap, you have capped the resource that was never scarce and left the scarce one unmanaged.

Only then raise the ceiling, in proportion to RAM, with buckets and max moved together, and with the slab arithmetic done in advance so you know what you are agreeing to spend. And if you want to remove the problem structurally rather than manage it, the real answer is to stop needing NAT: route guests with addresses that do not require translation, and forwarding goes back to being stateless and free. That is a bigger change with its own security posture to think through — a routable guest is a reachable guest — and stateful filtering re-engages conntrack the moment you add an established,related rule. But it is the only option on this list that changes the shape of the problem instead of the size of it.

Make it a metric before it makes itself an incident

The thing that turns this from a nasty incident into a boring capacity line item is embarrassingly simple: put conntrack utilisation on the same dashboard as CPU and memory, and treat it as a first-class fleet resource rather than a kernel implementation detail. It is two integers and a division. node_exporter's conntrack collector already exports the entries and the limit; newer versions export the failure counters too, and the textfile collector in the first snippet covers you either way.

Alert on utilisation crossing roughly seventy percent, page at eighty-five, and alert separately on any nonzero rate of drop, insert_failed or early_drop. Those three are qualitatively different from a utilisation gauge: utilisation is a forecast, and those counters are a record of harm that has already occurred to somebody's traffic. Also break the gauge down per namespace, because the aggregate tells you that the box is in trouble and the per-namespace view tells you which tenant to talk to — and on a multi-tenant host, the second question is the one you will be asked first.

The complementary practice is to measure workloads rather than guess at them. Before a new workload class gets scheduled across a fleet, run one instance and watch what it does to a flow counter. The number is stable enough to be useful for capacity planning and it is often startling — the workloads that turn out to be flow-hungry are rarely the ones anybody would have nominated.

# Measure the flow footprint of a workload BEFORE you run 200 of it.
#
# The number you want is not "how much CPU does this job use". It is "how many
# conntrack entries does one instance of this job hold at its peak", because
# that number times your concurrency is the load on a table you did not size
# for it.

import json
import time
from pandastack import Sandbox

PROBE = r'''#!/bin/sh
# Runs INSIDE the guest. The guest kernel has its own conntrack table, so this
# measures the flows this workload originates -- each of which will also occupy
# one entry in the host's root-namespace table, where the masquerade happens.
peak=0
for _ in $(seq 1 120); do
  n=$(cat /proc/sys/net/netfilter/nf_conntrack_count 2>/dev/null || echo 0)
  [ "$n" -gt "$peak" ] && peak=$n
  sleep 0.5
done
echo "PEAK=$peak"
ss -s | head -2
'''

sbx = Sandbox.create(template="base", ttl_seconds=900)
try:
    sbx.filesystem.write("/tmp/probe.sh", PROBE)
    sbx.exec("chmod +x /tmp/probe.sh")

    # conntrack is a module; on a stock guest you may need it loaded before
    # /proc/sys/net/netfilter exists at all. No rules required -- just the hook.
    sbx.exec("modprobe nf_conntrack 2>/dev/null; "
             "sysctl -w net.netfilter.nf_conntrack_acct=1 2>/dev/null || true")

    # Background the probe, then run the real workload against it.
    sbx.exec("setsid /tmp/probe.sh > /tmp/probe.out 2>&1 &")

    workload = sbx.exec(
        "cd /srv/app && npm ci && npm run test:integration",
        timeout_seconds=540,
    )

    time.sleep(2)
    peak = sbx.exec("grep PEAK /tmp/probe.out || echo PEAK=unknown")

    print(json.dumps({
        "exit_code": workload.exit_code,
        "peak_flows": peak.stdout.strip(),
    }, indent=2))
finally:
    sbx.destroy()

# A run that reports PEAK=90 is a workload you can safely pack. A run that
# reports PEAK=9000 is a workload that needs its own connection ceiling before
# it shares a host with anything you care about.
#
# The honest caveat: this measures the guest's own view. Flows that leave the
# box are counted a second time on the host. Budget for both.

What I would actually do on a dense host

Condensed, in order, assuming a host NATing a few hundred short-lived guests behind one address.

1. Read the current numbers before changing anything. count, max, buckets, and a full conntrack -S. Write them down. You cannot tell whether a change helped without a before.
2. Put utilisation and the three failure counters on a dashboard, with per-namespace breakdown. Do this even if you change nothing else, because the next incident is diagnosed in seconds instead of forty minutes.
3. Tighten the timeouts. time_wait and syn_sent first, then established from five days to something that matches your actual workload lifetimes. This is the cheapest real capacity you will find.
4. NOTRACK the internal control paths that are neither NAT'd nor stateful-filtered. Verify the entry count actually falls; if it does not, you did not find the traffic you thought you found.
5. Add a per-guest concurrent-connection ceiling and a per-guest new-connection rate limit. Set them generously enough that no legitimate workload notices and tightly enough that one guest cannot consume the box.
6. Raise buckets and max together, in proportion to host RAM, having first computed the slab cost. Verify the ratio stayed near four to one.
7. Flush conntrack for a subnet when its slot is released, before that address is reused. Stale entries surviving into a new tenant's occupancy of the same address is a correctness and privacy problem, not just a capacity one.
8. Watch insert_failed independently of utilisation, so that the day you exhaust the tuple space to one popular destination you diagnose it in a minute rather than spending an afternoon raising a limit that cannot help.

None of this is exotic. It is all standard netfilter administration that a network engineer would consider unremarkable. The reason it bites platform teams specifically is that the microVM story is so good everywhere else — hardware-enforced isolation, a separate guest kernel per tenant, namespace-per-sandbox, atomic teardown — that it is genuinely easy to believe the isolation goes all the way down. On the network path it goes all the way down to the NAT hop, and then it stops, and the thing on the other side of that hop is a fixed-size hash table that was sized by a heuristic at boot and is shared by everyone.

It is an unglamorous place for a multi-tenant platform's reliability to live. But a shared, exhaustible, unpriced resource with no quota system and no dashboard is going to find you eventually, and it is much nicer to meet it on a Tuesday afternoon while reading a graph than at three in the morning while reading dmesg and discovering that your entire fleet's networking went soft because one guest found nmap and got curious.