Firecracker Guest MTU and Network Tuning Explained

The bug report always arrives in the same shape: "the sandbox has network, but npm install just... stops." By the time it reaches you, someone has already checked DNS. Someone has already curled a health endpoint and got a 200 back in 40ms. The guest pings its gateway, resolves names, completes TLS handshakes, and posts telemetry happily. And then the first genuinely large response arrives, the connection goes quiet, and it stays quiet until something times out twenty minutes later and everybody blames the registry.

I'm Ajay, I built PandaStack — this post is about the failure mode that eats more engineering hours than any other in virtualized networking: MTU mismatch and path-MTU black holes. It is not exotic, it is not a Firecracker bug, and it is almost never diagnosed correctly on the first day, because every tool people reach for first sends packets small enough to fit.

The frustrating part is that nothing errors. A firewall that rejects gives you a RST. A DNS failure gives you NXDOMAIN. An MTU problem gives you silence, which the human brain reliably misreads as "slow" rather than "broken".

Every hop has an MTU, and the guest never asks

Start with the topology, because the fix lives at a specific hop and you cannot pick the hop without the map. A packet leaving an application inside a Firecracker microVM crosses this chain:

• Guest eth0 — a virtio-net interface. The guest kernel brought it up with whatever MTU it decided on, which is 1500 unless something told it otherwise.
• The virtio-net device — Firecracker's device model moves frames between guest memory and the host. It does not negotiate an MTU on the guest's behalf; there's no hypervisor-side authority quietly correcting the guest's assumption.
• Host TAP (tap0) — inside the sandbox's own network namespace. Its own MTU, set when the slot was built.
• A veth pair — vg-<id> inside the sandbox namespace, vh-<id> in the root namespace. Two more interfaces, two more MTUs, and they do not have to agree with each other.
• Root-namespace routing plus NAT/iptables — where the packet gets its source address rewritten and handed toward the uplink.
• The host uplink — a cloud VNIC, possibly on an overlay, possibly inside a tunnel, with an MTU set by someone you will never meet.
• Everything past that — transit networks, the far side's fabric, and any VPN in between. You control none of it and it is where the surprises live.

Seven MTU values in a chain, and the endpoint that decides how big to make packets — the guest — is the one with the least information. It picks 1500 because that is what Ethernet has meant since roughly forever, then discovers the truth the hard way, in production, at 3am.

In a conventional setup, DHCP option 26 could hand the guest a corrected MTU. In a snapshot-restore world the guest often isn't doing DHCP at all — it comes back with a baked static identity — so that correction channel doesn't exist either. Which brings us to the part where nothing fails loudly.

Why it hangs instead of failing

Look at the sizes of the packets involved in "the network works". A TCP SYN is about 60 bytes on the wire. The SYN/ACK is the same. An HTTP GET with a normal header set is a few hundred bytes. A TLS ClientHello is typically 300-600. Every single one of those fits comfortably inside any MTU anyone has ever misconfigured. The handshake completes. The request goes out. Your connectivity test passes.

Then the server replies with real data. With a 1500-byte MTU the guest advertised an MSS of 1460 (1500 minus 20 bytes of IPv4 header minus 20 bytes of TCP header; 1448 if TCP timestamps are on), so the peer sends 1460-byte segments — 1500 bytes on the wire, with the Don't Fragment bit set, because Linux sets DF on TCP by default so that Path MTU Discovery can work.

That full-size segment reaches a hop whose MTU is 1450. The router cannot fragment it, because DF. It is supposed to drop the packet and send back ICMP type 3 code 4 — "fragmentation needed and DF set" — carrying the correct MTU. The sender would then shrink its segments and everything would heal in under a round trip. That is the entire design of PMTUD, and it is elegant, and it works right up until it meets a firewall configured by someone who read "block ICMP" on a security checklist in 2004.

With the ICMP gone, the sender learns nothing. It retransmits the same oversized segment, waits, retransmits again with exponential backoff, and the connection sits there in ESTABLISHED, looking perfectly healthy in `ss` output, transferring exactly zero further bytes. Hence the symptom table:

• curl hangs right after the TLS handshake completes — cause: the handshake fit; the first full-size response segment did not. If it dies during the handshake instead, it's usually the server's certificate chain, which is the first multi-kilobyte thing on the wire.
• apt-get update stalls at 0% on a Packages or Contents file — cause: the index request is small, the index itself is megabytes. Small metadata fetches succeeded, so apt looks like it's "connected".
• npm install freezes on one large tarball while small packages installed fine — cause: package metadata is JSON that fits in a segment or two; a 40 MB tarball is thousands of full-size segments, all of which are too big.
• git clone stops at "Receiving objects: 12%" and never moves — cause: negotiation is chatty and small, the packfile is one enormous bulk transfer. Shallow clones of small repos will keep working, which makes it look intermittent.
• ping works, ssh connects, `ssh` then hangs when you `cat` a big file — cause: interactive keystrokes are tiny packets, `scp`/bulk output is not. This one convinces people the problem is "the app".
• It works from the host but not from inside the guest — cause: the host's stack learned the real path MTU via PMTUD or has a route MTU pinned; the guest, one hop further in, learned nothing.

Where the 1500 actually goes

1500 is not a constant of nature. It is a budget, and every encapsulation layer in the path takes a cut before your payload gets what's left:

• VXLAN over IPv4 — 50 bytes (14 inner Ethernet + 8 VXLAN + 8 UDP + 20 outer IP), so the classic effective MTU is 1450. Over IPv6 it's 70.
• GENEVE — 8 bytes of base header plus variable-length options plus the same UDP/IP outer, so at least as expensive as VXLAN and not a fixed number. Anything that advertises "extensible metadata" is advertising "unpredictable overhead".
• WireGuard — roughly 60 bytes of overhead on IPv4, which is why 1420 is the number you see hardcoded in half the WireGuard configs on the internet.
• IPsec — anywhere from ~50 to ~90 bytes depending on transport vs tunnel mode, cipher, and whether it's wrapped in UDP for NAT traversal.
• Cloud provider fabrics — some VPCs default below 1500 (Google Cloud's VPC MTU default has historically been 1460, and is configurable). Verify the current number against your provider's own docs rather than trusting a blog post, including this one.
• Nesting — a Firecracker guest inside a cloud VM inside a VPC overlay reaching a peer across a VPN stacks all of the above. The deductions compose; they do not overlap.

Every layer that promises you a flat network is charging you bytes per packet for the illusion.

This bites microVM platforms specifically because a microVM adds hops nobody counts. In a plain container the app's packets take the host's route with the host's MTU. In a microVM the guest has its own kernel making its own independent decision about segment size, two extra virtual interfaces to disagree with, and a NAT boundary in between — more places to be wrong, and a guest structurally unaware of all of them.

Diagnosing it in five commands

Run these inside the guest. The whole point is to prove the failure is size-dependent, which takes about ninety seconds and immediately rules out DNS, TLS, routing, and the eight other things people will suggest in the thread.

# 1. What does each interface think its MTU is?
ip -br link show
ip route get 1.1.1.1          # a per-route "mtu lock 1420" overrides the link MTU

# 2. Probe the real path MTU. -M do sets DF, so nothing along the way may fragment.
#    payload + 8 (ICMP header) + 20 (IPv4 header) = bytes on the wire.
for s in 1472 1452 1422 1372 1300; do
  ping -M do -s "$s" -c 1 -W 2 1.1.1.1 >/dev/null 2>&1 \
    && echo "ok    payload=$s  wire=$((s + 28))" \
    || echo "FAIL  payload=$s  wire=$((s + 28))"
done

# 3. tracepath runs the sweep for you and names the hop where the path shrinks
tracepath -n 1.1.1.1

# 4. The signature: a small request is fine, a bulk transfer dies at zero bytes
curl -sS -o /dev/null -w 'small: %{size_download}B in %{time_total}s\n' https://example.com/
curl -sS -o /dev/null --max-time 20 -w 'bulk:  %{size_download}B in %{time_total}s\n' \
  https://deb.debian.org/debian/dists/stable/main/Contents-amd64.gz

# 5. Confirm from the socket itself: negotiated mss, and a retrans counter climbing
ss -tin state established

The reading is simple. If `wire=1500` fails and `wire=1480` succeeds, you have found your ceiling and the fix is arithmetic. If `tracepath` prints `pmtu 1450` at hop 3, it has handed you both the number and the culprit. And in step 5, an established socket with `mss:1460` and a `retrans:` count that climbs while `bytes_received` doesn't is the smoking gun: a connection actively trying and failing to push segments that are too large.

The fix, in order of preference

There are four fixes and they are not equivalent. Do them in this order, because the later ones cost you performance and the earlier ones don't.

1. Match the MTU end to end. Set the same value on the TAP, both ends of the veth pair, and the guest link. If every hop agrees, nothing ever needs to fragment and PMTUD never has to fire. This is the fix; the rest are mitigations.
2. Clamp TCP MSS on forward. `iptables -t mangle -A FORWARD ... -j TCPMSS --clamp-mss-to-pmtu` rewrites the MSS option in every forwarded SYN so both endpoints negotiate a size that fits, whatever the guest believed. Belt and braces for paths you don't control.
3. Allow ICMP type 3 code 4 through your firewall. Stop blanket-dropping ICMP and let PMTUD do the job it was designed for. This one is free and fixes the class of problem rather than one instance of it.
4. Only then lower the guest MTU. It works, but it's the blunt option: smaller packets mean more packets, more per-packet overhead, and more boundary crossings for the same bytes. Reach for it when you genuinely cannot fix the path.

# Host side, root namespace. Names follow the per-sandbox netns convention.
SBX=8f3a1c02              # sandbox id
NS="ns-$SBX"
MTU=1420                  # 1500 minus your overlay/tunnel overhead

# 1. Match the MTU end to end so nothing in the chain has to fragment.
ip link set dev "vh-$SBX" mtu "$MTU"          # veth, root namespace
ip -n "$NS" link set dev "vg-$SBX" mtu "$MTU" # veth, sandbox namespace
ip -n "$NS" link set dev tap0 mtu "$MTU"      # the TAP Firecracker writes into

# 2. And inside the guest (bake this — see the snapshot section below):
#      ip link set dev eth0 mtu 1420
#    or pin it on the route instead of the link:
#      ip route change default via 10.200.0.1 mtu 1420

# 3. Belt and braces: clamp the MSS in every forwarded SYN to the path MTU.
iptables  -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
ip6tables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
# If the route MTU isn't discoverable, pin it explicitly: -j TCPMSS --set-mss 1380

# 4. Stop black-holing the ICMP that makes PMTUD work in the first place.
iptables  -I INPUT   -p icmp   --icmp-type fragmentation-needed -j ACCEPT
iptables  -I FORWARD -p icmp   --icmp-type fragmentation-needed -j ACCEPT
ip6tables -I INPUT   -p icmpv6 --icmpv6-type packet-too-big     -j ACCEPT
ip6tables -I FORWARD -p icmpv6 --icmpv6-type packet-too-big     -j ACCEPT

# 5. Guest-side fallback that needs no ICMP at all (RFC 4821 packetization-layer PMTUD).
sysctl -w net.ipv4.tcp_mtu_probing=1

Two honest caveats. MSS clamping only fixes TCP — QUIC mostly survives because it does its own probing and keeps datagrams near 1200 bytes, but a WireGuard tunnel started inside the sandbox will black-hole exactly like TCP did, one layer deeper. And IPv6 routers cannot fragment at all: ICMPv6 type 2 "Packet Too Big" is the only signal there is. Drop it and IPv6 doesn't degrade, it simply stops working for anything large.

The snapshot wrinkle: you bake the MTU you set

Here's the part specific to snapshot-restore platforms, and it deserves saying plainly because it cuts both ways. A guest's interface MTU is kernel state. It lives in the guest's memory, in the netdev struct, alongside its routes and its neighbour table. A Firecracker snapshot captures guest memory and device state. Therefore the MTU is part of the frozen state, and every clone restored from that snapshot inherits it.

This is the same mechanism as baked network identity generally: a restored guest comes up believing it has the IP, MAC, and gateway it had at bake time, and the host patches its side to match. MTU rides along in that same category of "things the guest already decided". So if you SSH into a running sandbox, run `ip link set dev eth0 mtu 1420`, watch the download complete, and declare victory — you have fixed exactly one VM and nothing else. A create is a snapshot restore rather than a boot (~49ms for the restore step, p50 179ms end to end), so the next sandbox never re-runs your command; it resumes a memory image that never had it.

Per-sandbox netns means the fix is per-slot

The other reason a hand-applied `ip link set` doesn't stick is that there is no shared network to fix. Each sandbox gets its own Linux network namespace — `ns-<id>` holding `tap0` and the guest side of the veth pair, with the host side in the root namespace. PandaStack pre-allocates 16,384 /30 subnets per agent (a full /16 carved into four-address blocks: network, host, guest, broadcast), building the namespace, veth pair, TAP, and iptables rules ahead of time so that allocating one at create is a few milliseconds instead of the ~100ms it takes to build a namespace cold.

That's excellent for boot latency, but it also means "the network config" is not one object you can edit — it's 16,384 of them, built by the pool builder. A durable MTU fix has to live in two places: the code that constructs a slot (so every TAP and veth is born with the right value), and the template bake (so every restored guest agrees). Fixing a live namespace by hand is a diagnostic, not a deployment.

The exception, usefully, is the MSS clamp. Forwarding happens in the root namespace on the way out, so a single `TCPMSS --clamp-mss-to-pmtu` rule in the root namespace's mangle FORWARD chain covers every slot at once, including slots that don't exist yet. That asymmetry is why the clamp is such a good first response during an incident: it's one rule, it's global, and it takes effect on the next connection rather than the next re-bake.

Where throughput actually goes once it works

Once packets flow, MTU stops being a correctness issue and becomes a cost-per-byte issue. Firecracker's virtio-net device is one RX virtqueue and one TX virtqueue, and the expensive part of moving a packet is not the copy — it's the notification, the guest-to-host transition, and the device thread waking up. That cost is per packet, not per byte, so a 1420-byte MTU means about 6% more packets than 1500 for the same payload, and every one of those extra packets pays the full crossing tax.

This is also why segmentation offloads matter more in a microVM than people expect: with TSO/GSO the guest hands over one large buffer and the segmentation happens downstream, so a single crossing carries many packets' worth of data. If you're chasing throughput rather than chasing a hang, the queue and offload mechanics are covered properly in /blog/firecracker-net-rx-tx-queue-tuning-explained — the short version is that the ceiling is set by serialization at the device boundary, not by bandwidth.

Running the probe inside a sandbox

The nice thing about disposable microVMs is that the diagnostic is cheap to run from outside. Spin one up on the same path your workloads take, run the sweep, read the answer, throw it away. No SSH, no jump host, no persuading anyone to give you a shell on a production node.

from pandastack import Sandbox

PROBE = r"""#!/bin/sh
echo "--- links ---"
ip -br link show
echo "--- route ---"
ip route get 1.1.1.1

echo "--- pmtu sweep (DF set) ---"
for s in 1472 1452 1422 1372 1300; do
  if ping -M do -s "$s" -c 1 -W 2 1.1.1.1 >/dev/null 2>&1; then
    echo "ok    payload=$s  wire=$((s + 28))"
  else
    echo "FAIL  payload=$s  wire=$((s + 28))"
  fi
done

echo "--- small vs bulk ---"
curl -sS -o /dev/null --max-time 10 -w "small: %{size_download}B\n" \
  https://example.com/ || echo "small: FAILED"
curl -sS -o /dev/null --max-time 25 -w "bulk:  %{size_download}B\n" \
  https://deb.debian.org/debian/dists/stable/main/Contents-amd64.gz || echo "bulk: BLACK HOLE"
"""

sbx = Sandbox.create(template="base", ttl_seconds=900)
try:
    sbx.filesystem.write("/work/mtu-probe.sh", PROBE)
    r = sbx.exec("sh /work/mtu-probe.sh", timeout_seconds=120)
    print(r.stdout)

    if "BLACK HOLE" in r.stdout and "small: " in r.stdout:
        print("\nVERDICT: small requests fine, bulk transfer dead.")
        print("That is MTU/MSS. It is not DNS, TLS, or the registry.")
finally:
    sbx.kill()

Worth wiring into CI for any environment whose network path you didn't build yourself — it catches the most common "works in dev, mysteriously hangs in prod" failure before a customer does. To be thorough, snapshot the probe sandbox once and fork it per region or egress path (same-host fork is 400-750ms, cross-host 1.2-3.5s) so you sweep every path in parallel.

The summary is short. Small packets prove nothing; the handshake always fits. Silence is a symptom, not slowness. Match your MTU end to end, clamp MSS as insurance, stop dropping the ICMP that would have told you the answer, and remember that on a snapshot-restore platform the guest's network config is something you bake rather than something you configure.

If you want the layer underneath this one, the full topology — TAP, veth, per-sandbox namespaces, NAT, and baked identity on restore — is in /blog/firecracker-networking-explained. Throughput rather than correctness is in /blog/firecracker-net-rx-tx-queue-tuning-explained. And if you're deciding how the guest should attach to the host network at all, /blog/firecracker-tap-vs-macvtap-networking covers the tradeoff.