Rate-Limiting the Network on a Per-Sandbox Basis

Everyone sets a CPU limit. Everyone sets a memory limit. Almost nobody sets a network limit, and then one Tuesday afternoon a single tenant runs a container pull in a loop, the host's NIC saturates, and forty other sandboxes on that machine get slow simultaneously for reasons that show up in none of their own metrics. Their CPU is fine. Their memory is fine. Their DNS lookups now take two seconds.

I'm Ajay, and I build PandaStack — a Firecracker microVM platform where each sandbox gets its own network namespace, its own veth pair and its own /30. That topology is why I have opinions here: it means the per-sandbox enforcement point already exists as a side effect of how isolation works, whether or not you ever attach a shaper to it. I'll be explicit up front about what we actually enforce today, because the honest answer is not "all of it," and I'd rather you get the mechanism right than take my word for a feature list.

The enforcement point already exists

The reason per-sandbox network limits are tractable in a microVM platform, and miserable in a shared-kernel one, is that the isolation you built for security handed you a per-tenant interface for free. On PandaStack every sandbox lives in its own Linux network namespace with its own veth pair, carved out of a single /16.

Concretely: a slot index gives a /30 out of 10.200.0.0/16 (which is where the 16,384-sandboxes-per-agent ceiling comes from — that is the subnet space, not a licence limit). The namespace is named after the slot; the root-side veth is vh-, the namespace-side veth is vg-, and inside the namespace sits tap0, the device Firecracker actually attaches the guest NIC to. The guest's own IP, MAC and gateway are baked into the snapshot and never change, which is exactly why each sandbox needs its own namespace: a thousand guests all believing they are 172.20.6.118 is fine as long as no two of them are ever in the same routing table.

# What the agent builds per sandbox (paraphrased from the real create path).
# idx -> /30 from 10.200.0.0/16; names derive from the slot index.
NS=ns-p0000001f
VH=vh-p0000001f     # root netns  (host side)
VG=vg-p0000001f     # sandbox netns (peer)

ip netns add "$NS"
ip link add "$VH" type veth peer name "$VG"
ip link set "$VG" netns "$NS"

ip addr add 10.200.0.125/30 dev "$VH" && ip link set "$VH" up
ip -n "$NS" addr add 10.200.0.126/30 dev "$VG" && ip -n "$NS" link set "$VG" up

# The device Firecracker binds the guest NIC to, inside the namespace.
ip netns exec "$NS" ip tuntap add dev tap0 mode tap
ip -n "$NS" addr add 172.20.6.117/30 dev tap0     # baked gateway
ip -n "$NS" link set tap0 up

# Everything above is on the HOST, in a namespace the guest cannot see.
# The guest is behind tap0, inside a different kernel entirely.

That last line is the part that matters for shaping. A container's traffic controls live in the same kernel as the workload; a microVM's live in a different one. The guest can do whatever it likes to its own routing table, its own qdiscs, its own iptables — it cannot see, reach or unwind anything on the host side of tap0. A shaper attached to vh- or vg- is not a policy the workload cooperates with. It is a property of the wire.

Per-sandbox network isolation gives you a per-sandbox enforcement point whether you use it or not. The question is never where to put the limit. It is which end of the pair, and what shape.

Which end of the veth pair — and why people get this backwards

A qdisc shapes packets that an interface transmits. That single sentence resolves the whole thing, and almost everyone (me, first time) still gets it wrong, because "host side" sounds like the place to control the guest's uploads. It isn't. A veth pair is a wire: what one end transmits, the other receives. So map the four combinations once and keep the map.

• Guest → internet (upload, exfiltration, the thing you fear): these packets are RECEIVED on vh- in the root namespace, and TRANSMITTED out of vg- inside the namespace. Queue them on vg-'s egress.
• Internet → guest (download, the npm install, the model weights): these are TRANSMITTED out of vh- toward the namespace. Queue them on vh-'s egress.
• tap0 is the same story one hop further in: its egress is traffic toward the guest, its ingress is traffic from the guest.
• Ingress on any of these can only be policed (drop over-rate packets) — never queued — because there is no queue on the receive path to hold them in.

The good news, and the thing that makes this much less painful than the general Linux traffic-control literature suggests, is that you own both ends. In a plain bridged setup you often have access only to the interface facing the workload, which is why the ifb (intermediate functional block) device pattern exists: redirect ingress into a virtual device so you have an egress path to queue on. With a veth pair inside a namespace you can usually skip that entirely and shape the other end, which is a real qdisc with a real queue and none of the policing tradeoffs. I'll still cover ifb, because sooner or later you meet a topology where you only have one end.

Shaping with a token bucket filter

The token bucket is the right default and the reason is behavioural, not mathematical. Tokens accumulate at a fixed rate up to a maximum; sending consumes them; a sender that has been quiet has a bucketful and can spend it all at once. That is precisely the shape of real workloads. An npm install or a git clone genuinely needs a burst — it is a short, bounded, legitimate spike followed by nothing — and a hard flat cap punishes it while doing nothing extra to stop the tenant who is malicious for six hours straight.

NS=ns-p0000001f
VH=vh-p0000001f
VG=vg-p0000001f

# ── Guest UPLOAD (guest -> internet): queue on the namespace-side veth.
# rate    : sustained ceiling, tokens/sec
# burst   : bucket size -- how much a quiet sandbox may spend at once
# latency : how long a packet may sit in the queue before it is dropped
ip netns exec "$NS" tc qdisc add dev "$VG" root tbf \
    rate 50mbit burst 512kb latency 50ms

# ── Guest DOWNLOAD (internet -> guest): queue on the root-side veth.
# Downloads are usually where you want the generous burst: dependency
# installs are bursty by nature and the tenant is not the one choosing
# the sender's rate.
tc qdisc add dev "$VH" root tbf \
    rate 200mbit burst 2mb latency 50ms

# Change a live limit without dropping the sandbox's connections:
tc qdisc replace dev "$VH" root tbf rate 400mbit burst 2mb latency 50ms

Size the burst deliberately, because this is where tbf configurations quietly fail. The bucket has to be at least as large as the number of bytes the rate implies for one timer tick, or the shaper cannot hand out tokens fast enough to reach its own configured rate and you end up throttled well below the number you wrote down. Too small and your throughput is mysteriously bad; too large and the cap is decorative for short transfers, which may be exactly what you want. A few hundred kilobytes at tens of megabits is a sane starting point, and then you measure.

Which brings us to the part people skip: verifying that the thing is on and biting. A shaper you have not watched drop a packet is a shaper you have not tested.

# Is the qdisc attached, and is it actually doing work?
tc -s qdisc show dev vh-p0000001f

# qdisc tbf 8001: root refcnt 2 rate 200Mbit burst 2Mb lat 50ms
#  Sent 41903118 bytes 30119 pkt (dropped 0, overlimits 812 requeues 0)
#  backlog 0b 0p requeues 0
#
# overlimits > 0  -> the shaper is engaging: packets were delayed. GOOD.
#                    This is the number that proves the limit is live.
# dropped    > 0  -> the queue overflowed 'latency'. Either the sender is
#                    far over the rate, or your latency budget is too tight.
# overlimits == 0 after a real transfer -> you shaped the wrong direction.

# Prove it end to end from inside the guest, not from the host:
#   guest$ curl -o /dev/null -w '%{speed_download}\n' https://example.com/1GB.bin
# Then compare against the same command with the qdisc removed:
#   tc qdisc del dev vh-p0000001f root

When you want a floor and a ceiling instead of one number

tbf gives you one rate. HTB gives you a guaranteed rate plus a ceiling the class may borrow up to when its parent has spare capacity, which is the shape you want if your objection to hard caps is that they waste bandwidth nobody is using.

VH=vh-p0000001f
tc qdisc add dev "$VH" root handle 1: htb default 10
tc class add dev "$VH" parent 1: classid 1:10 htb \
    rate 50mbit ceil 200mbit burst 512kb

# rate = what this class is guaranteed
# ceil = what it may burst to IF the parent has spare capacity to lend

And shaping the uplink has a second trap that costs an afternoon if you meet it cold. Guest traffic leaves the host through a shared MASQUERADE rule, so by the time packets reach the uplink's transmit queue their source address has already been rewritten to the host's. A tc filter matching on source IP matches nothing. Mark the packets before the address translation happens — mangle runs ahead of source NAT at the postrouting hook — and then filter on the mark.

WAN=ens4

# One class per sandbox under a shared parent, so idle capacity is lent out.
tc qdisc add dev "$WAN" root handle 1: htb default 999
tc class add dev "$WAN" parent 1:  classid 1:1  htb rate 9gbit
tc class add dev "$WAN" parent 1:1 classid 1:31 htb rate 50mbit ceil 2gbit

# Mark by the sandbox's UNIQUE /30 -- in mangle, which runs BEFORE the
# MASQUERADE rewrites the source and destroys the only thing you can key on.
iptables -t mangle -A POSTROUTING -s 10.200.0.124/30 -j MARK --set-mark 31

# Now the filter has something to match that survives NAT.
tc filter add dev "$WAN" parent 1: protocol ip handle 31 fw classid 1:31

# The per-sandbox /30 is doing real work here: it is a stable, collision-free
# classification key that exists because of how the network pool allocates,
# not because anyone designed it for shaping.

Ingress, policing, and the ifb device

If you are in a topology where you genuinely only have the interface facing the workload — a tap on a bridge, say — you hit the asymmetry head-on. Egress has a queue, so you can delay packets and smooth a flow. Ingress does not: the packet is already here, and your only options are accept or drop. Policing works, in the sense that it enforces an average, but it enforces it by discarding, which TCP interprets as congestion and reacts to with retransmits and window collapse. The measured throughput lands well under the number you configured, and every user reports it as "the network is broken," not "the network is limited."

# Option A: police on ingress. Crude, drops, but zero extra devices.
tc qdisc add dev tap0 handle ffff: ingress
tc filter add dev tap0 parent ffff: protocol ip u32 match u32 0 0 \
    police rate 50mbit burst 256kb drop flowid :1

# Option B: redirect ingress to an ifb device so you get a real egress
# path -- and therefore a real queue -- to shape on.
modprobe ifb
ip link add ifb-sbx type ifb && ip link set ifb-sbx up

tc qdisc add dev tap0 handle ffff: ingress
tc filter add dev tap0 parent ffff: protocol ip u32 match u32 0 0 \
    action mirred egress redirect dev ifb-sbx

tc qdisc add dev ifb-sbx root tbf rate 50mbit burst 512kb latency 50ms
tc -s qdisc show dev ifb-sbx     # overlimits here = shaping, not dropping

In a veth-per-sandbox layout I would reach for the other end of the pair before I reached for ifb. One fewer device per sandbox matters when the sandbox count is in the thousands and every one of them is created and destroyed in under a second.

In the VMM or on the host?

Firecracker has its own answer to this, and it is a good one. Each network interface takes an rx_rate_limiter and a tx_rate_limiter, and each of those is a pair of token buckets: one counting bandwidth in bytes, one counting operations. Each bucket has a size, a refill_time in milliseconds from which the refill rate is derived, and an optional one_time_burst that is spent before ordinary refilling begins — which is the mechanism that lets a fresh VM do its first big download at full speed and then settle into its sustained rate. The same structure exists on drives, and the API can patch both at runtime.

So why would you use tc at all? Because the two enforcement points have different properties, and on a snapshot-restore platform the difference is decisive. Device configuration that lives inside the VMM is part of the machine, and on this platform the machine arrives pre-built: every create is a restore of a snapshot that was baked once, elsewhere, possibly weeks ago. We have already been bitten by this class of thing more than once — the drive cache mode is a snapshot property that a partial-update cannot change, so flipping it means re-baking templates rather than shipping a config. Anything you want to vary per customer, per plan, or per abuse incident is better placed where you can change it with a command on a live host and see it take effect on the next packet.

Host-side shaping is also the layer that survives a guest doing something clever, because it is not in the guest's kernel, its virtio queues or its device tree. And it composes: the VMM limiter and the qdisc do not fight, they stack, and the effective rate is whichever is tighter.

The policy question, which is the part people get wrong

Mechanism is a weekend. Policy is the thing that generates support tickets for a year. Three arguments, in the order they usually come up.

First: a hard cap is simple, defensible and wasteful. If you sell N sandboxes on a host and give each a fixed slice of the NIC, then at any moment when most of them are idle — which is most moments, because sandbox workloads are bursty by nature — the majority of your bandwidth sits unused while an active tenant is being throttled next to it. Burst-then-shape fixes that without giving anything away: the sustained rate is your protection against the six-hour abuser, the burst is your accommodation of the ninety-second dependency install. The overwhelming majority of legitimate traffic in this kind of platform is short bursts, so the burst allowance is the parameter your users actually feel.

Second: size the sustained rate against the host, not the tenant. The question is not "how much does a sandbox deserve" but "what happens when every sandbox on this host does its worst simultaneously." You are going to oversubscribe, because not oversubscribing means selling a fraction of your NIC — the point of the sustained rate is to make the worst case degraded rather than catastrophic. Degraded is everyone getting less. Catastrophic is a saturated uplink taking out your control plane's health checks along with the tenants, at which point the scheduler starts marking healthy hosts dead and the failure spreads.

Third: pick your unit. Per-sandbox is the obvious granularity and the wrong one for billing-adjacent abuse — a tenant who is happy to create fifty sandboxes gets fifty times the bandwidth budget from a per-sandbox cap. Per-sandbox caps protect the host. Per-workspace quotas protect the business. Those are separate controls, enforced in separate places, and one does not substitute for the other.

Rate limiting is not metering (and our meter is not a limiter)

Here is the part I want to be concrete and honest about, because it is the gap between what a platform measures and what a platform enforces, and that gap is where abuse lives.

PandaStack meters egress. The implementation is deliberately unglamorous: the agent reads the cumulative receive-byte counter of each sandbox's host-side veth from sysfs, keeps an in-process watermark per sandbox, and records the delta. Counters are cumulative, so a shrinking counter means the interface was rebuilt — a namespace re-allocated on wake, say — and the watermark restarts rather than billing a whole counter over again. It runs on the usage meter loop, every five minutes by default. The rate is set to zero dollars per gigabyte, deliberately, because our pricing page says egress is not billed and a meter that quietly disagreed with the published price was a latent bug we chose to close in the direction of the promise.

# What the meter actually reads. Nothing exotic.
cat /sys/class/net/vh-p0000001f/statistics/rx_bytes   # guest -> out
cat /sys/class/net/vh-p0000001f/statistics/tx_bytes   # out -> guest

# Sampling that at a 5-minute interval tells you what happened.
# It does not tell you what is happening, and it stops nothing:
# a saturated 10 Gb link moves a lot of bytes between two samples.
# That is the whole argument for a qdisc. The meter is a witness;
# the shaper is a control.

Keeping the measurement even at a zero rate is the right call for a reason that has nothing to do with billing: a workspace pushing hundreds of gigabytes is worth seeing regardless of whether it costs the customer anything, and it is one of the clearer signals that something is being used as a proxy, a torrent client, or an exfiltration pipe. But a five-minute sampler is an audit trail, not a brake, and describing it as "we limit bandwidth" would be false. Different controls, different jobs — and if you only build one, build the one that observes, because you cannot set a sane limit for traffic you have never measured.

Connection counts are the sharper failure mode

Bandwidth is the limit people think about. Connection count is the one that actually takes down neighbours, and it does so at traffic volumes that would never register on a bandwidth graph. A crawler opening thousands of concurrent connections moves very little data per connection and can exhaust the host's connection-tracking table long before it comes anywhere near the NIC's capacity.

The reason this hurts other tenants is that connection tracking is a host-wide resource that network namespaces do not partition the way people assume. Every NAT'd guest flow costs an entry in a fixed-size table, and when that table is full the kernel starts dropping new connections — for everyone on the box, not for the tenant who filled it. The symptom is other sandboxes failing to open sockets, which almost nobody's monitoring attributes to a neighbour's crawler.

Our own design has a fingerprint of this. Because the guest IP is baked into the snapshot, every sandbox from a given template shares it — so the namespace rewrites that shared address to the sandbox's unique veth address on the way out, before packets reach the root namespace, or connection tracking would collide between tenants. The per-sandbox /30 is not decoration; it is what makes each tenant's flows distinguishable to the host's NAT.

# Watch the resource that actually runs out first.
sysctl net.netfilter.nf_conntrack_max
cat /proc/sys/net/netfilter/nf_conntrack_count

# Cap CONCURRENT connections per sandbox, keyed on its unique /30.
iptables -A FORWARD -s 10.200.0.124/30 -p tcp --syn \
    -m connlimit --connlimit-above 512 --connlimit-mask 0 -j REJECT

# Cap the RATE of new connections, with a burst so normal apps are unaffected.
iptables -A FORWARD -s 10.200.0.124/30 -p tcp --syn \
    -m hashlimit --hashlimit-name sbxconn \
    --hashlimit-above 100/sec --hashlimit-burst 200 -j DROP

# Note REJECT vs DROP: reject the concurrency cap so the guest's client
# library fails fast with a clear error, drop the rate cap so a scanner
# gets no feedback loop to tune against.

If you implement exactly one network limit in a multi-tenant sandbox platform, and bandwidth shaping feels like a project you cannot start this quarter, make it this one. It is two iptables rules, it costs nothing at steady state, and it defends against the failure mode with the widest blast radius.

What PandaStack actually enforces today

In the spirit of not describing techniques as features, here is the current state of the network controls in our agent, including the gaps.

• Cross-tenant isolation: the first rule in the host's forward chain drops any traffic from the sandbox pool to the sandbox pool. Without it a tenant could scan the /16 from inside its own VM and reach neighbours' SSH, app ports and databases, because the per-namespace forwarding rules expose every guest port to whoever can route to it.
• Cloud metadata block: the entire 169.254.0.0/16 link-local range is dropped for guest traffic. On our provider that endpoint hands out the host VM's service-account token, and a tenant with a shell and a curl would otherwise have had it.
• Egress port denylist: the well-known Stratum mining-pool ports are dropped at the forward chain, tunable through an environment variable. It is a denylist and it does not stop a determined miner on a custom port, but it kills the default configuration of essentially every off-the-shelf one — and crypto abuse is a recurring free-tier problem that also gets the whole project flagged by the cloud provider's own detectors.
• Egress metering: per-sandbox byte counts on a five-minute loop, at a zero-dollar rate, kept for visibility and abuse detection rather than billing.
• Control-plane rate limiting: the agent's HTTP API runs a per-workspace token bucket — 50 requests of burst, refilling at 25 per second — so an accidental loop gets a 429 rather than a saturated agent. Same primitive as everything above, applied to the API surface rather than the wire.
• Not implemented: per-sandbox bandwidth shaping via tc, and per-sandbox connection-count caps. Both are described above as general technique, not as things we ship.

The reason for that last line is ordering rather than disagreement. The controls that shipped first were the ones defending against a compromise — a tenant reaching another tenant, or stealing a token that reads every customer's data — and those are unbounded losses. A saturated NIC is a bad afternoon. When a per-sandbox rate limit does ship, it will be a token bucket on both veth ends with a generous burst and a sustained rate sized against the host's uplink, plus a connection cap keyed on the /30, because those are the shapes the workloads and the failure modes actually have.

If you are building this, in order

1. Measure before you limit. Read the per-interface byte counters into whatever your metrics stack is and look at the distribution for a week. You will find the p99 tenant is doing something you did not know your product was for, and any number you pick without that data will be wrong in a direction you cannot predict.
2. Cap connections first. It is two rules, it protects against the failure with the widest blast radius, and it does not require you to have picked a bandwidth number yet.
3. Shape downloads before uploads. Downloads are the volume, the burstiness and the noisy-neighbour problem. Uploads are the abuse problem, and abuse deserves a per-workspace quota more than it deserves a per-sandbox rate.
4. Attach the qdisc to the end that transmits the direction you mean, then verify with a real transfer from inside the guest and confirm the overlimits counter moves. If it does not move, you shaped the other direction.
5. Give the burst allowance room. Users experience the burst; the sustained rate is for the tail of tenants who will never notice it either way.
6. Make the numbers per-plan and changeable on a live host without restarting anything. The first time an enterprise customer's legitimate workload hits your cap at 2am, you want a command, not a deploy.
7. Keep the meter regardless of whether you bill for it. Enforcement without observation is how you end up arguing with a customer about a number neither of you can produce.

The summary

Per-sandbox network isolation gives you a per-sandbox enforcement point for free: a veth pair, a namespace, and a unique /30 that doubles as a classification key. Attach a token bucket to the end that transmits the direction you care about — the namespace-side veth for guest uploads, the host-side one for downloads — and remember that ingress can only be policed, which is why the ifb redirect pattern exists and why owning both ends of a veth pair is a small luxury.

Prefer burst-then-shape over a flat cap, because real workloads are bursty and a flat cap wastes capacity nobody is using while doing nothing extra against a patient abuser. Size the sustained rate against the host's uplink and the worst simultaneous case, not against what a single sandbox deserves. And keep the two controls separate in your head: the shaper is what stops one tenant from ruining the machine, the meter and the quota are what stop one workspace from ruining the business.

Then go cap connection counts, which is the limit almost nobody sets and the one that fails hardest — quietly, at low traffic volumes, taking out other people's networking rather than your own.