How Many Sandboxes Fit on One Machine?

"How many sandboxes can one machine run?" is the question that decides your unit economics, and it has no clean answer. I've spent a fair amount of time trying to derive one for PandaStack's fleet and I want to write down what I actually learned, including the parts where the intuitive approach is wrong.

The short version: there are four candidate ceilings, only one of them binds in practice, and the host-level metric you'd naturally use to measure it doesn't tell you the truth on a microVM host. If you're sizing a fleet, the useful work is figuring out which ceiling you're actually near.

The four ceilings, and which one matters

Start by enumerating what could stop you from packing another VM onto a host.

1. Network identity. Every sandbox needs an isolated network namespace, a veth pair, a tap device, and NAT rules. We pre-allocate these from a /16 carved into /30 subnets, which gives a hard ceiling of 16,384 per host.
2. Disk. Each sandbox gets a copy-on-write clone of a template rootfs. Reflink makes the clone nearly free at creation, but writes diverge and consume real blocks over time.
3. CPU. Guests are given generous vCPU counts for burst, with the host kernel's scheduler sharing cores under contention.
4. Memory. Each running guest holds a memory footprint that the host must actually back with pages.

The network ceiling sounds like the constraint and never is — 16,384 sandboxes on one machine is not a scenario anyone reaches. Disk matters over time but is manageable with copy-on-write and reaping. CPU is genuinely shared: our guests get 8 vCPUs each as burst capacity, and cgroup weights split cores fairly when everyone wants them at once. Nobody's workload is CPU-saturated continuously, so oversubscribing CPU is fine and expected.

Memory is what binds. It's the one resource you cannot meaningfully oversubscribe without risking the thing you must never do on a multi-tenant host, which is run out and let the OOM killer choose a victim.

The constraint people don't expect: memory is baked in

Here's the detail that surprises most people building on snapshot-restore. Firecracker cannot change a VM's memory size at restore. The guest's RAM is fixed at the moment the snapshot was taken, so a template's memory is a property of the template, not of the request that creates a sandbox from it.

That means "give this one 512 MiB and that one 8 GiB" isn't a per-request knob when you're restoring from a baked snapshot — it's a per-template decision made at bake time. Ask for a different size and you get the template's size. It's the right trade for boot latency, but it makes capacity planning coarser than a container platform, where memory limits really are per-instance.

The practical consequence is that raising a template's memory raises the floor for every sandbox created from it. Our base app template sits at 4 GiB not because most apps need it, but because JavaScript build tooling reliably runs out of memory below that. Every app pays for the worst-case build.

Why your host memory metric is lying

This is the part I'd most want someone else to know before they build an autoscaler.

Two techniques that make microVM fleets fast both distort host-level memory accounting, in opposite directions.

Hugepages register as used, permanently

Backing guest memory with 2 MiB hugepages cuts page faults on restore by a large factor — one fault covers 2 MiB instead of 4 KiB. But statically reserved hugepages show up as used memory the moment they're reserved, whether or not a guest is touching them, and they are not reclaimable by the kernel for anything else. A host with a big static reservation looks nearly full to any standard memory metric while sitting completely idle.

We measured this on our own fleet and found gigabytes per host committed to reservations nothing was using. If you scale out on host memory percentage, hugepage reservations will make an idle fleet scale itself out forever.

Demand-paged memory registers as free, until it isn't

The opposite distortion comes from lazy restore. When guest memory is served on demand — pages faulted in as the guest touches them — a freshly restored sandbox has barely any resident memory. Most of a guest's address space is never touched, and a large fraction of what is touched is zero-fill that never needs to come from anywhere.

So a host running many lazily-restored sandboxes genuinely looks underutilized, right up until several guests simultaneously touch a lot of memory and resident usage climbs fast. The metric was accurate; it just wasn't predictive.

Why flat caps beat computed capacity

The instinct is to compute available capacity precisely and admit work up to it. Having looked at how other people in this space solve it, the more common answer is blunter and probably better: enforce a flat cap on sandboxes per host, plus a separate small cap on how many can be *starting* at once, and tune both by observation.

The second cap is the interesting one. Booting a VM is far more expensive than running an idle one — it's disk reads, snapshot loading, and a page-fault storm all at once. A host can comfortably hold many running sandboxes while being able to start only a few simultaneously. Capping concurrent starts separately from total residents is the distinction that keeps a burst from making a host unresponsive.

The pattern that pairs with it: when a host refuses a start because it's at its concurrency cap, the scheduler should treat that as "try another host" rather than as a failure. It shouldn't burn a retry budget or mark the host unhealthy, because nothing is wrong — it's busy for a moment.

Checking your own numbers

If you run your own fleet, the two numbers worth watching are committed memory versus host memory, and boot latency at p99. Committed memory tells you how close you are to the ceiling that matters. Boot p99 tells you whether you're already past the point where starts contend.

# Boot duration and lazy-paging counters from an agent's metrics endpoint.
# If uffd counters are zero, restores are reading a local memory image and
# host memory usage will track resident guests closely. If they're high,
# memory is arriving on demand and host usage lags commitment.
curl -s http://localhost:9100/metrics \
  | grep -E 'pandastack_(sandbox_boot|uffd)'

And if you're a customer of a platform rather than an operator of one, the observable proxy is simple: create sandboxes concurrently and watch whether latency degrades. A platform near its per-host ceiling shows it as boot times that grow under burst, long before it shows it as errors.

The takeaway

There's no honest single number for sandboxes per host, but there is an honest method. Find which resource binds — on a microVM fleet it's almost always memory, and it's coarser than you expect because snapshot restore bakes guest memory into the template. Then account for what you've committed rather than trusting what the host reports, because both hugepages and demand paging make host memory metrics misleading in opposite directions.

Flat caps, tuned by observation, with a separate limit on concurrent starts, will get you further than a clever capacity model built on numbers that don't mean what they appear to mean.