Saying No Correctly: Admission Control for Sandbox Fleets

Every platform that hands out compute eventually has to say no. That is not the interesting part. The interesting part is that almost none of the capacity incidents I have worked through were actually about running out of memory. They were about saying yes to something that did not fit, or saying no to something that would have fit fine. The host was never the problem. The arithmetic in front of the host was.

I'm Ajay; I build PandaStack, a Firecracker microVM platform, so read this as opinionated and shaped by our own scar tissue. On PandaStack a create is a snapshot restore — 179ms at p50, around 203ms at p99 — which means the admission decision has to be arithmetic over a cached table and nothing else. A gate that takes 200ms to make up its mind has doubled the product's headline latency in order to protect a resource that was not under threat.

What an admission gate is actually deciding

Strip it down and there are two questions, and conflating them is where most designs go wrong. Placement asks: of the hosts that could take this, which one should? Admission asks: should anyone take this at all, right now? Placement is a preference and can be sloppy — pick a slightly worse host and you lose a little efficiency. Admission is a promise, and getting it wrong costs you either a dead guest or a lost customer.

It helps to be concrete about which resource is even in contention. On our agents it is not the network: each one pre-allocates 16,384 /30 subnets, along with the namespace, veth pair and tap device for each, so a network slot is effectively free and the pool depth is a warm-start optimisation rather than a concurrency cap. When the free list drains, allocation falls back to building a slot from scratch and costs a few hundred milliseconds. It does not refuse. Memory is what refuses. CPU is what degrades. Everything below is about memory, because memory is the one that kills things rather than slowing them down.

Committed vs working-set: the number you charge is the design

Here is the default that almost everyone ships first, because it is obviously safe. Sum the configured memory of every sandbox on the host. If that sum plus the new request exceeds physical RAM minus a headroom band, refuse. This is committed admission, and it has one enormous virtue: it cannot lose. You never promise a byte you do not have.

Now do the arithmetic on a 32 GiB host running 4 GiB guests. Seven. You get seven sandboxes, maybe eight if your headroom band is brave, and then the host starts refusing work. Meanwhile, if you go and look at what those seven guests are actually resident in — not what they were promised, what they have touched — the honest answer on a fleet of mostly-idle agent sandboxes is a small fraction of it. A guest that was handed 4 GiB and has touched 300 MB is consuming 300 MB. The other 3.7 GiB is a number in a config file.

That gap is not an accident of our design; it is a direct consequence of how snapshot-restored microVMs work. Firecracker fixes guest memory size when the snapshot is baked and cannot change it at restore, so every sandbox from a template inherits that template's size whether it needs it or not. And guest pages fault in lazily — on our streaming restore path they are pulled from object storage on demand, page by page, as the guest touches them. So 'configured' and 'resident' are two genuinely different quantities, and committed admission charges the first one while the host pays the second.

Committed accounting bills your host for memory nobody has touched, then refuses paying customers on the strength of that bill.

Working-set admission replaces the sum-of-promises with something closer to the truth: measured resident memory for live guests, plus a reserve for creates admitted but not yet measured, plus a fixed headroom band, checked against the same budget. PandaStack ships both modes and the fleet default is working-set. That is not a hedge — the mode is a per-host environment flag precisely because we wanted to run one host on the new arithmetic while the rest of the fleet stayed on the old one, and compare.

Which means overcommit is only half a design. The other half is what happens when the bet loses, and it has to be a policy you chose rather than the kernel OOM killer choosing for you. The OOM killer is an unsentimental capacity planner: it always arrives, it always frees exactly enough, and it has no opinion about which of your customers deserved to stay up. Anything you would prefer must run before it does, on a signal that leads the collapse — host memory pressure, not free-memory-hit-zero. And it must be able to slam the intake shut, which is the part that belongs here: when pressure crosses a line, the gate stops admitting, with hysteresis so it does not chatter open and closed at the boundary.

The reserve that makes measurement usable

Working-set admission has an obvious hole and you should find it before production does. Measured resident memory is a lagging quantity. A create that was admitted two hundred milliseconds ago has faulted in almost nothing, so it measures as roughly zero. Fire twenty creates at once and all twenty read the same near-zero usage, all twenty are admitted against the same free memory, and the host discovers the truth several seconds later at page-fault time, when there is nothing useful left to do about it.

The fix is a feed-forward term: charge every admitted create a reserve immediately, hold it until a measurement actually sees that guest, and expire it on a TTL in case the release never happens. Ours is a fraction of the baked size with a floor — a quarter, or 512 MB, whichever is larger — so a 4 GiB sandbox is admitted against 1 GiB. That is deliberately several times its typical real footprint and still far denser than charging it the whole 4 GiB.

Two details do the load-bearing work. First, the reserve must expire. A claim about the future that never times out will slowly convince your gate that the fleet is full, and it will do so silently — a leaked reservation from a failed create is subtracted from capacity forever. Export a gauge of outstanding reserved memory per host; on an idle fleet it must return to zero, and if it only ever climbs you have found tomorrow's phantom outage today.

Second — and this is the one people skip — the reserve has to be visible to every process that makes admission decisions. A ledger in one API replica's memory is correct for that replica and invisible to the other three, so a burst split across four schedulers gives you back a quarter-amplitude version of the original bug. It looks like the fix partially worked, which is the most expensive result available, because it does not force you to keep looking. The cleanest escape we found is to move the arithmetic to the resource: the agent computes its own admittable capacity, using the same formula its local gate enforces, and publishes that single number on its heartbeat. The control plane reads a number rather than re-deriving a formula. Two implementations of one rule is a bug with a delivery date.

// What the agent publishes on its heartbeat. The scheduler treats
// memory_mb_admittable as an opaque verdict, not an input to re-check.
{
  "agent_id": "agent-3f2a",
  "region": "us-central1",
  "heartbeat_at": "2026-09-08T11:04:22Z",
  "cpu_total": 16,
  "cpu_used": 4.2,
  "memory_mb_total": 32768,
  "memory_mb_used": 4180,

  "memory_admission": "working-set",
  "memory_mb_admittable": 25600,

  "natid_free": 16290,
  "stream_restore_enabled": true
}

The gate itself

Here is the shape, with the parts that matter and none of the parts that do not. Note what is absent: no synchronous probe of the host, no database round trip, no lock held across I/O. It is a scan over a cached table, because it sits on the hot path of a create that is supposed to complete in under two hundred milliseconds.

// admit.go -- placement + admission in one pass over cached heartbeats.
// Pure arithmetic on purpose: this runs inside a 179ms p50 create.

const (
	heartbeatStale = 30 * time.Second // older than this: not a candidate
	reserveFloorMB = 512
	reserveFrac    = 0.25
)

type Agent struct {
	ID          string
	HeartbeatAt time.Time

	CPUTotal, CPUUsed     float64
	MemMBTotal, MemMBUsed int64

	// Published by the agent, computed ONCE with the arithmetic its own
	// local gate enforces. The scheduler does not recompute this.
	MemAdmission    string // "committed" | "working-set"
	MemMBAdmittable int64

	StreamRestore bool
}

// reserveFor: what a create is charged between admission and the first
// measurement that actually sees it. Stateful workloads are charged in
// full and are never part of the bet.
func reserveFor(req Request) int64 {
	if req.Stateful {
		return req.MemMB
	}
	if r := int64(float64(req.MemMB) * reserveFrac); r > reserveFloorMB {
		return r
	}
	return reserveFloorMB
}

// fits: the gate. pendingMB is this fleet's outstanding in-flight
// reservations for the host, summed across every scheduler that shares
// the ledger -- or zero, if the agent already netted them out for you.
func fits(a Agent, req Request, pendingMB int64) bool {
	if a.MemAdmission == "working-set" && !req.Stateful {
		return a.MemMBAdmittable-pendingMB >= reserveFor(req)
	}
	// Committed arithmetic: promises, not measurements.
	return a.MemMBTotal-a.MemMBUsed-pendingMB >= req.MemMB
}

// score: pure load spreading. Every create is a snapshot restore that
// runs identically anywhere the seed exists, so there is nothing to
// bin-pack FOR -- prefer the emptiest host and keep the fleet flat.
func score(a Agent) float64 {
	freeCPU := a.CPUTotal - a.CPUUsed
	freeGiB := float64(a.MemMBTotal-a.MemMBUsed) / 1024
	s := 0.6*freeCPU + 0.3*freeGiB
	if a.StreamRestore {
		s += 5.0 // boots without pulling the whole memory file first
	}
	return s
}

func Pick(agents []Agent, req Request, led *Ledger) (string, func(), error) {
	now := time.Now()
	best, bestScore := "", math.Inf(-1)
	sawLive := false

	for _, a := range agents {
		if now.Sub(a.HeartbeatAt) > heartbeatStale {
			continue // silent host looks idle to a free-capacity scorer
		}
		sawLive = true
		if !fits(a, req, led.Pending(a.ID)) {
			continue
		}
		if s := score(a); s > bestScore {
			best, bestScore = a.ID, s
		}
	}

	if best == "" {
		// Distinguish the two nos. One is "come back in a moment";
		// the other is "your fleet is gone" and pages a human.
		if !sawLive {
			return "", nil, ErrNoHealthyAgents
		}
		return "", nil, ErrNoCapacity // retryable, and says so
	}
	return best, led.Reserve(best, reserveFor(req)), nil
}

The release function returned alongside the host is not decoration. Call it on every exit path — success, failure, panic recovery — and let the TTL sweep catch the ones where the process died between reserving and releasing. If you only release on the happy path, every failed create permanently shrinks a host in your gate's eyes, and the failure mode is a cluster that reports itself full while sitting idle.

Never overcommit anything with state in it

Notice the Stateful branch in reserveFor. On PandaStack, managed PostgreSQL databases are charged committed memory, always, no exceptions, regardless of which admission mode the host is running. They are excluded from the bet entirely.

The reasoning is one sentence long and I would put it on a wall. An OOM in a stateless sandbox is a retry; an OOM in a customer's Postgres is an incident. A code-interpreter guest that dies mid-execution costs someone a few seconds and a re-run, and the platform's own snapshot-restore path means the replacement is up in under a fifth of a second. A database that gets killed mid-write costs a recovery window, possibly a support conversation about durability, and permanently costs you some of the trust that made them put their data on your platform in the first place.

The asymmetry is worth stating precisely because density arguments are seductive and will absolutely come for your database tier. Someone will point out — correctly — that an idle Postgres VM is also mostly untouched memory, and that you could fit three times as many. They are right about the arithmetic and wrong about the risk. The whole basis for overcommit is that the consequence of losing the bet is cheap. For anything durable, the consequence is not cheap, so the premise does not hold and the density is not yours to take. Charge it in full, publish that you charge it in full, and let the stateless tier pay for the fleet's efficiency.

Rejecting correctly: a 503 is a tempo, not a verdict

Now the half that is genuinely under-discussed. Assume your gate is right and this create really does not fit at this instant. How you communicate that decides whether the user experiences a five-second wait or a broken product.

The rule: capacity rejection is a statement about right now, and everything downstream of it must treat it that way. On a fleet where creates complete in a couple hundred milliseconds and idle sandboxes hibernate continuously, 'full' is a condition with a half-life measured in seconds. Encoding it as a terminal state is a category error.

We learned this in the least pleasant way available. Our app-hosting layer wakes a hibernated app on demand and deploys it from a git push, and both paths ask the scheduler for a host. When the scheduler refused for capacity, the app went to status 'error'. Which is defensible for about half a second, until you notice what it means: the request that would have succeeded on the next attempt has instead written a permanent failure into the app's record, the health monitor now sees a broken app, and the customer opens the dashboard to a red badge on a service that is completely fine. A transient refusal was laundered into a durable fact. The fix was a distinct waiting_capacity state and a retry with backoff — the app parks, retries, and comes up. It never enters error on a 503.

Four things make a rejection honest. Use 503, not 500 and not 429 — 429 means the caller misbehaved, and a fleet at capacity is not the caller's fault. Send Retry-After with a real estimate, because a client that has to guess will guess wrong in whichever direction hurts you. Put a stable machine-readable code in the body so a client can distinguish 'no capacity' from 'no such template', which are the same status class and completely different problems. And never emit it from a cached negative, which is the next section.

The client half matters just as much, and it is the part a platform can actually ship for its users. Both our SDKs handle this so callers do not have to think about it, but the shape is simple enough to write yourself:

# Retry a capacity rejection properly: exponential backoff with FULL
# jitter, an honest deadline, and a hard distinction between "not now"
# and "not ever". The jitter is not optional -- without it, every client
# rejected in the same second returns in the same second.

import random
import time

import httpx

from pandastack import Sandbox

RETRYABLE = {503, 502, 504}
DEADLINE_S = 90.0      # your users' patience, expressed as a number
BASE_S = 0.25
CAP_S = 8.0


def create_with_backoff(template: str, ttl_seconds: int = 300) -> Sandbox:
    started = time.monotonic()
    attempt = 0

    while True:
        try:
            return Sandbox.create(template=template, ttl_seconds=ttl_seconds)
        except httpx.HTTPStatusError as exc:
            status = exc.response.status_code

            # 4xx that is not 429 is your bug, not the fleet's. Retrying a
            # bad template name a hundred times just makes the logs worse.
            if status not in RETRYABLE:
                raise

            elapsed = time.monotonic() - started
            if elapsed >= DEADLINE_S:
                # Out of patience. Park the work as WAITING, not FAILED --
                # the distinction is the whole point of this function.
                raise CapacityUnavailable(
                    f"no capacity after {elapsed:.1f}s"
                ) from exc

            # Honour the server if it told us; otherwise back off ourselves.
            hinted = exc.response.headers.get("Retry-After")
            if hinted and hinted.isdigit():
                delay = float(hinted)
            else:
                window = min(CAP_S, BASE_S * (2 ** attempt))
                delay = random.uniform(0, window)   # full jitter

            time.sleep(min(delay, DEADLINE_S - elapsed))
            attempt += 1


class CapacityUnavailable(Exception):
    """Transient. Retry later. Do NOT write this to a status column."""

Full jitter rather than the usual exponential-plus-a-little-noise is a deliberate choice. Deterministic backoff synchronises the herd: fifty clients rejected at the same moment all wait exactly 500ms and return as one wave, get rejected again, wait a second, return as one wave. Sampling uniformly from zero to the window smears them across it, and the capacity that frees up gets consumed smoothly instead of in a thundering retry.

Your capacity view is always a little bit in the past

Heartbeats arrive on an interval and caches hold results for a window, so the numbers your gate reads describe a host as it was some seconds ago. There is no fixing this — it is the shape of distributed state — but there are two rules that make it survivable, and they point in opposite directions, which is why people get one right and the other wrong.

Rule one: exclude stale hosts entirely. A host that has stopped heartbeating looks, to a free-capacity scorer, like an increasingly attractive destination, because its last reported usage stopped rising while everyone else's kept climbing. Without a staleness cutoff, a crashed machine becomes the highest-scoring host in your fleet and you place every create onto a corpse. Least-loaded scoring is drawn to dead things like a moth to a very quiet lamp. Thirty seconds against a ten-second heartbeat is a reasonable cutoff.

Rule two, and this is the hard-won one: never trust a cache for a negative. We shipped a scheduler cache that could not distinguish 'this host reported unhealthy' from 'this host was not in the result set when I last read'. A cached absence became a cached death, and a meaningful slice of placement attempts returned capacity errors against hosts that were healthy and heartbeating perfectly. A cache is a record of what you saw. It is not evidence about what you did not see.

So: cache positive facts freely, and before you return a capacity error to a customer, re-read the source. The cost is a fresh query on the rare path where you were about to fail anyway, which is the cheapest possible place to spend it. Separate your timeouts too — the interval at which you stop trusting a heartbeat for scheduling should be much shorter than the one at which you declare a host dead and start evacuating, because the first decision is a hint and the second is irreversible.

Spreading vs packing, honestly

The scorer above spreads: it prefers the emptiest host. That is a real choice with a real cost, and the opposite choice is defensible in other architectures, so it is worth writing down why we land where we do rather than pretending it is obvious.

Bin packing — filling one host before touching the next — is what you want when a host is expensive to keep alive and cheap to drain, because packing tightly is what lets you turn machines off. It is also what you want when placement has genuine locality value: workloads that share a page cache, or a warm image, or a dataset already on that machine's disk. Kubernetes has a whole scheduling vocabulary for this and it earns its keep.

None of those conditions hold for us, and the reason is the boot path. Every create is a restore of a baked snapshot, so any host holding the seed produces an identical result at the same latency — there is no warm state on host A that makes host A better than host B. Locality has nothing to purchase. Meanwhile the cost of packing is concrete: a full host is one where a spike from any tenant lands on neighbours, where a snapshot restore competes for page cache with everything else already running, and where a single machine going away takes a disproportionate slice of the fleet's work with it. Spreading gives every guest more thermal room for the same fleet-wide density, and the arithmetic is simpler to reason about at three in the morning.

The honest caveat: spreading and scale-to-zero are in tension at the fleet level. If you spread perfectly, no host is ever empty enough to retire, and you pay for capacity you are not using. We deal with that at a different layer than the scorer, and I would not pretend the tension is fully resolved. If your economics depend on turning hosts off aggressively, weight your scorer toward packing and accept the noisier-neighbour outcome — that is a legitimate trade, just make it on purpose.

Four admission regimes, compared honestly

These are not four points on a quality axis. Each is right for some fleet, and the top one is right more often than its reputation suggests.

• No admission control — What it gets right: nothing to build, nothing to leak, no phantom refusals, and the resource itself becomes the enforcer. On a small internal fleet with cooperative users this is genuinely fine and I have shipped it deliberately. What it costs: the first bad afternoon, when the OOM killer picks your victims for you, and it picks the biggest process rather than the least important one. There is no backpressure signal, so clients cannot slow down even if they wanted to.
• Committed-only admission — What it gets right: it cannot lose the bet, the arithmetic is trivial to explain to a customer, and it is the only defensible model for stateful workloads. Provisioning maths is a spreadsheet. What it costs: enormous idle waste. A 32 GiB host stops at roughly seven 4 GiB sandboxes while the overwhelming majority of that RAM has never been touched, which shows up as capacity errors on hosts that are, physically, mostly empty — and as a hardware bill three or four times larger than the workload justifies.
• Working-set admission with in-flight reservations — What it gets right: it charges what workloads actually use and reclaims the headroom committed accounting wastes, while the reserve covers the blind window before a new guest can be measured. This is our fleet default. What it costs: you are now overcommitted, which means you need a pressure signal, a policy for losing the bet, a reserve TTL that will silently shrink your fleet if you get it wrong, and shared visibility of reservations across every scheduler process. Four new things that can be subtly broken.
• Hard per-tenant quotas — What it gets right: it is the only one that bounds a single tenant's blast radius, it makes cost predictable per customer, and it fails in a way you can put in a docs page. It composes with all of the above rather than replacing them. What it costs: it says nothing about whether a host can take this create right now — a tenant well inside quota can still land on a full machine. Quota is a fairness and billing mechanism; treat it as a capacity gate and you will 403 someone who had room.

In practice you want the third and the fourth together, plus a per-host circuit breaker underneath both: a semaphore bounding concurrent VM starts on the agent itself, so that when the scheduler is the thing that is broken, the resource can still defend itself. That last layer is the only one that keeps working during an incident whose cause is your control plane.

The four numbers that tell you whether it works

1. Refusal rate split by reason, and never aggregated. 'No capacity', 'no healthy agents', 'tenant over quota' and 'host starts saturated' are four different pages with four different responses, and a single counter called capacity_errors_total will hide three of them behind whichever is loudest.
2. Ratio of resident to committed memory, fleet-wide. This is the number that tells you what committed accounting is costing you, and it is the number you show the person who has to approve the overcommit. If it is close to 1, do not overcommit — your workload does not have the idleness the bet requires.
3. Outstanding reservations per host, which must oscillate and return to zero on an idle fleet. Monotonic growth is a leaked release, and it will present as capacity errors on an empty cluster — a failure that looks nothing like its cause.
4. Time from first 503 to eventual success, per request rather than per attempt. This is the only metric that measures what the user felt. A fleet that refuses 5% of creates and satisfies all of them within two seconds is healthy; one that refuses 0.5% and leaves them stuck for four minutes is not, and a plain error-rate dashboard scores the second one better.

The summary

Admission control is the arithmetic your platform does in the moment before it commits. Charge committed memory and you refuse paying work on hosts that are mostly untouched. Charge measured working set and you reclaim that headroom, at the price of a bet you now have to hedge with a reserve, a pressure signal and a plan for losing. Charge stateful workloads committed regardless, because the premise of overcommit — that losing is cheap — is false the moment a customer's data is involved.

And when the answer really is no, say it in a way the system can recover from. A 503 with Retry-After and a stable error code, a client that backs off with full jitter, and a state machine that parks work as waiting rather than failed. The difference between those two words is the difference between a five-second delay nobody notices and a support ticket about an outage that ended before anyone read it.