How a Sandbox Scheduler Decides Where Your VM Lands

When you create a sandbox, something has to decide which physical machine it runs on. On a single-node setup that decision is trivial. As soon as you have three hosts it becomes the difference between a create that returns in 180ms and one that lands on a box already at load average 40 and takes several seconds — same code, same template, same API call.

I build PandaStack, a Firecracker microVM platform, so I've had to write this component and then watch it fail in production. This post is the honest version: the scoring function we run, why the obvious design has a bug that only shows up under burst, and what actually fixed it. If you're building your own placement layer, the failure mode described here is the one you'll hit, and it won't show up in any test that creates sandboxes one at a time.

What the scheduler is actually choosing between

Start with what makes this problem easier than general cluster scheduling. Our create path restores a baked snapshot, and every host that has the template seed restores it the same way — there's no "this host has a warm copy and that one doesn't" advantage to chase. That collapses a lot of complexity. The scheduler isn't trying to find a special host. It's trying to spread load evenly and avoid the hosts that can't take the work.

So the inputs are narrow. Each agent heartbeats every 10 seconds with its capacity: total and used CPU, total and used memory, and a flag for whether it supports streaming restore. The control plane keeps that in Postgres and caches reads for 30 seconds. From those numbers, placement is a scoring function over candidate hosts, highest score wins.

// The scoring function, essentially verbatim. freeCPU is in cores,
// freeMem in MiB — hence the /1024 to put both terms in comparable units.
score := float64(freeCPU)*0.6 + float64(freeMem)/1024.0*0.3

// A host that can stream guest memory on demand gets a fixed bonus:
// it can start a sandbox without downloading the whole memory image first.
if a.Capacity.StreamRestoreEnabled {
    score += streamBoost // 5.0
}

Two things are worth pointing out. First, CPU carries twice the weight of memory. That's deliberate — CPU contention is what users feel as slowness, while memory is mostly a hard admission question (either it fits or it doesn't). Second, the streaming bonus is a flat 5.0 rather than a multiplier, because it's a capability difference, not a load difference. It breaks ties toward hosts that can boot without a multi-gigabyte download, and gets swamped by the load terms once a host is genuinely busy — which is what you want.

Before scoring, candidates get filtered. Hosts whose heartbeat is more than 30 seconds stale are excluded entirely — a host that stopped reporting is assumed dead, not idle. Hosts without enough free memory for the request are excluded. And hosts whose lease has expired don't get considered, which is what stops a rebooting machine from silently accepting work.

The bug: capacity-aware scoring can't spread a burst

On 22 August 2026 we watched five create requests arrive inside one second. All five landed on the same host. That host went to load average 40 on 8 cores while a second, perfectly healthy agent sat at load 0.08 doing nothing.

Nothing was wrong with the scoring. The problem is that all five requests scored against identical inputs. Placement decisions don't show up in the capacity numbers the scheduler reads until the agent heartbeats (up to 10s) and the cache expires (up to 30s). So within that window, every request in a burst sees the same picture of the fleet, computes the same "best" host, and picks it. The scheduler wasn't making five decisions — it was making the same decision five times.

This is a general result and it's worth stating plainly: a capacity-aware scheduler cannot spread a burst that fits inside its own staleness window. No amount of better scoring fixes it, because the scoring inputs are stale by construction. Anyone doing least-loaded placement against periodically-refreshed metrics has this bug. It's invisible in sequential tests and obvious the first time real traffic arrives in a clump.

The fix: remember what you just placed

The fix is to stop relying entirely on observed capacity and start accounting for capacity you've promised but not yet observed. When the scheduler picks a host, it records a reservation against it. Subsequent scoring subtracts outstanding reservations from that host's free capacity. The second create in a burst now sees the first one's cost and moves on.

// Reserve while still holding the lock used for scoring + selection.
// Without that, N goroutines all read capacity before any of them writes
// a reservation, and you're back to the original race.
sort.Slice(candidates, func(i, j int) bool {
    return candidates[i].score > candidates[j].score
})
a := candidates[0].Agent
s.reserveLocked(a.ID, req, now)
return &a, nil

Two design details carry most of the weight here. The reservation is taken while holding the same lock that covers scoring and selection. If you score first, release, then reserve, concurrent requests interleave and you've rebuilt the bug with extra steps.

And reservations expire on a timer rather than being explicitly released. That sounds sloppy but it's the safe direction. If a create fails after placement, an explicit release might never run and the host would look permanently busier than it is. With a TTL, the worst case is that a host looks busy for a few seconds longer than necessary and the fleet spreads slightly wider than optimal. For this bug, erring toward spreading is always the right error to make.

What we deliberately haven't built

Worth being clear about the limits, because "our scheduler" can imply more machinery than exists.

• No random-of-K sampling. Some platforms sample a few hosts at random and pick the best of those, specifically because strict least-loaded on stale metrics hotspots. It's a good technique and it's on the list. At a handful of nodes, in-flight reservations solved the problem we actually had.
• No node-side admission semaphore. Our in-flight map lives in the API process, and running multiple API replicas means each has its own view. A cap on concurrent starts enforced by the agent itself would be the real cross-replica backstop.
• No live migration or rebalancing. Once a sandbox is placed it stays there. Capacity is recovered when sandboxes exit or hit their TTL, not by moving running work around.
• No automated scale-in. Adding capacity under load is a different problem from placement, and conflating them is how you get a fleet that evacuates a host while it's serving traffic.

What this means if you're using a sandbox platform

Mostly you shouldn't have to care — that's the point of the abstraction. But two things leak through and are worth knowing.

First, if you create sandboxes in a tight burst, you're exercising exactly the path described above. If your platform hasn't dealt with it, a burst of ten can land on one host and you'll see latency that looks random. It's worth testing: fire ten concurrent creates, then check whether your p99 looks like your p50.

import concurrent.futures, time
from pandastack import Sandbox

def timed_create(_):
    t0 = time.perf_counter()
    sbx = Sandbox.create(template="code-interpreter", ttl_seconds=300)
    sbx.exec("echo ready")            # first byte our own code produced
    dt = (time.perf_counter() - t0) * 1000
    sbx.kill()
    return dt

# Burst, not a loop. A sequential loop hides placement bugs completely,
# because each create is observed before the next one is scheduled.
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as ex:
    times = sorted(ex.map(timed_create, range(10)))

print(f"p50 {times[len(times)//2]:.0f}ms  max {times[-1]:.0f}ms")

Second, placement affects fork locality. Forking a sandbox on the host that already holds the parent's memory and disk is dramatically cheaper than doing it across hosts, so forks default to the parent's host. If you're building something fork-heavy, that default is doing real work for you and overriding it has a cost.

The takeaway

The interesting part of scheduling isn't the scoring function. Ours is two terms and a bonus, and it's been essentially unchanged since we wrote it. The interesting part is that every input to it is stale, and the moment requests arrive faster than your metrics refresh, correct scoring produces incorrect placement.

If you take one thing from this: test your scheduler with concurrent creates, not a loop. A sequential test will pass on a scheduler that piles every burst onto one machine, because sequential requests give the fleet time to notice each placement. We had that test. It was green the whole time.