How to Benchmark Sandbox Cold Start Honestly

Every sandbox vendor publishes a cold-start number. Mine included — you'll find p50 179ms and p99 203ms on our own pages, and I'll defend those numbers later in this post. But here's the uncomfortable thing about the whole category: almost none of us are measuring the same event. One number is the time for the control plane to return a 201. Another is the hypervisor's snapshot-restore step in isolation. Another is a full round trip on a machine sitting in the same rack, warmed up, running one request at a time, with the failures quietly excluded. All four are honest measurements of something. None of them are comparable to each other, and only one of them resembles what your users feel.

I'm Ajay; I build PandaStack, a Firecracker microVM platform, so I have an obvious commercial interest in you believing my latency numbers. This post deliberately doesn't ask you to. It's a method: how to define the stopwatch, which stages to split out, which statistic to care about, and the traps that will make your benchmark measure your laptop's Wi-Fi instead of anyone's platform. Run it against us, run it against everyone else, publish what you find. A benchmark you ran yourself, on your workload, from your region, is worth more than every vendor blog post in this space — including this one.

Step one: define where the stopwatch starts and stops

Almost all cold-start confusion collapses into one question: what are the two endpoints? Pick them wrong and everything downstream is noise. My argument for the honest pair is this. The clock starts when your process hands the create request to the network — not when a connection is established, not when a pool hands you a worker, not after some setup you excluded because it felt like overhead. And the clock stops when your code produces its first byte inside the sandbox.

That second endpoint is the one that gets fudged. A 201 from a control plane means a row was written and a scheduler picked a host. It does not mean a kernel booted, it does not mean the guest network is up, and it emphatically does not mean anything you wrote can run. On some platforms the 201 arrives well before the sandbox is usable, and the remaining wait is charged to whatever your code does next — which is to say, to your user, invisibly, in a place your benchmark isn't looking. The only endpoint that can't be gamed is a byte your own code produced. Print a `1` and read it back.

import time
from pandastack import Sandbox

# The measurement that gets published:
t0 = time.perf_counter()
sbx = Sandbox.create(template="code-interpreter", ttl_seconds=900)
print("create returned in", (time.perf_counter() - t0) * 1000, "ms")

# The measurement your users actually experience. Same start, different
# finish line: a byte that OUR code produced inside the guest. Everything
# between the 201 and this line is latency someone is paying for, whether
# or not it appears on a vendor's pricing page.
t0 = time.perf_counter()
sbx = Sandbox.create(template="code-interpreter", ttl_seconds=900)
r = sbx.exec("python3 -c 'print(1)'")
print("usable in", (time.perf_counter() - t0) * 1000, "ms", r.stdout.strip())
sbx.kill()

If the gap between those two prints is small, the vendor's published number is roughly honest. If it's large, you've just learned the most important thing your benchmark will tell you, and you learned it in fifteen lines.

Step two: split the interval into stages

A single end-to-end number tells you whether you're happy. It never tells you what to do about it. Split the interval into stages and the answer usually falls out, because in my experience the dominant stage is rarely the one people are optimizing. These are the six worth instrumenting, roughly in order:

• Control-plane scheduling — auth, quota checks, a database write, and picking a host. Pure software, usually tens of milliseconds, and the first thing to get pathological under burst because it's where the locks and the shared tables live.
• VM process spawn — fork and exec the VMM, set up its jailer, hand it a config. Small and stable, but it's where you'll find surprises like a cold page cache or an overloaded host.
• Snapshot load and restore, or full boot — the actual hypervisor work. On PandaStack this step is around 49ms of the total; a first-ever boot of a template, with no snapshot to restore from, is about 3 seconds. Those are two completely different events and mixing them in one distribution is the single most common way benchmarks lie.
• Guest network up — the tap device, the namespace, the routes, DHCP or a baked-in address. Cheap when it's pre-allocated, brutal when it isn't: creating a network namespace and its iptables rules from scratch is on the order of 100ms, which can dwarf the restore itself.
• First TCP accept — the moment something inside the guest is listening and completes a handshake. This is the boundary between 'the VM exists' and 'the VM is reachable', and it hides a surprising amount of time in service startup ordering.
• Language runtime import — the interpreter starting and your imports resolving. Python importing a scientific stack can cost more than the entire VM lifecycle above it. If you don't split this out, you will attribute your own dependency graph to your vendor.

You won't get vendor-side stage breakdowns for a platform you don't operate, and that's fine. From the outside you can still cleanly separate three of them: time-to-201 (control plane plus most of the machinery), time-from-201-to-first-byte (boot, network, accept), and time-for-a-heavier-import (your runtime tax). Three buckets is enough to know who to blame.

Step three: agree on what "cold" means

"Cold start" is doing a lot of unpaid labor as a term. There are at least four distinct events wearing that name, and a vendor is free to publish whichever one flatters them, because there is no shared definition to violate.

1. First-ever boot: no snapshot exists, the guest kernel boots from scratch, userspace initializes. Seconds, not milliseconds. On PandaStack this is roughly 3s and it happens once per template.
2. Snapshot restore: a pre-baked memory and disk image is restored on demand. This is what most modern microVM platforms mean by 'cold start' and it's legitimately fast — 179ms p50 / 203ms p99 end-to-end in our case, of which the restore step is about 49ms.
3. Fork or clone from a running parent: copy-on-write from an existing sandbox's state. Fundamentally different work again — 400-750ms same-host for us, 1.2-3.5s cross-host, because crossing hosts means moving bytes.
4. Secretly warm: a pool of already-running sandboxes was pre-created before the timer started, and 'create' is really 'check out'. This can be a completely legitimate product design. It is not a cold start, and if a benchmark doesn't say which one it measured, assume the flattering one.

Step four: p50 is the least interesting number you will produce

Median latency is the number everyone publishes and the number nobody experiences. Consider an agent loop that creates a sandbox on every tool call — twenty calls in a task is unremarkable. At p99 = 1 in 100, a twenty-step task has roughly an 18% chance of hitting at least one p99 event. Run a thousand tasks a day and the tail isn't an edge case, it's a daily standup topic. Your users don't live at p50; they live at the worst thing that happened to them today, and they remember it.

So report p50, p90, p95, p99, and max — and report the sample count next to them, because a p99 computed from 50 samples is not a p99, it's the second-worst number you happened to see. Two hundred attempts is a floor for a p99 you can talk about with a straight face; a thousand is better. And always publish the failure count in the same breath, which brings us to the two ways percentiles get quietly laundered.

Trap: counting only the successes

A request that times out at 30 seconds did not take zero milliseconds and it did not fail to happen. But in most naive harnesses it simply isn't in the array, which means the slowest requests systematically remove themselves from the statistics. Under load this inverts the result completely: a platform that starts shedding load at high concurrency can post a *better* p99 than one that serves everything slowly, because its worst requests vanished. The fix is boring — score each failure at the deadline value (or higher) and include it in the distribution, then report a separate success rate. A benchmark with a latency table and no error column is not a benchmark, it's a highlight reel.

Trap: coordinated omission

This one is subtle and it has ruined more load tests than any other single mistake. If your harness sends a request, waits for the response, then sends the next one, a slow response doesn't just get recorded as slow — it also delays every subsequent request, so those requests never get sent during the bad period. You measure the system's behavior mostly when it's healthy, and the tail evaporates. The name for this is coordinated omission, and the standard fix is to hold a fixed arrival schedule: decide in advance that request number 47 is due at t+2.35s, and measure its latency from that intended time, not from the moment a worker thread was finally free to send it. Queueing on your side is still latency your user would feel.

The rest of the traps

• Measuring from the wrong continent. A benchmark run from a laptop in Lisbon against a us-east-1 API measures the Atlantic. Round-trip time to the control plane can exceed the entire boot you're trying to measure. Run the harness from a cloud VM in the same region as the API, and say which region in your writeup.
• Benchmarking a free tier. Free tiers are frequently rate-limited, capacity-deprioritized, or pinned to smaller shapes than paid usage. This is a reasonable business decision and a terrible benchmark substrate. Pay for the thing you're evaluating, or state loudly that you didn't.
• One sequential loop. A `for` loop of 100 creates measures a system with zero contention, which is the one condition your production traffic will never be in. It is a useful baseline and a useless headline.
• Letting your dependency install dominate. If your probe is `pip install pandas && python analyze.py`, you have benchmarked PyPI. Bake dependencies into the template snapshot, then measure the runtime import separately as its own stage.
• Your own machine is the bottleneck. Two hundred threads on a four-core laptop means your harness is queueing on itself, and you will attribute your own scheduler to the vendor. Check load average during the run; if the harness is saturated, the numbers are fiction.
• Comparing across templates. A 4 GiB image with a browser baked in and a 256 MiB minimal image are not the same benchmark. Fix the guest size and the installed software across every platform you test, or note honestly that you couldn't.
• One run, one day. Cloud hosts have noisy neighbors, deploy windows, and regional bad afternoons. Run the same suite at three different times before you believe a delta smaller than about 20%.

The harness

Here's the whole thing. It records per-stage timestamps, treats failures as data rather than as absences, supports both a genuine burst and a fixed-arrival-rate open loop, and measures latency from the intended send time. It's written against the PandaStack SDK because that's the one I can test; the shape ports to any vendor by swapping the three lines that create, exec, and kill.

"""cold_start_bench.py -- an honest sandbox cold-start harness.

Measures request-sent -> your-code's-first-byte, splits the stages, counts
failures as failures, and drives a burst instead of a polite loop.
Run it from a machine in the same region as the API, and not a busy one.
"""
import argparse
import json
import time
from concurrent.futures import ThreadPoolExecutor

from pandastack import Sandbox

# Stage probe: the cheapest round trip that proves OUR code ran. If this
# returns, the guest booted, the network came up, exec was accepted, and a
# process of ours produced a byte. That -- not the 201 -- is "usable".
PROBE = "python3 -c 'print(1)'"

# Runtime-import probe: the tax your users pay on their first real call.
# Swap in whatever your agent actually imports.
HEAVY = "python3 -c 'import json, urllib.request; print(1)'"


def ms(a, b):
    return None if (a is None or b is None) else round((b - a) * 1000.0, 2)


def one_attempt(template, deadline_s, intended=None):
    """One create -> usable measurement. Never raises; failures are data."""
    t_intended = intended if intended is not None else time.perf_counter()
    t_send = time.perf_counter()
    t_created = t_usable = t_imported = None
    err = None
    sbx = None
    try:
        sbx = Sandbox.create(template=template, ttl_seconds=300)
        t_created = time.perf_counter()      # control plane returned 201

        r = sbx.exec(PROBE, timeout_seconds=deadline_s)
        t_usable = time.perf_counter()       # first byte of OUR code
        if r.exit_code != 0:
            err = "probe_exit_" + str(r.exit_code)
        else:
            sbx.exec(HEAVY, timeout_seconds=deadline_s)
            t_imported = time.perf_counter()
    except Exception as exc:                 # timeouts, 429s, 503s, resets
        err = type(exc).__name__ + ": " + str(exc)[:160]
    finally:
        if sbx is not None:
            try:
                sbx.kill()
            except Exception:
                pass

    return {
        "ok": err is None,
        "error": err,
        # Measured from the INTENDED arrival time, not from when a worker
        # thread happened to be free. That is the coordinated-omission fix:
        # queueing on our side is still latency a user would feel.
        "usable_ms": ms(t_intended, t_usable),
        "queued_ms": ms(t_intended, t_send),
        "stage_control_plane_ms": ms(t_send, t_created),
        "stage_boot_to_first_exec_ms": ms(t_created, t_usable),
        "stage_runtime_import_ms": ms(t_usable, t_imported),
    }


def pct(values, p):
    if not values:
        return None
    xs = sorted(values)
    k = (len(xs) - 1) * (p / 100.0)
    lo = int(k)
    hi = min(lo + 1, len(xs) - 1)
    return round(xs[lo] + (xs[hi] - xs[lo]) * (k - lo), 2)


def block(xs):
    return {"n": len(xs), "p50": pct(xs, 50), "p90": pct(xs, 90),
            "p95": pct(xs, 95), "p99": pct(xs, 99), "max": pct(xs, 100)}


def summarise(records, deadline_s):
    ok = [r for r in records if r["ok"]]
    bad = [r for r in records if not r["ok"]]
    good_ms = [r["usable_ms"] for r in ok]

    # A request that timed out did not take zero time and it did not fail to
    # happen. Score it at the deadline so it lands in the tail instead of
    # quietly improving the average by leaving the dataset.
    all_ms = good_ms + [deadline_s * 1000.0] * len(bad)

    stages = ("queued_ms", "stage_control_plane_ms",
              "stage_boot_to_first_exec_ms", "stage_runtime_import_ms")

    return {
        "attempts": len(records),
        "failures": len(bad),
        "success_rate": round(len(ok) / max(len(records), 1), 4),
        "usable_ms_successes_only": block(good_ms),
        "usable_ms_with_failures_at_deadline": block(all_ms),
        "stages_p50_ms": {
            k: pct([r[k] for r in ok if r[k] is not None], 50) for k in stages
        },
        "error_counts": {
            e: sum(1 for r in bad if r["error"] == e)
            for e in sorted({r["error"] for r in bad})
        },
    }


def run(template, n, concurrency, rate, deadline_s):
    t0 = time.perf_counter()
    with ThreadPoolExecutor(max_workers=concurrency) as pool:
        futures = []
        for i in range(n):
            # rate <= 0 means "all at once": a genuine burst, which is what a
            # traffic spike looks like. Otherwise hold a fixed arrival
            # schedule and never let a slow response delay the next send.
            intended = t0 if rate <= 0 else t0 + i / float(rate)
            gap = intended - time.perf_counter()
            if gap > 0:
                time.sleep(gap)
            futures.append(
                pool.submit(one_attempt, template, deadline_s, intended))
        return [f.result() for f in futures]


if __name__ == "__main__":
    ap = argparse.ArgumentParser()
    ap.add_argument("--template", default="code-interpreter")
    ap.add_argument("--n", type=int, default=200)
    ap.add_argument("--concurrency", type=int, default=50)
    ap.add_argument("--rate", type=float, default=0.0)   # 0 = pure burst
    ap.add_argument("--deadline", type=float, default=30.0)
    ap.add_argument("--raw", default="raw.json")
    a = ap.parse_args()

    recs = run(a.template, a.n, a.concurrency, a.rate, a.deadline)
    with open(a.raw, "w") as fh:
        json.dump(recs, fh, indent=2)
    print(json.dumps(summarise(recs, a.deadline), indent=2))

Running it, and reading the tail

Three runs, in this order, every time. A sequential baseline tells you the floor. A burst tells you what a traffic spike feels like. An open loop at a fixed rate tells you whether the tail is contention or collapse.

# 0. Run this from a cloud VM in the SAME region as the API under test.
#    A benchmark run from your laptop measures your ISP.
python3 -m venv .venv && . .venv/bin/activate
pip install pandastack
export PANDASTACK_API_KEY="..."      # a PAID key, not a free tier

# 1. Warm the path once and throw the results away. A first-ever template
#    boot is a different event from a snapshot restore; mixing them
#    poisons every percentile downstream.
python3 cold_start_bench.py --n 5 --concurrency 1 --raw warmup.json > /dev/null

# 2. Sequential baseline: one polite user, zero contention. The floor.
python3 cold_start_bench.py --n 100 --concurrency 1 \
  --raw seq.json | tee seq.summary.json

# 3. Burst: 200 creates, 50 in flight. This is the number that matters,
#    because real traffic does not arrive one at a time.
python3 cold_start_bench.py --n 200 --concurrency 50 \
  --raw burst.json | tee burst.summary.json

# 4. Open loop at a fixed 20 creates/sec. The arrival schedule is held even
#    when the platform slows down -- the coordinated-omission control.
python3 cold_start_bench.py --n 600 --concurrency 200 --rate 20 \
  --raw rate.json | tee rate.summary.json

# 5. Read the tail, not the headline. Note that p50/p99 here INCLUDE the
#    failures, scored at the deadline.
for f in seq burst rate; do
  echo "== $f"
  jq '{p50: .usable_ms_with_failures_at_deadline.p50,
       p99: .usable_ms_with_failures_at_deadline.p99,
       failures, attempts, success_rate}' "$f.summary.json"
done

# 6. Where did the time actually go?
jq '.stages_p50_ms' burst.summary.json

# 7. Did YOUR machine become the bottleneck? If load average is above core
#    count during the run, the numbers above are fiction.
uptime; nproc

Now read it. The ratio of burst p99 to sequential p50 is the single most informative number in the whole exercise: it tells you what happens to a user who arrives at the same moment as everyone else. Under 2x is excellent. Somewhere around 5x means there's a queue you can't see. Above 10x, or with failures appearing, you've found a hard capacity edge — and the honest thing to do at that point is not to declare a winner but to note the concurrency at which it appeared, since it's a property of the account, region, and hour as much as the platform.

Then look at the stage split. If `stage_control_plane_ms` grows under burst while `stage_boot_to_first_exec_ms` stays flat, the bottleneck is scheduling and database contention, not the hypervisor — which incidentally is the most common finding, and the reason "our VM boots in N milliseconds" is such a weak claim on its own. If `stage_runtime_import_ms` dominates everything, congratulations: your cold start problem is your own import graph, and no vendor can fix it for you.

What a headline number covers vs. what yours will

Side by side, the difference between a published figure and a measurement you can act on:

• Stopwatch endpoints — Typical headline: request sent to control-plane 201, or the hypervisor's restore step in isolation. Honest harness: request sent to first byte of your code inside the guest, with the 201 recorded as an intermediate split.
• Warm vs cold — Typical headline: unspecified, and often a restore or a pre-warmed pool checkout. Honest harness: first-ever boot, snapshot restore, and fork measured as three separate distributions, each labelled.
• Concurrency — Typical headline: one request at a time, on an idle account. Honest harness: a sequential floor plus a burst plus a fixed-rate open loop, with the concurrency stated next to every number.
• Failures — Typical headline: excluded, so the slowest requests remove themselves from the tail. Honest harness: scored at the deadline and included, with a separate success rate and an error histogram.
• Statistic reported — Typical headline: p50, sample size unstated. Honest harness: p50/p90/p95/p99/max with n, because a p99 from 50 samples is just the second-worst sample.
• Network position — Typical headline: unstated, often same-region or same-rack. Honest harness: same region as the API, stated explicitly, with the client machine's load checked during the run.
• Dependency setup — Typical headline: folded into the number or silently excluded. Honest harness: baked into the template, then measured separately as a runtime-import stage.
• Reproducibility — Typical headline: a blog post. Honest harness: a script, raw per-attempt JSON, the region, the template, the date, and the account tier.

Holding my own numbers to it

It'd be cheap to write all that and not apply it. So: PandaStack's published create latency is p50 179ms, p99 203ms. That's a snapshot restore — a pre-baked Firecracker snapshot loaded on demand, not a warm pool and not a first boot. The restore step itself is about 49ms of it; the rest is control plane, network setup, and the readiness probe. A first-ever boot of a template, before any snapshot exists, is around 3 seconds, and it happens once. Forks are separate again: 400-750ms same-host, 1.2-3.5s cross-host, because cross-host means moving memory over a network and physics gets a vote.

The honest caveats: those are measured from the same region, on a paid path, with the template's dependencies already baked in, and the p99 comes from a healthy fleet rather than one mid-deploy. Your burst p99 on a bad afternoon will be worse, and that gap is real information rather than something I want to argue you out of. Which is exactly why I'd rather hand you the harness than the number — if you run it and the ratio looks bad, that's a bug report with a stack trace attached.

When this is overkill

Most teams don't need any of this. If your sandbox creates happen once per user session and a human is reading a page while it warms up, the difference between 200ms and 2s is invisible and you should go spend the afternoon on something your users can feel. If you're prototyping, run five creates, eyeball the wall clock, and move on. The full harness earns its keep in exactly two situations: you're choosing between vendors and the decision is expensive to reverse, or you have an agent loop creating sandboxes dozens of times per task where the tail compounds into something users describe as "it feels slow" without being able to point at a single request.

And be honest about what the method still doesn't give you. It measures create latency, which is one axis among several — it says nothing about how the platform behaves at hour six of a long-lived sandbox, whether the filesystem is fast, whether exec throughput holds up under streaming, or what happens when a host dies mid-session. It's also a snapshot in time: everyone in this category ships continuously, so a benchmark is a photograph, not a portrait, and any result older than a quarter should be re-run before you cite it. Cold start is the number the industry chose to compete on because it's easy to put on a landing page. Measure it properly, then go measure the boring things that actually decide whether the thing works.