How to cut your sandbox compute bill: an engineer's playbook

Every few weeks somebody sends me a screenshot of a compute bill with a question attached: "is this normal?" It usually isn't, and the reason is never the one they expect. They want to talk about the per-second rate. But the rate is a small number multiplied by a number they never measured, and it is the second number that is wrong. Bills are high because things ran that nobody was watching, for longer than anyone intended, doing work already done once.

I'm Ajay; I run PandaStack, a Firecracker microVM platform, so I see both the bills our customers generate and the ones we pay for the fleet underneath. What follows is the order I attack an ephemeral-compute bill in — sandboxes, agent runs, CI jobs, preview environments. It is a vendor-neutral playbook, not a pricing comparison, and nearly all of it transfers to whatever platform you are on.

Step zero: you cannot cut a bill you cannot attribute

The first thing I ask when someone shows me a bill is "which customer is that?" and the answer is almost always a shrug. That shrug is the problem. A single aggregate number is not something you can optimize, only something you can feel bad about. Before anything else, you need to slice spend by customer, feature, environment, and individual run.

This is the one item with a hard deadline, which is why it goes first. Attribution happens at create time or not at all. You can retroactively re-tag an S3 bucket; you cannot retroactively label a VM that stopped existing four hours ago, because there is nothing left to attach the label to. Untagged ephemeral compute is unattributable forever — every week you delay is a week of spend you will never explain, and therefore never cut.

from pandastack import Sandbox

# Two things every create should have: a TTL, and enough metadata to
# answer "who did this and why" three weeks from now.
sbx = Sandbox.create(
    template="base",
    ttl_seconds=900,  # 15 minutes. The platform enforces this, not you.
    metadata={
        "org": customer_id,        # who to bill, or who to go talk to
        "env": "prod",             # prod | staging | ci | dev
        "feature": "code-review",  # which product surface spawned it
        "run_id": run_id,          # ties this box to one agent run / one CI job
        "owner": "team-agents",    # who gets the email when it lands in a report
    },
)

try:
    result = sbx.exec("python analyze.py --input /work/repo")
finally:
    sbx.destroy()  # the optimization; the TTL above is the safety net

Five keys, one dictionary literal, and it is the difference between "our bill went up 40%" and "one customer's nightly job started forking twice per document on the 12th." Pick the keys once, put them in a helper, and make that helper the only way anyone creates a sandbox. If someone can call the raw create API directly, someone eventually will — at 2am, during an incident — and that box will still be running next quarter.

The dominant cost driver is idle time and forgotten resources

Once you can slice the bill, the same picture appears nearly every time: a long tail of sandboxes that outlived their purpose. Not a hot loop, not a runaway process — just boxes sitting there alive, doing nothing, because whatever was supposed to clean them up did not. This usually outweighs every other optimization here combined, and is the cheapest to fix.

So: a TTL on every sandbox, no exceptions, set at create time — not as a fallback for the sloppy path, but as the primary mechanism. Explicit destroy is the optimization that takes you from fifteen minutes to forty seconds. The TTL guarantees fifteen minutes is the worst case.

The obvious objection is "we already clean up in a finally block." I wrote that code too. It does not hold, because the process running the finally block is exactly the process that crashes: the worker is OOM-killed, the pod is evicted mid-rollout, the container takes a SIGKILL as the node drains, the network partitions. The cleanup path is the casualty, and the sandbox — a separate, healthy VM with no idea anything went wrong — cheerfully continues to exist. A cleanup mechanism that shares a fate with the thing it cleans up is not a mechanism. It is a hope.

TTL is the only colleague who never forgets, never gets paged, and never has a bad deploy.

• Set the TTL to roughly twice your p99 job duration, not your average — and measure it, because guesses are always too generous.
• For interactive sessions, use a short TTL and extend it on activity. Someone who walked away at lunch should not cost you the afternoon.
• Never set a TTL you would be embarrassed to explain. "24 hours, to be safe" means every leaked box costs a full day, and you will leak boxes.
• Alert on TTL expirations, not just failures. A sandbox that routinely dies by TTL rather than explicit destroy is telling you the cleanup path is broken somewhere.

The asymmetry worth designing around: CPU-seconds vs committed GiB-hours

This is the part of the model that most often flips people's intuition. Ours is per-second-style usage billing, but the two resources are not metered the same way. CPU is billed on active CPU-seconds actually burned — an alive-but-idle sandbox burns almost none. Memory is billed on committed GiB-hours: guest RAM is reserved the moment the VM exists, touched or not, for as long as the box is alive.

• An idle-but-alive sandbox is nearly free on CPU and fully priced on memory. Idle does not mean cheap; it means paying for the expensive half and getting nothing back. That is the argument for aggressive TTLs in one sentence.
• Bursty CPU is not the enemy. If a job pins eight cores for ten seconds and exits, you paid for ten seconds of cores. Do not architect around a fear of short spikes; spikes are the cheap shape.
• Wall-clock time is the thing to attack, because memory bills against it. Finishing sooner with more parallelism is usually a straight win: same CPU-seconds, fewer GiB-hours.
• Waiting is the worst state of all: an agent holding a sandbox open while it waits thirty seconds on a model call pays committed memory for thirty seconds of nothing.

Other platforms split this differently — some bill full lifetime for both resources, making idle time more expensive still. Check which model you are on before optimizing; the correct move is different in each case.

Right-sizing memory, and the snapshot-restore twist

Memory being the committed resource, right-sizing it is where the money is — but on a snapshot-restore platform that does not mean what it means on a container platform. Every create restores a previously baked Firecracker snapshot — that is how the boot path lands at roughly 179ms p50 and 203ms p99 instead of the ~3s a genuine cold boot takes. But a snapshot is a frozen machine, and a frozen machine has a fixed amount of RAM. Guest memory is a property of the snapshot itself, baked in when it was taken; the VM comes back exactly as large as it was when it was frozen, whatever number you pass the restore.

So the per-request memory dial you are used to does not exist. Right-sizing means picking the right template — or baking one if none fit — a decision made once per workload class rather than per call.

The honest trade: an oversized template costs committed GiB-hours on every run, forever, for memory nobody touches. An undersized one gives you a build that OOMs at the worst moment, far more expensive in engineer-hours than the RAM ever was. So measure: run the workload on the big template, watch the peak resident set, add real headroom, then move to the smallest template above that line.

Scale to zero for the long-lived and mostly idle

TTLs handle short-lived work. They are the wrong tool for the thing that must exist for weeks but is used for forty minutes of that: a dev environment, a preview deploy, a sales demo, an internal tool three people open on Tuesdays.

The traditional answer is to keep it warm because starting it is slow — a hostage negotiation with your boot time, unnecessary once boot is fast. If a restore lands in a couple hundred milliseconds, snapshot the environment when it goes idle, release the memory entirely, and restore on the next request. The user sees a pause between imperceptible and about a second, and you stop paying committed memory for the 99% of the week nobody was looking.

The details are boring: pick the idle threshold from real access patterns, not vibes; make sure health checks and synthetic probes do not count as activity (I have watched one monitor single-handedly stop a demo environment from ever sleeping); and trigger the wake on the request itself. If waking needs a human, nobody will let it sleep.

The biggest lever for agents: snapshot once, fork per run

If you run agent workloads this is the item most likely to move your bill, and people reach for it last because it sounds like a performance feature. It is also a cost feature, for the simple reason that work you do not repeat is work you do not pay for.

Look at what a typical agent run does before anything useful. It boots, pip-installs the same forty packages as last run, downloads the same dataset, warms the same headless browser. On a ninety-second run, sixty of those seconds can be setup producing a byte-for-byte identical result to the previous run — paid at full committed memory, plus egress for every byte.

Pay for it once. Build the environment, snapshot it, fork per run. A fork is copy-on-write on both memory and disk, so the child starts from the parent's exact warmed state without redoing any of it — same-host forks land in roughly 400–750ms, cross-host in 1.2–3.5s when state has to travel.

from pandastack import Sandbox

# --- pay the expensive setup exactly once -------------------------------
golden = Sandbox.create(
    template="base",
    ttl_seconds=3600,
    metadata={"env": "build", "owner": "team-agents", "feature": "golden-image"},
)
golden.exec("pip install -r /work/requirements.txt")
golden.exec("python -c 'import pandas, numpy, transformers'")  # warm import caches
golden.filesystem.write("/work/dataset.parquet", DATASET_BYTES)
golden.exec("python /work/warm_browser.py")                    # pre-launch chromium

# durable copy, so the golden box is reproducible after a deploy or a restart
golden.snapshot()


# --- per run: no install, no download, no warm-up ----------------------
def run_task(task: str) -> str:
    child = golden.fork()   # CoW memory + reflinked disk; setup already done
    try:
        return child.exec(f"python /work/agent.py {task}").stdout
    finally:
        child.destroy()

Two notes from getting this wrong. The golden sandbox is now infrastructure with a lifecycle: it needs rebuilding when dependencies change, and that rebuild belongs in CI rather than in someone's head — the snapshot makes it reproducible. And forks inherit everything, secrets included. Build the golden box from a clean, non-secret-bearing state and inject per-run credentials into the child. A fork is a very efficient way to copy a mistake.

Batching and concurrency shape

Many short sandboxes versus one long one turns on the ratio of setup cost to task duration. Short boxes win when setup is cheap or already amortized by a fork, when tasks are independent, and when isolation matters: each box lives exactly as long as its task, so committed memory tracks real work almost perfectly. Fast restore makes this viable — at three seconds a create, per-task boxes are painful; at a couple hundred milliseconds, it is just how you write the loop.

One long batch box wins when tasks share expensive mutable state, or are so short that lifecycle overhead dominates — spinning a VM for an 80ms task is silly. The trap is that it sits idle between batches, straight back in the expensive quadrant. Make it die when the queue drains rather than waiting politely for more work.

The failure mode I see most is a fixed-size worker pool sized for peak and kept alive around the clock at maybe 15% utilisation overnight — the worst of both: full committed memory, almost no CPU-seconds. If you have a pool, it should shrink.

Egress: the line item nobody models

The compute is cheap and then something pulls a 2 GB model file on every run. It surprises people because it is invisible in the code — one line in a Dockerfile, one `from_pretrained` call — and it does not appear in the part of the bill they are staring at.

The fix is to move the bytes from run time to build time. Bake model weights, image layers, npm and pip caches and browser binaries into the template — fetched once at build, local for every sandbox afterwards. It compounds with the fork pattern: a golden sandbox forked a thousand times pulls the dataset once.

• Audit what a cold run actually fetches. Watch the network from inside the box for one run; there is usually something on the list you did not know about.
• Pin dependency versions — an unpinned install refetches whenever upstream publishes, turning a cached path into an uncached one at random.
• Watch the sneaky ones: telemetry SDKs phoning home, a package manager checking for updates on every invocation, a `latest` tag that re-resolves.
• Large outputs count too. If every run uploads a multi-gigabyte artifact, ask whether the consumer needs all of it.

Storage: the stuff that survives the compute

Compute stops. Snapshots and volumes do not. They accrue quietly, and because each is individually tiny nobody deletes any of them. Six months later there are eleven thousand and the storage line is a real number.

The fix is a retention policy on a schedule, not a cleanup you promise to do. Mine: golden and base snapshots kept until explicitly replaced; pipeline-created ones expire after a fixed window; hand-made ones too, with a longer window and a warning first. A volume unattached for a week is a bug report, not a resource.

import datetime as dt
from pandastack import Sandbox

NOW = dt.datetime.now(dt.timezone.utc)
MAX_AGE = {"ci": dt.timedelta(hours=2), "dev": dt.timedelta(hours=12)}

unattributed = []
for sbx in Sandbox.list():
    meta = sbx.metadata or {}
    age = NOW - sbx.created_at

    # 1. anything without an owner is a finding, not a resource
    if not meta.get("owner"):
        unattributed.append((sbx.id, sbx.template, age))
        continue

    # 2. environment-specific reaping for whatever outlived its TTL story
    limit = MAX_AGE.get(meta.get("env", ""))
    if limit and age > limit:
        print(f"reaping {sbx.id} env={meta['env']} run={meta.get('run_id')} age={age}")
        sbx.destroy()

if unattributed:
    print(f"{len(unattributed)} unattributed sandboxes -- fix the create path:")
    for sid, template, age in unattributed:
        print(f"  {sid} template={template} age={age}")

Run it nightly. The reaping matters less than the report: unattributed sandboxes mean some code path creates compute without going through your helper, and that path is where your next surprise comes from.

The levers, ranked by effort against impact

• Attribution metadata on every create — Effort: one dictionary and a helper. Typical impact: none directly, decisive indirectly. Nothing else here is measurable without it, and the window closes when the box dies.
• A TTL on every sandbox — Effort: one argument. Typical impact: large, and the only lever that keeps working while your cleanup code is crashing.
• Snapshot once, fork per run — Effort: a day restructuring your run loop. Typical impact: large for agent and CI workloads that reinstall identical dependencies each run; negligible if setup is trivial.
• Pick the smallest template that fits — Effort: one benchmark per workload class. Typical impact: large on memory-dominated bills, near zero on CPU-dominated ones. Find out which you have first.
• Hibernate long-lived idle environments — Effort: an idle policy plus a wake path. Typical impact: large for dev boxes, previews and demos; irrelevant for short runs.
• Cache dependencies and models into the template — Effort: a rebuild plus a pipeline to keep it fresh. Typical impact: situational — large if you pull big artifacts per run, invisible otherwise.
• Retention policy on snapshots and volumes — Effort: a nightly cron. Typical impact: small this month, large this year, because storage only accumulates.
• Reshape batching and concurrency — Effort: high; it changes your architecture. Typical impact: situational, and the last one worth reaching for. Do the seven above first.

The audit you can run this week

1. List everything running now and count how much has no owner tag. That number is your attribution debt, and the only item here with a deadline.
2. Find every create call and check it passes a TTL. Grep for the raw API too, not just your helper — the bypass is always the interesting one.
3. Take the longest-lived sandbox in your account and find out why it exists. Better-than-even odds nobody knows.
4. Compare your p99 job duration against your TTLs. More than about 3x the p99, tighten it.
5. Instrument one representative run: peak resident memory, CPU-seconds, bytes fetched. Those three numbers tell you which lever is yours.
6. Time the setup phase of an agent run as a fraction of the whole. Over a third, and the fork pattern is your biggest win.
7. Count your snapshots and volumes, check the age of the oldest, then write the retention policy before the number gets embarrassing.
8. Give one environment nobody uses at night an idle policy, and confirm your probes are not keeping it awake.

None of this is clever. It is tagging, timeouts, not repeating work, and deleting things — four ideas that have been cutting infrastructure bills for twenty years, in new clothes. It stays worth writing down because ephemeral compute makes all four easy to skip: everything is supposed to be temporary, so nobody plans for the case where it isn't. And nothing is as permanent as a temporary environment. The cheapest sandbox is still the one you remembered to destroy.