Can You Run a Sandbox Fleet on Spot Instances?

Somebody works out the compute line of your cloud bill, notices that the same machine costs 60 to 90 percent less if you accept that it can be taken away, and asks the obvious question. For a platform whose entire cost structure is CPU and RAM, that discount is not a rounding error. It is the difference between a gross margin you can defend and one you cannot.

I'm Ajay; I build PandaStack, a Firecracker microVM platform where every sandbox create is a snapshot restore. I have run this analysis on my own fleet more than once, and the honest answer is neither yes nor no. It is that spot pricing is not a pricing decision at all. It is a statement about your workload: whether a machine can be yanked out from under it and the work reconstructed somewhere else, cheaply enough and fast enough that nobody files a ticket. Some of what I run passes that test easily. Some of it fails so badly that using spot would be negligent.

Price is the wrong question

The question that gets asked is "can we use spot?" The question that decides it is "what is the blast radius of one interruption, and how often does it fire?" Those are not the same question, and the second one has a different answer for every workload sitting on the same fleet.

Think about the two extremes on a sandbox platform. A user calls an API to run a 30-second Python snippet in an isolated VM. The host is reclaimed mid-execution. What was lost? One request, whose inputs you still have, which you can re-run on another host and return a couple of hundred milliseconds later than you otherwise would. The user may not even be able to tell. That is a nearly perfect spot workload — the work is short, the inputs are external, and reconstruction is the same code path as the original request.

Now take a managed Postgres VM holding a customer's data on a disk attached to that host. The host is reclaimed. What was lost? Depending on how careful you were, anything from ninety seconds of availability to the last few minutes of committed writes. There is no "retry" for that. The work is not reconstructible from the request, because the request was three weeks ago and the state has been accumulating ever since. Running that on spot to save money is a decision you get to explain exactly once.

Spot is not cheap compute. It is compute you have agreed to lose. If losing it costs you a retry, take the discount. If losing it costs you data, the discount is a loan against an incident.

The spectrum, from perfect to disqualifying

Most real platforms are not at either extreme, so it helps to rank the workloads rather than argue about the category. Roughly, from best to worst fit:

• Short stateless execution (a code-interpreter call, an eval step, a scraping job). Seconds of work, inputs held by the caller, retry is free. Take the discount.
• CI and build jobs. Minutes of work, deterministic inputs, and every CI system already understands a failed job as something to re-run. Losing a 12-minute build to a preemption costs 12 minutes of machine time and a red check that turns green on retry. Still a good fit, and this is why so many CI runner fleets are spot-only.
• Batch and scheduled work. Fine if the unit is idempotent and checkpointed. Bad if it is a four-hour job with no checkpoints, because expected completion time can exceed the mean time between interruptions and the job never finishes at all.
• Long-lived agent sessions. The interesting middle. The valuable state is in the guest's memory — a loaded context, an installed dependency tree, a half-finished checkout — and it exists nowhere else. Reconstructible in principle, expensive in practice, and the user is watching.
• Hosted apps serving live traffic. Reconstructible from git, which is the saving grace, but reconstruction means a build and a boot while requests are failing. Acceptable if you have more than one instance and a load balancer. Not acceptable if the app is a single VM.
• Databases and anything on a host-pinned disk. Disqualifying, and not because of the interruption rate. It is because the recovery path involves someone else's data and your recovery-point objective, and no discount improves that trade.

The pattern under the list is not "stateful versus stateless" — that framing is too coarse, and it makes people believe an agent session is fine because it is technically ephemeral. The real axis is where the authoritative copy of the work lives. If it lives outside the VM (in the request, in git, in object storage, in a queue), the VM is disposable and spot is fine. If the only copy is in that guest's RAM or on that host's disk, the VM is not disposable no matter what you called it.

Doing the blast-radius arithmetic

Before any of the engineering, do the multiplication, because it frequently ends the discussion. The expected cost of running a workload on spot is roughly the interruption rate multiplied by the amount of work in flight when it fires multiplied by what redoing that work costs you — and that last term is the one people leave out, because it includes the user-visible part.

Two workloads with identical interruption rates land in completely different places. A fleet of 30-second executions loses about 15 seconds of work per interruption and the redo is one retry. A fleet of two-hour sessions loses an hour, and the redo includes an apology. Same hardware, same discount, same reclaim probability; the third term differs by four orders of magnitude.

The other term worth measuring rather than assuming is the interruption rate itself. AWS publishes per-instance-type interruption frequency, and the spread between machine families in one region is wide enough that instance choice matters more than the choice to use spot at all. Which brings up the constraint specific to microVM platforms: Firecracker needs hardware virtualization in the guest, which narrows the menu hard. On GCP that means families supporting nested virtualization; on AWS it generally means bare metal, because most EC2 types will not give you a working KVM. You are not bidding into the deep, diversified pool of general-purpose instances but into a much thinner one, where a capacity crunch is likelier and diversification is scarcer.

What a preemption notice actually gives you

Every provider promises a warning, and the warning is shorter than people design for. GCP flips a metadata key and sends the guest an ACPI shutdown signal, giving you on the order of 30 seconds. AWS publishes an instance-action document to the instance metadata service roughly two minutes ahead, and can additionally send a rebalance recommendation earlier when capacity looks tight. Azure surfaces the same idea through Scheduled Events. Treat all of these as best-effort: they are a courtesy that usually arrives, not a contract that always does, and the correct design assumes some fraction of your losses will be silent.

Consuming the notice is not hard, which is why there is no excuse for not doing it. GCP's metadata server supports hanging GETs, so you can block on the change rather than poll. AWS returns 404 on the spot action path until there is something to say, so you poll it at a few seconds' interval and treat the first non-404 as the starting gun.

#!/bin/sh
# preempt-watch: notice early, cordon fast, evacuate what fits in the window.
# Runs as a systemd unit alongside the host agent. Everything it does must be
# idempotent -- notices can arrive twice, and a slow retry is worse than a
# duplicate.
set -eu
AGENT_ADDR=http://127.0.0.1:7070

drain() {
  reason=$1
  logger -t preempt-watch "preemption notice ($reason): cordoning host"

  # 1. CORDON. Stop new sandboxes being scheduled here. This call must be
  #    cheap and must not block on anything, because step 2 needs the window.
  curl -sf -m 3 -XPOST \
    -H "X-Node-Token: $PANDASTACK_NODE_TOKEN" \
    "$AGENT_ADDR/internal/drain?reason=$reason" >/dev/null || true

  # 2. EVACUATE, with a hard deadline strictly inside the notice window.
  #    Whatever does not finish in time is a loss you already budgeted for.
  timeout 20 /usr/local/bin/evacuate --deadline 18s || \
    logger -t preempt-watch "evacuation incomplete -- falling back to reconstruct"

  exit 0
}

# GCP: hanging GET. Returns when instance/preempted flips to TRUE, so this
# costs one idle connection rather than a poll loop.
if curl -sf -m 2 -H 'Metadata-Flavor: Google' \
     http://metadata.google.internal/computeMetadata/v1/instance/id >/dev/null
then
  while :; do
    v=$(curl -s -H 'Metadata-Flavor: Google' \
      'http://metadata.google.internal/computeMetadata/v1/instance/preempted?wait_for_change=true&timeout_sec=3600')
    [ "$v" = "TRUE" ] && drain gcp-preempted
  done
fi

# AWS: IMDSv2. 404 until the notice exists; then a JSON body carrying an
# action and an RFC3339 deadline, typically about two minutes out.
tok=$(curl -sX PUT -m 2 http://169.254.169.254/latest/api/token \
       -H 'X-aws-ec2-metadata-token-ttl-seconds: 21600')
while :; do
  body=$(curl -s -m 2 -H "X-aws-ec2-metadata-token: $tok" \
    http://169.254.169.254/latest/meta-data/spot/instance-action || true)
  case "$body" in
    *'"action"'*) logger -t preempt-watch "notice: $body"; drain aws-spot ;;
  esac
  sleep 5
done

Snapshots turn a preemption into a migration, up to a point

Here a microVM platform has an option a container platform does not. If you can capture a running guest — RAM, vCPU state, device state — and push it to object storage, a preemption stops being a failure and becomes a migration: the sandbox vanishes on the doomed host and resumes on another one at the instruction it was on. That machinery already exists on PandaStack because it is the normal boot path — templates are baked snapshots in object storage and restore is how every create works — so using it for evacuation is a scheduling problem, not a new capability.

So do the arithmetic on whether it fits in the window, because this is exactly where optimism kills you. A 4 GiB guest is 4096 MiB of memory image. Pushing that inside a 30-second GCP notice needs about 137 MiB/s sustained, roughly 1.1 Gbps, on a NIC you are sharing with every other guest on the host and with whatever else is evacuating at the same moment — and that is before the time it takes to pause the VM and serialize the state at all. Inside a two-minute AWS window the same image needs about 34 MiB/s, which is comfortable. The same technique is therefore fine on one cloud and marginal on the other, for the same guest.

Two things make it better than the raw arithmetic suggests, and one thing makes it worse. Better: most of a typical guest's memory is zero pages, and if your snapshot format records which chunks are non-zero — ours does, so restore can zero-fill instead of fetching — you upload materially less than the nominal size. Better again: a guest whose memory is already streamed from object storage only needs its dirtied pages moved. Worse: the window is per host, not per guest, so a host packed with 20 sandboxes has 30 seconds total, not 30 seconds each. Evacuation is a bandwidth budget you divide, which means triage rather than a loop over everything.

The triage rule I would use: evacuate the guests whose state is irreproducible and small, abandon the ones that are cheap to reconstruct, and never let an evacuation attempt on a big guest starve a small one. Which leads to the point people miss — for a lot of workloads, reconstruct beats evacuate. A hosted app whose source is in git and whose template snapshot is already in object storage does not need its RAM saved. It needs a fast restore somewhere else, which is a thing you already have. Saving memory is the expensive answer to a question that sometimes has a cheap one.

Draining and evacuating are different jobs

Draining means stop scheduling new work onto this host. Evacuating means move the work already on it. Draining is easy, fast and should be the first thing your notice handler does; evacuating is slow, bandwidth-bound and may not finish. Conflating them is the most common mistake I see, because Kubernetes gave everyone one verb for both.

The good news is that if you already have a heartbeat-and-lease model, the cordon is nearly free. Our agents write a row every 10 seconds and the scheduler only considers agents whose row says active with a heartbeat inside 30 seconds. So marking a host as draining is one UPDATE, and every scheduler in every region stops placing on it without any coordination protocol, any consensus, or any notification fan-out.

// Agent side: the entire cordon. One row, one column, no coordination.
// (agent/internal/registry/registry.go)
func (r *Registry) Drain(ctx context.Context) error {
	_, err := r.db.ExecContext(ctx,
		`UPDATE agents SET status='draining' WHERE id=?`, r.id)
	return err
}

// Scheduler side: the candidate loop that makes the UPDATE meaningful.
// (api/internal/scheduler/scheduler.go)
for _, a := range agents {
	if a.Status != "active" {
		continue // cordoned, or never came up
	}
	// ... region filter, pool filter, memory admission, scoring
}

// And the query that feeds it, which is the second half of the same idea:
//
//   SELECT ... FROM agents
//   WHERE status = 'active'
//     AND last_heartbeat > now() - interval '30 seconds'
//
// A host that dies WITHOUT a notice -- the silent case -- falls out of
// placement on its own within 30 seconds, because absence of a heartbeat is
// the signal. That property is what makes an unreliable fleet survivable:
// you are not relying on the dying host to tell you it died.

There is a subtlety in that design that spot makes urgent. Schedulers cache the agent list — ours for 30 seconds — because reading the table on every create is wasteful. That cache is the reason a cordon is not instant. On a two-minute AWS notice, a 30-second cache is a quarter of the window and nobody cares. On a 30-second GCP notice, a 30-second cache means new sandboxes can keep landing on a host that is already dying for most of the time you had. If you are going to run spot on a short-notice cloud, the cordon has to bypass the placement cache, not just update the row it was built from.

The other half is bookkeeping, and it pays to know what your system claims after a host vanishes. When one of ours disappears without releasing anything, routing notices in 30 seconds, but the sandbox rows themselves stay marked running until their lease expires — and the default lease TTL is 24 hours. A sweeper eventually flips expired-lease sandboxes to failed, and a restarting agent reconciles the ones it no longer holds. Fine for a fleet where hosts rarely disappear. On a spot fleet, a database that lies for up to a day about what is running is a real problem, and that TTL has to come down to something near the interruption interval.

Why stateful hosts resist all of this

There is a structural reason the database half of a platform cannot follow the sandbox half onto spot, and it also explains why stateful hosts are hard to autoscale down on ordinary on-demand capacity. The moment durable data lives on a disk attached to a specific host, that host is not interchangeable. Placement becomes sticky for the life of the data. Scale-in becomes dangerous, because removing the wrong instance strands a volume. On GCP this is not even a judgement call: a managed instance group with a stateful policy cannot have an autoscaler attached at all. I found that out the direct way, by discovering that the fleet I believed was autoscaling had never been able to. Meanwhile the platforms that do scale their sandbox pools cleanly — E2B's infrastructure is public and worth reading — keep worker nodes genuinely stateless, with scratch local SSD and persistent volumes on network storage, precisely so no node is ever irreplaceable.

Our answer was to split the fleet into two pools rather than pretend one policy fits both. The scheduler knows about a stateful pool and an ephemeral pool, a volume placement is pinned to the stateful pool no matter what the caller asked for, and managed databases inherit that pinning through the volume they sit on. The default direction of the fallback is the part I would defend hardest: an agent that does not report a pool is treated as stateful, because calling an ephemeral host stateful costs a slightly worse placement, while calling a stateful host ephemeral lets a scale-in strand somebody's data.

Default toward the safe error. If you cannot tell whether a host is holding something irreplaceable, assume it is.

That split is exactly the boundary a spot tier would follow. The ephemeral pool has no durable disk, so losing a host there costs running sandboxes and nothing else. The stateful pool holds volumes, so it stays on-demand forever. Which is a much more useful way to think about it than "should we use spot" — the question becomes which of your pools has already been designed to survive losing a host, and the answer is usually exactly one of them.

The risk nobody budgets for: no capacity at all

Interruption gets all the attention. The failure mode that actually hurts is unavailability: you ask for spot capacity and there is none. It is not random, and that is the important part. Spot capacity evaporates when on-demand demand rises, which is to say during regional busy periods, during large customer events, during exactly the hours your own traffic peaks. Your scale-out request and every other tenant's scale-out request arrive at the same moment, and the pool that funds all of them is the leftovers.

So the correlation runs the wrong way twice: you are likeliest to be interrupted when you are busiest, and likeliest to fail to replace the lost capacity at the same moment. An all-spot fleet does not degrade gracefully under that, it degrades at once — which users experience as creates failing with a capacity error. Add the thin nested-virt or bare-metal pool from earlier and an all-spot fleet is one regional squeeze away from being entirely down.

The sane architecture is boring: an on-demand baseline sized to your steady-state load and the workloads that cannot be interrupted, plus a spot burst tier that absorbs peaks. You capture most of the discount, because peaks are where the marginal capacity is, and you never end up in a state where a capacity squeeze takes the whole product offline. If you also diversify machine types and zones inside the burst tier, you are back to a fleet where a single pool going dry is an inconvenience.

# Two pools, one module, different scheduling. The baseline is boring on
# purpose: it carries the durable disk and anything that cannot be rebuilt.
module "agents_baseline" {
  source          = "./modules/gcp-agent-mig"
  agent_count     = 3
  use_preemptible = false     # STANDARD provisioning, automatic_restart on
  pool            = "stateful"
}

# The burst tier. Cheap, disposable, holds nothing that matters. Diversify
# machine types across a few MIGs so one pool going dry is not fleet-wide.
module "agents_burst" {
  source          = "./modules/gcp-agent-mig"
  agent_count     = 0         # floor of zero; scales up under load
  use_preemptible = true
  pool            = "ephemeral"
}

# What use_preemptible actually flips, inside the module:
#
#   scheduling {
#     preemptible                 = true
#     automatic_restart           = false
#     provisioning_model          = "SPOT"
#     instance_termination_action = "STOP"
#     on_host_maintenance         = "TERMINATE"   # nested virt cannot migrate
#   }
#
# Read that last line twice. Nested virtualization means the host CANNOT live
# migrate you, so ordinary host maintenance already terminates the instance.
# On a microVM fleet you are living with interruption whether or not you take
# the discount -- which is an argument FOR spot, not against it.

What actually happens when a host disappears

I want to ground this in something that really happened, because the theory above is tidy and the practice is not. We lost a batch of managed databases on a host that was not on spot at all. Sequence: GCP ran host maintenance; nested virtualization means the instance cannot live migrate, so it terminated; the managed instance group's autohealer repaired it by recreating the instance, which gave it a fresh boot disk. Every hibernation snapshot lived on that boot disk. All of them were gone.

What survived was the separately attached persistent disk holding the actual database volumes — every one of them intact, no customer data lost. The lesson has nothing to do with spot and everything to do with spot: your disk layout decides what a host loss costs you, and you find out which files were on the boot disk at the worst possible moment. Every artifact on the boot disk of a machine that can be recreated is a cache, whether or not you designed it as one. If you cannot rebuild it from object storage or a separately attached volume, it is not a cache, and it should not be there.

The second lesson was about silence. The status flips were invisible — no events, no audit rows, nothing saying which database failed when — because the code paths that marked them failed emitted nothing, and the host's journal died with its boot disk. Ship logs off the host before you need them, and make every state transition emit an event. On a fleet where hosts disappear routinely, the post-mortem has to be reconstructable from data that was never on the machine.

What PandaStack actually does today

Being precise about the difference between what I have built and what I am recommending, because those are not the same list. The Terraform modules for our agent groups take a preemptible flag, and it flips exactly the scheduling block shown above. Production sets it to false. The fleet is on-demand today, and the reason is the one this whole post has been circling: the pool that holds durable volumes cannot be on spot, and until the ephemeral pool carries enough of the load to be worth the engineering, running two different reliability models buys complexity rather than savings.

What does exist is the machinery you would need. Heartbeats every 10 seconds with a 30-second staleness cutoff, so a vanished host leaves the placement set on its own. Leases that record which host owns which sandbox, plus a sweeper for the ones whose owner never came back. A pool split that already pins volumes to non-autoscaling hosts. A snapshot-restore path that is the ordinary create path, which is what an evacuation would use. And a capacity-refusal path that parks an app in a waiting state and retries with backoff rather than failing it permanently when no host will take it.

What does not exist: nothing in our codebase reads a preemption notice. There is no metadata poller, no drain endpoint on the agent, and the function that marks an agent as draining is written but never called — a graceful shutdown today simply stops heartbeating and ages out of the placement set after 30 seconds. That is adequate for on-demand hosts and would be the first gap to close before turning that flag on. I would rather say that than imply a capability I have not shipped.

import time
from pandastack import Sandbox

# The client-side half of a spot strategy, which is where most of the value
# is: make the unit of work small enough that losing it is a retry. Nothing
# here is spot-specific -- that is the point. Code that survives an
# interruption is just code that survives a host failure.

def run_untrusted(source: str, attempts: int = 3):
    last = None
    for attempt in range(attempts):
        sbx = None
        try:
            # Every create is a snapshot restore, so a retry costs a couple
            # of hundred milliseconds of platform time, not a boot.
            sbx = Sandbox.create(template="code-interpreter", ttl_seconds=120)
            sbx.filesystem.write("/tmp/job.py", source)
            result = sbx.exec("python3 /tmp/job.py", timeout=60)
            return result
        except Exception as err:          # host vanished, proxy refused, timeout
            last = err
            time.sleep(0.5 * (2 ** attempt))
        finally:
            if sbx is not None:
                try:
                    sbx.kill()
                except Exception:
                    pass                  # the host may already be gone
    raise RuntimeError(f"job failed after {attempts} attempts") from last

# The design rule this encodes: the authoritative copy of the work is the
# "source" string, held by the caller. The sandbox holds only a derivative.
# Anything you would be unwilling to lose does not belong inside the guest.

The decision framework, compressed

If you want the whole post as a sequence of questions to answer in order, it is this.

1. Where does the authoritative copy of the work live? Outside the guest means spot is on the table. Only inside the guest's RAM, or on a host-pinned disk, means it is not.
2. How long is the unit of work relative to the mean time between interruptions? If a job routinely takes longer than the interval between reclaims, it does not just get slower — it can fail to complete at all. Checkpoint or shorten it before you consider the discount.
3. What is the notice window on your cloud, and what can you actually move inside it? Do the megabytes-per-second arithmetic for one guest, then divide by the number of guests per host. If the answer is that you can evacuate two of twenty, plan for triage instead of evacuation.
4. Can you cordon faster than the notice? A drain that takes effect after a 30-second placement cache is not a drain on a 30-second window.
5. Does the silent case work? Some fraction of losses arrive with no notice at all, so absence of a heartbeat must be sufficient to remove a host from placement and to stop routing requests to it.
6. Which pool is this? Split the fleet before you split the pricing. The pool that carries durable disks stays on-demand; the pool that carries disposable work is the one that gets the discount.
7. What is on the boot disk? Anything that cannot be rebuilt from object storage or a separately attached volume is going to be lost the first time an instance is recreated, spot or not.

The answer to the headline question, then, is that you can run most of a sandbox fleet on spot and you should not run all of it. The execution tier — short, stateless, reconstructible, exactly the workload people build sandbox platforms for — is close to an ideal spot workload, and there is a real argument that a microVM fleet should be there already, given that nested virtualization means ordinary host maintenance already terminates you. The state tier is not, will not be, and is the part your customers would actually notice.

The engineering that makes the difference is not clever bidding. It is the unglamorous stuff: leases, heartbeats, a fast cordon, an idempotent retry, snapshots you can restore anywhere, and a very clear line between the hosts you can lose and the hosts you cannot. Build those and spot becomes a pricing choice you can make per-pool. Skip them and the discount is just a faster route to an incident.