Running Long-Lived AI Agent Tasks in microVMs Without Paying for Idle

The demos make agents look instantaneous, but the real ones are slow in a very specific way: they run for a long time and do almost nothing for most of it. A deep-research agent crawls forty pages, reads them, decides what to fetch next, and crawls forty more — over twenty minutes of wall-clock, maybe ninety seconds of which is actual computation and the rest is waiting on the model to emit the next plan. A coding agent clones a repo, installs dependencies, edits files, runs the test suite, reads the failures, edits again — an hour of work, most of it spent staring at a token stream. These tasks are long-lived and stateful: they need a working directory, installed packages, and running processes that persist across dozens or hundreds of tool calls. And they are, embarrassingly, idle most of the time. This post is about the execution model that fits that shape — one durable sandbox per agent run that survives every step, hibernates when the agent is thinking, and wakes exactly where it left off — instead of the stateless fresh-VM-per-call model that quietly makes long tasks impossible.

Why the stateless fresh-VM-per-call model breaks on long tasks

The clean, stateless way to run agent code is one throwaway sandbox per tool call: spin up a fresh VM, run the single command, capture the output, tear it down. It's a lovely model for a stateless code interpreter — evaluate this expression, plot this dataframe — and it's exactly wrong for a task that runs for an hour and builds up state as it goes. The reason is that everything the agent accumulates lives in that VM, and if the VM dies after every call, nothing accumulates.

• The working directory evaporates. Step 3 clones a repo into /work; step 4 lands in a brand-new empty VM with no /work at all. The agent's edits from ten steps ago are simply gone, so it either re-does the work every call or fails outright reasoning about files that no longer exist.
• Installed dependencies don't survive. A coding agent runs `npm ci` or `pip install -r requirements.txt` — thirty seconds to two minutes of setup — and the next tool call starts from a machine that never heard of any of it. Re-installing the world on every step turns a one-hour task into a ten-hour one.
• Process state is lost. The dev server the agent started to test its change, the database it seeded, the background worker it launched — all gone the instant the VM recycles. There's no way to 'start a server in step 5 and hit it in step 8' if steps 5 and 8 are different machines.
• There's no memory between calls. The whole point of a multi-step agent is that step N depends on the results of steps 1 through N-1. A stateless-per-call model forces you to serialize all that state out of the VM and rehydrate it back in every single call — you end up rebuilding a persistent workspace by hand, badly, on top of a model that was designed to prevent one.

You can paper over some of this by mounting a shared volume and re-attaching it each call, but you can't paper over running processes or warm in-memory state, and you pay the reconnect/rehydrate cost on every step. The honest conclusion is that a long-running, stateful agent wants the opposite of stateless-per-call: it wants one machine that stays its machine for the whole run.

The pattern: one durable sandbox per agent run

The model that actually fits is one long-lived sandbox per agent run. When the run begins, you create a single persistent microVM. That VM holds the working directory, the installed dependencies, and any processes the agent starts — and it stays alive across every tool call for the duration of the run. Each step the agent takes is an `exec` or a filesystem write against that same machine, so step 8 sees exactly what step 5 did to it. When the run finishes, you collect the results and delete the VM.

This is the natural shape: the sandbox's lifetime matches the run's lifetime, not the tool call's. The agent gets a stable workspace it can build up incrementally, exactly like a human developer's checkout that persists between commands in a terminal. Isolation is per-run rather than per-call, which is also what you want — one runaway agent's mess is contained to its own VM (more on that below), while its own steps share state freely with each other. Create latency is a snapshot restore, so standing up that per-run machine is a p50 of about 179ms (p99 around 203ms), and a `ttl_seconds` backstop means a run that hangs or a supervisor that crashes doesn't leave a VM renting a room forever.

from pandastack import Sandbox

# One durable sandbox for the WHOLE agent run. It restores a baked template
# snapshot (~179ms p50), and the ttl is a backstop so a hung run reaps itself
# even if your supervisor dies. This VM is the agent's workspace for every step.
sbx = Sandbox.create(
    template="agent",
    ttl_seconds=3600,                       # 1h ceiling; the run should finish well inside it
    metadata={"run": run_id, "kind": "deep-research"},
)

try:
    # Expensive setup happens ONCE and persists for the rest of the run.
    sbx.exec("git clone --depth 1 https://github.com/acme/service /work")
    sbx.exec("cd /work && pip install -r requirements.txt")  # deps stay installed

    # The agent loop: each step is an exec or fs write against the SAME machine.
    # Step N sees everything steps 1..N-1 did to the filesystem and processes.
    for step in agent.plan(run_id):
        if step.kind == "write":
            sbx.filesystem.write(step.path, step.content)   # edits accumulate
        else:
            result = sbx.exec(step.command, timeout_seconds=120)
            agent.observe(result.stdout, result.stderr, result.exit_code)

        # ...and between steps, the agent calls the LLM to decide what's next.
        # During that call, this VM is doing nothing but holding its state.
    outcome = sbx.filesystem.read("/work/report.md")
finally:
    sbx.delete()   # collect results first; idle cost after this is ~zero

That loop is the whole idea: create once, `exec` and `filesystem.write` many times against the same durable machine, collect, delete. But look at the comment on the last line of the loop body — between every step, the agent calls the model to decide what to do next, and during that call the VM is holding a warm working directory and doing absolutely nothing with the CPU it's been allocated. On a long task that dead time dominates. That's the problem hibernation solves.

Hibernate while the agent is thinking (scale-to-zero)

Here is the uncomfortable arithmetic of a long agent run: the model's latency, not your code's, sets the clock. A deep-research step might spend a second fetching a page and then fifteen seconds waiting for the LLM to read it and plan the next fetch. A coding step runs a test suite in twenty seconds and then waits a minute for the model to reason about the stack trace. Sum it up over a twenty-minute run and the VM is genuinely busy for maybe a tenth of that wall-clock. The other 90% it sits idle, fully resident, burning a scheduled vCPU and holding its RAM, waiting for a token stream. You are paying for a hot CPU to wait.

Your agent spends most of its wall-clock waiting for the next token, not running code. A always-on VM bills you for every idle second of that wait. Hibernation bills you for the seconds the agent is actually doing something.

Hibernate flips it. When the agent hands control back to the model — the point where it's about to wait — you snapshot the VM and pause it: the working directory, the installed deps, the running processes, the warm page cache, all frozen to disk exactly as they were. Nothing runs, so idle cost rounds to zero. When the model comes back with the next step, you wake the VM and it resumes mid-instruction, exactly where it left off — the dev server is still listening, the repo is still checked out, the interpreter's caches are still warm. There is no reboot and no re-setup, because you didn't tear anything down; you froze it. Wake is a restore, so it lands in a fraction of a second rather than the ~3 seconds a genuine cold boot would cost.

# The agent yields to the LLM between steps. That wait is the expensive idle
# time. Hibernate across it: freeze the whole machine, pay ~0 while thinking,
# then wake exactly where it left off — no reboot, no re-clone, no re-install.

while not agent.done():
    sbx.wake()                    # resume the frozen machine (a restore, sub-second)

    step = agent.next_step()      # cheap: reads state the agent already has
    result = sbx.exec(step.command, timeout_seconds=120)
    agent.observe(result)

    # Now the agent needs the model to plan the next move. Instead of leaving
    # the VM hot and idle for the whole (slow) LLM call, freeze it.
    sbx.hibernate()               # snapshot + pause; idle cost ~zero while frozen

    plan = llm.think(agent.context())   # seconds-to-minutes of pure waiting —
                                        # and you're not paying for a live VM for it
    agent.apply(plan)

# The processes, working dir, and warm caches survived every hibernate/wake
# cycle because you never tore the machine down — you paused a running one.

Whether hibernating across every single LLM call is worth it depends on how long your think-time gaps are — for a step where the model returns in half a second, the snapshot/wake overhead isn't worth it, and you'd only hibernate across the genuinely long waits (a research crawl blocked on a slow model, an agent parked waiting for a human review). But the mechanism is the same, and the shape of the win is clear: on a task that's mostly waiting, hibernating across the waits turns a bill sized by wall-clock into a bill sized by actual work.

Snapshots as durable checkpoints for crash recovery

The same primitive that powers hibernation gives you something a long run desperately needs: crash recovery. An hour-long agent task is an hour-long opportunity for something to go wrong — the host reboots, the supervisor process dies, the agent wedges itself, a deploy rolls the fleet underneath you. If the entire run's accumulated state lives only in a live VM's RAM, any of those events vaporizes an hour of work and you start over from step one.

A snapshot is a durable checkpoint. At meaningful milestones in the run — after the repo is cloned and deps are installed, after each phase of a multi-phase research plan, before a risky operation — take a snapshot of the sandbox. It freezes the whole machine's state to disk as a restorable artifact. If the run then crashes, you don't restart it; you restore the last checkpoint and resume from there, with the working directory, installed packages, and process state exactly as they were at that milestone. Checkpoint every phase and the worst case stops being 'lose the whole run' and becomes 'lose the work since the last checkpoint.'

# Checkpoint at milestones so a crash costs you the last phase, not the run.
checkpoints = []

sbx.exec("cd /work && pip install -r requirements.txt")
checkpoints.append(sbx.snapshot())      # durable checkpoint: env is ready

for phase in research_plan:
    run_phase(sbx, phase)
    checkpoints.append(sbx.snapshot())  # freeze state after each phase

# If the host reboots or the supervisor dies mid-run, don't start over:
# restore the last good checkpoint and resume with the workspace intact.
#   recovered = Sandbox.restore(checkpoints[-1])
#   run_phase(recovered, next_unfinished_phase)

Fork to branch a long task and explore alternatives

Because the whole machine is snapshottable, you can also branch a run instead of just resuming it. Fork restores a copy of the sandbox as a new, independent VM that starts from the parent's exact state — same working directory, same deps, same warm processes — and then diverges on its own. For a long task, that turns a linear run into a tree: reach a decision point, fork the current state N ways, let each branch pursue a different approach in isolation, and keep the one that pans out.

This is genuinely cheap because a fork moves no bulk data. A same-host fork lands in roughly 400–750ms (cross-host 1.2–3.5s) by reflinking the rootfs and mapping the parent's memory copy-on-write, so each branch inherits the entire hour of accumulated setup for free and only allocates storage for the bytes it changes. A coding agent that's fixed the build but has two candidate refactors can fork the fixed state, try both, and run their test suites in parallel from the same warm baseline — instead of exploring one, discovering it's worse, and manually unwinding to try the other. The branches that lose barely diverged from the parent, so discarding them reclaims almost nothing.

Per-run isolation: a runaway agent can't hurt its neighbors

A long-running agent is a long-running opportunity to misbehave. The longer a run goes and the more autonomy it has, the likelier it eventually does something pathological: an infinite loop that pins a core, a runaway process that forks until it exhausts PIDs, a log or a download that fills the disk, a script that tries to scan the local network. On a shared host with no isolation, one such agent degrades or takes down every other run co-tenanted with it. The per-run sandbox model contains all of it by construction, because each run lives in its own microVM with its own guest kernel behind a hardware virtualization boundary.

• A CPU-bound infinite loop pins that VM's vCPUs, not the host's. The runaway agent starves itself; its neighbors, in their own VMs, are untouched.
• A disk-filling run exhausts its own copy-on-write rootfs, not shared host storage. When it hits its ceiling it fails inside its own machine, and no other run's writes are affected.
• A fork bomb or PID exhaustion is trapped in the guest's own process table — a separate kernel from every other run's, so it can't spill over into a sibling's.
• A destructive command — an agent that decides to 'clean up' with `rm -rf` or wipes its own working directory — destroys exactly one run's workspace. The blast radius is that run and nothing else, and a checkpoint from earlier can restore even that.

Per-run isolation is also what makes the durable-workspace model safe to hand real autonomy. Because the machine is a genuine microVM and not a shared namespace, you can let a long-running agent install whatever it wants, run whatever processes it needs, and make a mess — and know the mess is contained to a VM that belongs to that run alone. Combined with a `timeout_seconds` on each `exec` (a circuit breaker for a single wedged command) and the `ttl_seconds` backstop on the sandbox itself, a misbehaving run is bounded in space and in time.

Stateless-per-call vs. durable-per-run, always-on vs. hibernate-when-idle

There are really two axes here: how long the sandbox lives relative to the work, and what you pay while the agent is idle. Laying them out side by side:

• State across steps — Stateless fresh-VM-per-call: none; every tool call is a new machine, so the working directory, installed deps, and processes vanish between steps and must be rehydrated by hand. Durable per-run sandbox: full; one VM holds the working directory, deps, and running processes for the entire run, so step N sees everything steps 1..N-1 did.
• Setup cost — Stateless per-call: paid on every call, because each fresh VM starts from the bare template and re-clones/re-installs. Durable per-run: paid once at the start of the run and reused for every subsequent step.
• Fit — Stateless per-call: correct for a stateless code interpreter (evaluate one expression, no accumulated state). Durable per-run: correct for a multi-step agent whose steps depend on each other — deep research, coding tasks, background workers.
• Idle cost, always-on VM: you keep the run's VM live and resident the whole time, so you pay for a scheduled vCPU and held RAM during every LLM think-gap — which on a long task is most of the wall-clock. Hibernate-when-idle: you snapshot-and-pause across the waits, so nothing runs between steps and idle cost rounds to zero; you pay for the seconds the agent is actually computing.
• Crash recovery — Always-on, no checkpoints: a host reboot or supervisor crash vaporizes the run's in-RAM state and you restart from step one. Durable per-run with snapshot checkpoints: restore the last checkpoint and resume with the workspace intact — worst case is losing the last phase, not the whole run.
• Blast radius — Shared host: one runaway agent (infinite loop, disk-fill, fork bomb) degrades every co-tenant. Per-run microVM: each run is isolated behind its own guest kernel and hardware boundary, so a runaway agent hurts only itself.

The honest summary: stateless-per-call is the right tool for a stateless code interpreter and a trap for a long, stateful agent run; an always-on durable VM fixes the state problem but bills you for every idle second the agent spends waiting on the model; and a durable per-run sandbox that hibernates when idle gives you the persistent workspace and the near-zero idle cost at the same time — because snapshot-restore is what makes waking cheap enough to pause across every wait.

The summary

Long-running agents — deep-research crawls, multi-step coding tasks, background workers — are stateful and mostly idle: they need a workspace that persists across many tool calls, but they spend most of their wall-clock waiting on the model. The stateless fresh-VM-per-call model can't hold that state and forces you to rehydrate the workspace every call. The fix is one durable microVM per run: create it once (a ~179ms restore), `exec` and write against it for every step so state accumulates, and delete it at the end. Hibernate — snapshot-and-pause — across the LLM think-gaps so idle cost rounds to zero and the machine resumes mid-instruction on wake, no reboot or re-setup. Take snapshots at milestones as durable checkpoints, so a crash costs the last phase instead of the whole run. Fork at decision points to branch the accumulated state and explore alternatives from a warm baseline. And per-run isolation means a runaway agent — infinite loop, disk-fill, `rm -rf` — is contained to its own VM. You get a durable workspace that survives the whole run and a bill sized by real work, not by the hours your agent spent waiting for a token.

Every sandbox, hibernate, snapshot, and fork on PandaStack runs on the same Firecracker snapshot-restore path, and the core is open source under Apache-2.0 — so you can run the control-plane API and per-host agent on your own Linux KVM hosts and watch the wake timings yourself. For the snapshot mechanics start with /blog/microvm-snapshotting-warm-starts; for fanning out many agents at once, /blog/agent-swarm-parallel-sandboxes; for the fork-tree pattern, /blog/snapshot-fork-tree-of-thought.