Always-On vs Scale-to-Zero AI Agent Infrastructure

Here is the shape of almost every AI agent workload, and it's the shape nobody puts on the pricing page: bursty, and idle most of the time. A user asks a question, the agent thinks and runs a few commands for thirty seconds, and then the user goes to lunch. A coding agent works for two minutes and waits four for a review. A per-user workspace is touched twice a day. The compute you provisioned for that agent spends the overwhelming majority of its wall-clock doing absolutely nothing — and the entire cost/latency architecture of an agent platform comes down to one question: what do you run, and what do you pay, during the idle.

I'm Ajay — I build PandaStack, an open-source Firecracker microVM platform — so I spend my days on exactly this trade-off, from both ends: the latency users feel on the next request and the idle bill that quietly accrues while nobody's looking. There are three real points on the spectrum, and most teams pick one by accident rather than on purpose. This post lays them out honestly: always-on warm, hibernate-and-wake, and cold-create-per-request — what each costs while idle, what each costs in latency on the next request, and when your requirements actually force one over the others.

Idle is where the bill quietly bleeds you

Start with the economics, because it drives everything else. An agent's cost is not dominated by the seconds it computes — it's dominated by the seconds it exists while idle. A VM that runs for thirty seconds of real work and then sits warm for ten minutes waiting on a user or a model has spent 95% of its lifetime billing you to do nothing. Multiply that by every agent, every user session, every per-tenant workspace, and the idle time — not the busy time — is the line item that scales with your user count.

This is the uncomfortable arithmetic of agent infrastructure: your fleet size is set by concurrent idle sessions, not concurrent active work. If a thousand users each have a warm agent sitting around and only fifty are doing anything at any instant, you are paying for a thousand and using fifty. The busy fraction is the useful fraction; the idle fraction is the tax. Any architecture decision that doesn't start with 'what runs during idle' is optimizing the 5% and ignoring the 95%.

The three points on the spectrum

Every agent platform lands somewhere on a line between 'keep everything warm forever' and 'run nothing until the exact instant it's needed.' There are three points worth naming, because each is the right answer for a different workload.

Always-on warm: instant, and you pay for it

Keep a live VM per agent or per user, running the whole time the session is open. The next request lands on a machine that's already booted, already has the working directory and warm caches, and responds in the time it takes to run the command — no boot, no restore, nothing. This is the gold standard for latency and the worst case for idle cost: you are renting a hot CPU and holding its RAM for every second of every session, including all the seconds nobody's using it. Always-on is correct when instant response is a hard product requirement — an interactive REPL the user is typing into, a live collaborative session, anything where even a fraction of a second of wake latency is unacceptable.

Hibernate-and-wake: freeze the idle machine, resume on demand

The middle path, and usually the sweet spot. When the agent goes idle — hands control back to the model, waits on a user, parks between bursts — you snapshot the VM and pause it: the working directory, installed dependencies, running processes, and warm page cache all frozen exactly as they were. Nothing runs, so idle cost trends toward zero. When the next request arrives, you wake the VM and it resumes mid-instruction, exactly where it left off, with no reboot and no re-setup. You pay a wake latency instead of an idle bill — and because waking is a restore, not a cold boot, that latency is small. This is the model that gets you most of always-on's responsiveness at most of scale-to-zero's cost.

Cold-create per request: nothing exists until it's needed

The purest scale-to-zero: hold no VM at all between requests. When a request comes in, create a fresh machine, do the work, tear it down. Idle cost is genuinely zero because there is no idle machine — an idle agent is just a database row. The cost you pay is that every request stands up a machine from nothing, and the state from the last request is gone unless you serialized it out. This is the right model for stateless, one-shot work — evaluate this expression, run this scheduled job, handle this webhook — where there's no accumulated workspace to preserve and each request is independent.

Side by side

• Always-on warm — Idle cost: high; a live VM billing for every idle second of the session. Response latency: instant; the machine is already running. Best fit: interactive REPLs, live collaborative sessions, anything where sub-second wake is a hard requirement and idle cost is acceptable.
• Hibernate + wake — Idle cost: near-zero; the machine is frozen to disk and nothing runs while idle. Response latency: a wake, which is a restore — fast for a warm resume, up to tens of seconds for a full cold wake depending on the path. Best fit: long-running agents, per-user workspaces, bursty sessions with real think-gaps where you want state preserved but not a 24/7 bill.
• Cold-create per request — Idle cost: zero; there is no idle machine, just a database row. Response latency: a create every time — a snapshot-restore create, so still sub-second, but state doesn't survive between requests. Best fit: stateless one-shots, scheduled jobs, webhooks, code-interpreter calls with no accumulated workspace to keep.

The pattern most platforms actually converge on is a mix: cold-create for the stateless long tail, hibernate-and-wake for stateful sessions that go quiet, and always-on reserved only for the handful of workloads where instant response is genuinely non-negotiable. The mistake is picking one globally. Idle cost and wake latency pull in opposite directions, and different workloads sit at different points on that pull.

Why snapshot-restore makes scale-to-zero viable

Scale-to-zero has an obvious objection: if you tear everything down when idle, don't you pay a brutal cold-boot on the next request? Booting a VM from scratch — kernel init, systemd, service startup, dependency re-install — is genuinely slow, seconds at best, and if that's the price of waking, nobody would choose scale-to-zero for anything a user is waiting on. The reason the model works at all is that you don't cold-boot on wake. You restore.

On PandaStack, there is no warm pool of idle VMs. Every create is a snapshot-restore of an already-booted template — the restore step itself is around 49ms, and an end-to-end create is p50 179ms and p99 about 203ms. A true cold boot, kernel and all, happens only on the very first spawn of a template and lands around 3s; after that, you're paying restore prices. That's the mechanism that flips scale-to-zero from 'theoretically cheap, practically painful' to 'actually usable': waking a frozen machine or standing up a fresh one is a sub-second restore, not a multi-second boot, so the wake tax on scale-to-zero is small enough to live with for most workloads.

Hibernate rides the same path. Hibernate is snapshot-then-pause; wake is resume-from-snapshot. Because it freezes a running machine rather than shutting one down, the guest sees no reboot on wake — processes that were mid-execution keep running, open files stay open, no init handshake, no dependency re-install. A warm resume of a recently-frozen VM is fast; a full cold wake that has to stream the machine's memory back from object storage is slower, up to tens of seconds depending on the path. The exact number depends on how the wake resolves, but the shape is the same: you traded a 24/7 idle bill for a bounded, one-time latency you pay only when someone actually shows up.

The idle-hibernate / auto-wake pattern in code

Here's the middle path concretely. You keep one durable sandbox per session, hibernate it when it goes quiet, and wake it on the next request. The session's state survives every cycle because you froze the machine instead of destroying it, and you pay near-zero while it's frozen:

from pandastack import Sandbox

# One durable sandbox per session. Restores a baked snapshot (~179ms p50);
# the ttl is a backstop so a forgotten session reaps itself.
sbx = Sandbox.create(template="agent", ttl_seconds=3600, metadata={"session": sid})

def handle_request(user_input: str) -> str:
    # Someone showed up. Wake the frozen machine — a restore, not a cold boot.
    # The working dir, installed deps, and warm caches are exactly as we left them.
    sbx.wake()

    result = sbx.exec(agent.next_command(user_input), timeout_seconds=120)

    # Request handled. The session is about to go idle (user reads, thinks,
    # or wanders off). Don't rent a hot CPU for that gap: freeze the machine.
    sbx.hibernate()          # snapshot + pause; idle cost rounds to ~zero
    return result.stdout

# While hibernated, this session costs ~nothing — it's a frozen snapshot on
# disk, not a live VM. Wake is paid only when the next request actually lands.
# When the session ends for good, collect results and delete:
#   sbx.delete()

And here's the cost intuition the pattern is built on — a back-of-envelope sketch of why idle, not active work, is the thing to attack:

# Idle is the line item that scales with users, not with load.
sessions          = 1000     # concurrent open sessions
active_fraction   = 0.05     # ~5% are doing real work at any instant
session_hours     = 8        # hours the average session stays "open"

# Always-on: you pay for every open session for its whole lifetime.
always_on_vm_hours   = sessions * session_hours              # 8,000 VM-hours

# Scale-to-zero / hibernate: you pay ~only for the busy fraction.
# Idle sessions are frozen snapshots on disk, not live VMs.
hibernate_vm_hours   = sessions * session_hours * active_fraction  # 400 VM-hours

print(always_on_vm_hours, "vs", hibernate_vm_hours, "live VM-hours")
# 8000 vs 400  -> ~95% of always-on's compute is spent on idle you could freeze.
# (No dollar figures here on purpose — plug in your own instance pricing; the
#  RATIO is the point, and the ratio is set by how idle your agents really are.)

The numbers are illustrative, not a quote — the real ratio depends entirely on how idle your agents are. But that's exactly the point: the more bursty and idle-heavy your workload (which is most agent workloads), the larger the gap between what always-on bills you and what scale-to-zero does, and the more the wake-latency tax is worth paying.

How warm-pool platforms differ

A lot of sandbox platforms hit their fast-start numbers by keeping a pool of pre-booted VMs warm and ready to accept work. When a request comes in, one is plucked from the pool, so the create is instant. It's a legitimate way to get low latency — but the warm pool is, definitionally, idle compute you're paying for. Someone has to keep those VMs booted and resident so they're ready, and 'kept booted and resident while doing nothing' is the exact definition of the idle cost we started with. The fast start is real; it's paid for by an idle pool sitting behind it.

PandaStack's design removes the pool from the equation: there is no warm pool of idle VMs, because every create is a snapshot-restore fast enough that a pool isn't needed to hit sub-second. Idle sessions aren't held-warm VMs — they're frozen snapshots (hibernate) or nothing at all (cold-create), so idle trends toward zero cost rather than being a standing line item. The trade is honest and worth stating plainly: you accept a small restore latency on wake in exchange for not paying for a warm floor. If your workload is mostly idle, that's a great trade. If it's mostly active, the warm-pool approach's idle cost is smaller and the calculus shifts. (Exact warm-pool behavior varies by platform — verify against their docs.)

When instant-response requirements force always-on

I don't want to oversell scale-to-zero, because there's a real cost and it's the wake latency. Every scale-to-zero request pays for the machine to come back — a sub-second restore in the good case, up to tens of seconds for a full cold wake that streams memory back from object storage. For most agent work that's invisible: the user is already waiting on the model, a few hundred milliseconds of wake disappears into the LLM's own latency. But not always.

• Interactive REPLs and live terminals — a user typing commands and watching output expects the machine to be there the instant they hit enter. A wake on every keystroke-driven command is felt. Keep it always-on for the duration of the interactive session.
• Sub-100ms hard SLAs — if your product promises a response faster than a restore can complete, you cannot pay a wake tax on the request path. Always-on, or a warm floor, is the only way to hit it.
• Sustained high-throughput streams — a session that's continuously busy has no idle to reclaim. Hibernating a machine that's active every second just adds snapshot overhead for no idle savings. Leave it warm.
• Very short, very frequent think-gaps — if the agent pauses for 200ms between steps, the snapshot-and-wake overhead can exceed the idle you'd save. Hibernate across the long waits, not the blinks.

The clean way to express 'this one stays warm' is opt-in. On PandaStack, marking a sandbox `persistent: true` exempts it from the idle reaper, so it stays alive when you genuinely want always-on — the interactive session, the hot workspace, the SLA-bound endpoint — while everything else defaults to freeze-when-idle. You're not choosing one model for the whole fleet; you're choosing per workload, which is the only honest way to do it.

Putting it together

Agents are bursty and mostly idle, and that single fact decides your cost/latency architecture. Always-on warm gives instant response and bills you for every idle second — right when instant is a hard requirement, wrong as a default. Cold-create per request gives zero idle cost and a fresh machine every time — right for stateless one-shots, wrong when state has to survive. Hibernate-and-wake is the middle path that preserves the session's state, drops idle cost to near-zero, and pays only a bounded wake latency on the next request. What makes the whole spectrum practical is that on a snapshot-restore platform, waking isn't a cold boot — it's a sub-second restore — so scale-to-zero stops being a painful trade and becomes the sensible default, with always-on reserved, opt-in, for the workloads that truly can't tolerate a wake. Attack the idle, keep the state, and let the fast path make the trade cheap.

The core of PandaStack is open source under Apache-2.0, so you can run the control-plane API and per-host agent on your own Linux KVM hosts and measure the wake and restore timings against your own workload. For the mechanism that makes idle-free fast starts work, see /blog/snapshot-restore-vs-warm-pools; for the durable-workspace-plus-hibernate pattern on long tasks, /blog/microvm-llm-agent-long-running-tasks; for holding one machine across a whole session, /blog/long-running-sandboxes-for-ai-agents; and for how the idle bill compounds at scale on warm-pool platforms, /blog/e2b-cost-at-scale.