AI Agent Runtime Infrastructure: A Guide

There's a lot of writing about the agent loop — planning, tool calls, memory, orchestration frameworks with confident opinions about who calls whom. There's much less about the thing every one of those loops eventually hits: the moment the agent runs code somewhere. That somewhere is the runtime layer, and it's the part of the stack that decides whether a confidently-wrong model action is a shrug or an incident, whether your agent feels snappy or laggy, and whether ten thousand tenants can share a fleet without reading each other's data. This is a reference for that layer — the checklist of capabilities a serious agent runtime has to provide, what to look for in each, and where PandaStack (the open-source Firecracker microVM platform I build) lands. Full disclosure up top so you can weight everything accordingly.

The framing I want you to leave with: the runtime is not an implementation detail you pick last. It's the substrate that determines the ceiling on everything above it. A brilliant agent on a runtime that cold-boots in four seconds feels sluggish; a mediocre agent on a runtime that forks state in half a second can do best-of-N search the fancy one can't. If you're building or buying an execution layer for agents, these are the nine things it has to get right. I've linked out generously — this post is the map, and each capability has a deeper post behind it.

The nine-capability checklist

Here's the whole thing in one place. The rest of the post takes each in turn — what to look for, why it matters for agents specifically, and where PandaStack sits. If you only read one list in this post, read this one.

1. Isolation boundary — the kernel-level answer to 'what happens when the model runs a bad line?' Process < container < microVM, and untrusted agent code demands VM-grade.
2. Fast create — cold-start shape is the agent-loop's felt latency. Snapshot-restore vs. warm pools vs. cold boot.
3. Statefulness — ephemeral-per-task vs. persistent-per-session, plus hibernate/wake so idle sessions trend toward zero cost.
4. Networking + egress control — a sandbox that can reach your internal network is a sandbox that can exfiltrate from it. Default-deny outbound.
5. Filesystem + durable volumes — copy-on-write ephemeral rootfs for throwaway work; durable volumes for state that must survive the VM.
6. Snapshot + fork — branch agent state cheaply for best-of-N search and warm starts. The capability that's painful to bolt on later.
7. Multi-tenancy + resource limits — one tenant per boundary, hard CPU/memory ceilings, so a runaway job is capped to one VM you can kill.
8. Observability — logs, metrics, exec/PTY. If you can't see what the agent's code did, you can't debug it or bill for it.
9. Build vs. buy vs. self-host — the org decision that constrains all eight above. Open-source and self-hostable, or hosted-only.

1. Isolation boundary: process vs. container vs. microVM

This is the load-bearing one, so it goes first. The instant a model writes the code you're about to run, that code is untrusted — not malicious necessarily, just not something a human reviewed before it executed. The question 'where does that code run?' is not plumbing; it's the security model of your product. There are three honest answers in increasing order of how well you'll sleep, and the difference between them is the kernel.

• In-process (exec / eval in your app) — no boundary at all. A bad line reaches everything your process can reach: your database, your secrets, your filesystem. Fine for a demo, catastrophic the moment it ships. Prototypes ship.
• Containers (namespaces + cgroups + seccomp) — fast, cheap, dense, and sharing the host's one kernel across every tenant. A container is a polite suggestion to the kernel about what a process should see; a kernel-level escape ignores the suggestion and takes the host and every neighbor with it. Right for code you wrote, risky for code a model wrote.
• Hardware-virtualized microVMs (Firecracker, Kata) — each sandbox boots its own guest kernel isolated by KVM. Guest code makes syscalls against its own kernel and never touches the host's; the only path out is a tiny, heavily-audited virtual device model behind a jailer. This is the right default for arbitrary agent-written code — the same model AWS Lambda uses to run everyone's functions on shared hardware.

What to look for: does the runtime give the guest its own kernel, or does it share yours? If the answer is 'shared kernel with hardening' (seccomp, dropped caps, user namespaces), that's a real improvement over nothing, but it's mitigation, not isolation — one kernel privilege-escalation bug reachable through an allowed syscall still threatens the host. The honest caveat cuts both ways: 'microVM' is not 'immune' — VMMs have had bugs, KVM has had bugs. The accurate claim is a dramatically smaller, better-audited attack surface than the full Linux syscall interface a container leaves exposed. The full ranking is in /blog/firecracker-vs-docker and the container-specific case is /blog/firecracker-vs-podman.

Where PandaStack lands: every sandbox, database, and hosted app is its own Firecracker microVM with its own guest kernel, run under a jailer that chroots and drops privileges. The boundary is VM-grade by default, not opt-in. That's the whole bet — a kernel bug in the agent's code stays inside one disposable guest.

2. Fast create: cold-start shape is the agent-loop's felt latency

The create call sits inside the agent loop, and you may hit it dozens of times per trajectory — per task, sometimes per step. So its latency is not a benchmark curiosity; it's how the product feels. If a fresh, isolated computer costs four seconds, you'll be tempted to nurse one long-lived box and reuse it across tasks — which quietly rebuilds the shared-state problems the boundary was supposed to solve. If a fresh computer costs a couple hundred milliseconds, disposable-per-task becomes the default and the whole architecture gets simpler.

There are three families of answer, and they get conflated constantly. Cold boot: actually boot a kernel and userland every time — correct, isolated, and slow (seconds). Warm pool: keep idle VMs pre-booted and hand one out on demand — fast to hand out, but you pay to keep the pool warm whether or not anyone's using it, and idle capacity is a real line on the bill. Snapshot-restore: boot once, snapshot the running machine, and restore that snapshot on every create — fast like a warm pool, but with no idle fleet to pay for. The trade-offs are their own post: /blog/snapshot-restore-vs-warm-pools.

What to look for: ask which of those three a vendor actually does, because 'fast' is the easiest metric to mis-measure. A warm-pool 'sub-100ms' is not the same claim as a cold-boot 'sub-100ms,' and 'in your region on your template' is not the same as a headline figure from a launch blog. Benchmark it on your own workload — including PandaStack's numbers. Treat every advertised cold-start as a hypothesis to test.

Where PandaStack lands: snapshot-restore on every create, with no warm pool of idle VMs. The restore step is around 49ms; an end-to-end create is p50 179ms and p99 about 203ms. A true cold boot happens only on the very first spawn of a template — around 3s — after which every create pays restore prices. The design consequence that matters: idle tenants cost nothing because there's no pool sitting warm, and a disposable-computer-per-task is a latency rounding error rather than a budget line.

3. Statefulness: ephemeral, persistent, and hibernate/wake

Agents come in two shapes and your runtime needs to serve both. The one-shot task — run this, give me the result, throw the machine away — wants an ephemeral sandbox: nothing survives, no residual state leaks from one run into the next, idle cost is zero because there's nothing left running. The interactive or long-horizon agent — a coding session that installs packages and edits files over twenty minutes, a research agent that pokes at a workspace all afternoon — wants a persistent sandbox that holds state across steps so you're not re-cloning the repo every turn.

What to look for: can you choose per-sandbox, or is the runtime opinionated one way? Ephemeral-only forces you to re-provision state constantly; persistent-only means idle sessions burn money until something reaps them. The best answer is both, plus a third option for the long-running case that's neither hot nor gone: hibernate. A hibernated agent session is snapshotted and paused so it costs (almost) nothing while idle, then wakes on the next request with its state intact. That's what makes 'a warm workspace per user' economically sane at scale. The deeper treatment is /blog/long-running-sandboxes-for-ai-agents.

Where PandaStack lands: sandboxes are ephemeral by default (a TTL reaps them; the SDK context manager destroys them on block exit). Mark one `persistent=True` and the idle reaper leaves it alone — that's your long-lived session. Hibernate snapshots and pauses a sandbox, wake resumes it, so an idle interactive agent trends toward zero cost while a returning user gets a warm resume rather than a cold rebuild. The rule that never bends: one tenant per persistent sandbox, forever — the instant you reuse one across two tenants you've erased the boundary by hand.

4. Networking and egress control

Isolating the machine contains what the code can do to your host. It does nothing about where the code can reach out to. An agent's code with unrestricted outbound can hit your internal network, your metadata endpoint, a data-exfiltration server, or a crypto-mining pool — none of which the kernel boundary stops, because they're all just packets leaving a well-isolated VM. Egress control is a separate capability from isolation, and a runtime that nails the first and ignores the second is only half a sandbox.

What to look for: per-sandbox network namespaces (so one tenant's traffic can't be seen or spoofed by another), and a default-deny-outbound posture you can allowlist against — the task needs PyPI and one API, not the whole internet and your VPC. Ask how the runtime provisions networking, too, because naive per-sandbox netns setup is slow (creating a namespace, veth pair, and iptables rules cold is on the order of 100ms) and that cost lands right in your create latency. The threat model and the controls are in /blog/controlling-network-egress-untrusted-code.

Where PandaStack lands: each agent pre-allocates a pool of 16,384 /30 subnets (a per-sandbox network namespace + veth + tap, built ahead of time), so per-sandbox networking is patched in single-digit milliseconds on create instead of built cold. Every sandbox gets its own isolated netns; egress is filtered per sandbox. The pre-allocation is why VM-grade networking doesn't show up as a latency tax.

5. Filesystem and durable volumes

Two different needs hide under 'the sandbox has a filesystem.' The first is a throwaway working disk: the agent clones a repo, writes files, makes a mess, and it all vanishes when the VM dies. You want that disk to be cheap to create — copy-on-write, so a fresh rootfs is an O(metadata) clone that shares blocks with the template until something is written, not a multi-gigabyte copy on every create. The second is state that must outlive the VM: a database's data directory, a user's uploaded files, anything you'd be sad to lose when the ephemeral machine is reaped.

What to look for: copy-on-write on the ephemeral rootfs (fast create, small disk footprint), and a separate durable-volume primitive for persistent state so you're not conflating 'the machine' with 'the data.' Conflating them is how teams end up afraid to destroy a sandbox because the data lives on the same disk — which defeats the disposability that made the whole model safe.

Where PandaStack lands: the rootfs is an XFS-reflink copy-on-write clone of the template disk — data is shared until a write diverges it, so create is fast and disk-cheap. Durable volumes are a separate primitive that survive the sandbox lifecycle; managed databases use them (more on that in the multi-tenancy section). The split is deliberate: destroy the VM freely, keep the volume when you mean to.

6. Snapshot and fork: branching agent state

This is the capability that's easy to overlook and painful to bolt on later, and it's where a microVM runtime earns its keep in a way containers can't easily match. Agents are unreliable per-attempt but much stronger in aggregate: generate five candidate fixes, run each, keep the one whose tests pass. The naive way spins up five environments and re-runs setup five times. The fast way warms one environment — repo cloned, dependencies installed, dataset loaded — then forks it N times via copy-on-write and explores each branch in parallel. Every branch inherits the warm state for free.

A fork clones a running sandbox with copy-on-write: guest memory is shared through MAP_PRIVATE (the kernel copies a page only when a child writes it) and the rootfs is cloned with a reflink (O(metadata), data shared until divergence). Snapshotting does the same trick across time instead of across branches — freeze a warm, configured machine and restore it later or elsewhere as a warm start. What to look for: are snapshot and fork first-class primitives, or something you'd have to reconstruct from raw VM images? If best-of-N or tree-search is central to your agent, weight this heavily. The walkthrough is /blog/how-ai-agent-sandboxes-work.

Where PandaStack lands: snapshot and fork are exposed as SDK primitives. A same-host fork lands in roughly 400–750ms (local reflink + memory restore); a cross-host fork runs about 1.2–3.5s (download plus restore). That's cheap enough to make tree-of-thought physical — each branch is a real isolated machine that diverged from shared warm state, and dead ends are discarded for the cost of a kill().

7. Multi-tenancy and resource limits

The moment more than one customer's agents share your fleet, two failure modes appear. The noisy neighbor: one tenant's agent writes a runaway loop or a 40GB allocation, pins a core or triggers the host OOM killer, and a neighbor's perfectly reasonable job gets degraded or killed for a mistake it didn't make. The cross-tenant leak: a worker process that has held tenant A's data and your platform credentials runs tenant B's code, and that code reads what's still in memory. Both are architecture problems, not bugs you can patch.

What to look for: is the isolation boundary also the tenancy boundary (one tenant per VM, so a leak is physically impossible rather than policy-enforced), and are resource limits hard ceilings rather than best-effort? A microVM with a memory and CPU budget baked into its template turns a 40GB allocation into 'that one guest OOM-kills its own process' instead of 'the host evicts whatever it evicts.' The runaway job's entire blast radius becomes one disposable VM you can kill. The per-tenant pattern in depth is /blog/microvm-per-tenant-analytics-query.

Where PandaStack lands: it's multi-tenant by construction — orgs and members up top, and each sandbox is its own VM with CPU/memory fixed by its template, so a runaway job hits its own ceiling and nobody else's. A managed database is a dedicated microVM plus its own durable volume — not a schema carved out of a shared instance — created in 30–90s (the create blocks until Postgres is genuinely accepting connections). So a per-tenant database is a hardware-enforced boundary, not a WHERE clause a query can forget. Networking capacity is rarely the constraint (16,384 subnets per agent); the practical ceiling is host memory and CPU.

8. Observability: logs, metrics, exec/PTY

If you can't see what the agent's code did, you can't debug it, can't let the model self-correct on real failures, and can't bill for it. The runtime has to surface three things cleanly. Exec results — stdout, stderr, and the exit code, because the agent needs to observe real failures (a non-zero exit, an error on stderr) to fix them, and hiding failures behind vibes is how agents loop forever being confidently wrong. Logs — the machine's console and the app's own output, streamable so you can follow a long build. And an interactive path — a PTY / terminal channel for when a human needs to drop into the guest and poke around.

What to look for: structured exec results (not just a merged text blob), streaming logs, a real PTY for interactive debugging, and per-sandbox metrics you can attribute to a tenant for billing and abuse investigation. Tag every sandbox with a tenant id so a runaway run, a suspicious egress attempt, or a billing line traces back to exactly one customer.

Where PandaStack lands: exec returns stdout/stderr/exit_code as structured fields; there's an SSE exec-stream for live output and a WebSocket PTY (xterm.js-compatible) for interactive sessions. Firecracker console logs and the app's runtime logs are both tailable, lifecycle events stream over SSE, and per-sandbox metrics feed a ClickHouse sink for attribution. The guest is reached over a vsock bridge with an SSH fallback — low-latency exec without exposing a network service.

9. Build vs. buy vs. self-host

This is the org-level decision that constrains all eight capabilities above, and it's mostly about who owns the KVM hosts and whether you can read the source. Building your own on raw Firecracker is absolutely doable — the primitives are open — but you're signing up to implement snapshot-restore, a networking pool, a scheduler, egress control, and multi-tenant billing yourself, which is a platform, not a weekend. Buying a hosted sandbox is fastest to first-run but puts your untrusted-code execution on someone else's fleet, on their pricing, with their data-residency story. Self-hosting an open-source platform is the middle path: someone else built the platform, you run it on your own hosts.

What to look for: is the core open-source and self-hostable, or hosted-only? For a lot of agent products the answer matters for real reasons — data residency, air-gapped or regulated environments, cost at scale, and not wanting your execution layer to be a vendor you can't inspect or move off of. The honest build-vs-buy math, including when rolling your own is the right call, is in /blog/build-vs-buy-firecracker-sandbox, and the self-host path specifically is /blog/self-hosted-code-execution-sandbox.

Where PandaStack lands: open-source and self-hostable — you run the agent and its Firecracker microVMs on your own KVM hosts, with the control plane (a Go API plus Postgres) coordinating them. The architecture is a clean three-layer model, which is worth understanding because it's roughly what you'd have to build yourself if you went the raw-Firecracker route: a control plane (REST API + Postgres, with a deterministic scheduler and multi-region coordination), a per-host agent (snapshot-restore boot path, the NATID networking pool, snapshot store, Firecracker lifecycle), and the Firecracker microVMs themselves (guest kernel, per-VM netns, copy-on-write rootfs). You can take the hosted service or run the whole thing yourself.

Tying it together in code

Half these capabilities show up in a handful of SDK calls. Here's a single flow that exercises the ones a real agent leans on: create with a template, a TTL backstop, and tenant metadata (fast create + statefulness + multi-tenancy attribution); exec with a timeout (observability + resource limits); snapshot and fork a warm state (branching); and destroy (ephemeral teardown). The latency numbers in the comments are only ever PandaStack's own measured figures.

from pandastack import Sandbox

# 1) FAST CREATE + STATE + TENANCY: restore a baked snapshot (~49ms restore;
#    ~179ms p50 / ~203ms p99 end-to-end). ttl_seconds is the backstop that
#    reaps a leaked VM even if our process crashes; metadata attributes the
#    run to one tenant for audit + billing.
base = Sandbox.create(
    template="code-interpreter",
    ttl_seconds=900,
    metadata={"tenant_id": "acme", "task": "fix-flaky-test"},
)

# 2) OBSERVABILITY + LIMITS: exec returns structured stdout/stderr/exit_code.
#    timeout_seconds is the circuit breaker for the while-True the model will
#    eventually write. It kills THIS command, no one else's.
base.filesystem.write("/workspace/setup.sh", "git clone --depth 1 https://example.com/repo /workspace/repo\n")
setup = base.exec("bash /workspace/setup.sh", timeout_seconds=120)
assert setup.exit_code == 0, setup.stderr
base.exec("cd /workspace/repo && pip install -r requirements.txt", timeout_seconds=300)

# 3) SNAPSHOT: freeze the warm, configured machine as a reusable warm start.
snap = base.snapshot()   # restore this later to skip clone + install

# 4) FORK (best-of-N): branch the warm state N times via copy-on-write.
#    Same-host fork ~400-750ms; each child inherits the install for free.
def try_candidate(patch: str) -> dict:
    child = base.fork()
    try:
        child.filesystem.write("/workspace/repo/fix.patch", patch)
        child.exec("cd /workspace/repo && git apply fix.patch")
        r = child.exec("cd /workspace/repo && pytest -q", timeout_seconds=120)
        return {"passed": r.exit_code == 0, "out": r.stdout[-1000:]}
    finally:
        child.kill()   # dead-end branches cost a kill(); the trunk lives on

results = [try_candidate(p) for p in generate_candidate_patches(n=5)]

# 5) EPHEMERAL TEARDOWN: destroy the trunk. Memory, rootfs, netns, and any
#    half-written state vanish with the VM. Nothing leaks to the next task.
base.destroy()
print(f"{sum(r['passed'] for r in results)}/5 candidates passed; snapshot={snap.id}")

That's eight of the nine capabilities visible in one snippet — the ninth (isolation) is the substrate underneath all of it, silently making every one of those calls a VM-grade boundary instead of a shared-kernel suggestion. The networking and egress controls are per-sandbox and mostly configured at the platform level rather than per-call, which is exactly where you want them: not something the agent's code can talk its way around.

The bottom line

The runtime layer is not the glamorous part of an agent stack, but it's the part that sets the ceiling on everything above it. Nine capabilities decide whether that ceiling is high: a VM-grade isolation boundary (because the code is untrusted the moment a model wrote it), fast create (because cold-start shape is felt latency), the right statefulness model with hibernate/wake, per-sandbox networking with default-deny egress, copy-on-write filesystems plus durable volumes, cheap snapshot and fork for branching, one-tenant-per-boundary multi-tenancy with hard limits, real observability, and a build/buy/self-host answer you can live with. Evaluate a runtime against these nine and you'll stop being surprised by it in production. PandaStack's bet is to provide all nine on an open-source, self-hostable Firecracker substrate — snapshot-restore on every create (179ms p50), copy-on-write forking (~400ms same-host), and a boundary that keeps a confidently-wrong model contained to one disposable guest. Whatever you pick, pick it deliberately, and benchmark the numbers — including ours — on your own workload.