Running User-Uploaded Automation Scripts Safely

It starts as a small feature. Your product does data sync, or workflow automation, or webhook routing, and a customer asks: "Can I just write a little script to transform the payload before it lands?" You ship a "Run Code" step — a text box where users paste Python or JavaScript — and it becomes the stickiest thing you've ever built. Zapier has code steps. Every ETL tool has custom transforms. Low-code platforms have "run this function" blocks. Webhook gateways let you rewrite payloads inline. The feature is a magnet. It is also the moment you agreed to run arbitrary code, written by strangers, at whatever scale they trigger it, on your infrastructure.

I'm Ajay — I build PandaStack, a Firecracker microVM platform — so I spend most of my time thinking about where the blast radius of "run this user's code" actually stops. This post is about running user-uploaded automation scripts so that one user's script is contained to one user's script: each execution gets its own ephemeral microVM, receives its inputs, runs under hard CPU/memory/time limits with controlled egress, hands back its output, and is destroyed. No shared worker, no shared kernel, no shared blast radius. First, though, the shortcuts that look tempting and quietly end in an incident.

The shortcuts that look fine until they don't

There are three popular ways to run user scripts, and they're popular because they're easy to ship in an afternoon. Each of them trades away something you'll wish you had back the first time a user's script is hostile, buggy, or just greedy.

Shortcut 1: exec it in a shared worker

The naive version runs each user's code inline in your worker process — `exec()` the Python, `eval` the JS, `subprocess` the shell. It works flawlessly in the demo. Then a user writes `while True: pass` and pegs a core forever. Another allocates a 40GB list and OOM-kills the worker, taking every other user's in-flight job down with it. A third writes `import os; print(os.environ)` and now holds your database URL, every integration's API key, and your queue credentials. The user's code is running in the same process as your secrets, which is not an isolation boundary — it's a shared bag of everyone's credentials with a text box attached.

Shortcut 2: a bare Docker container

The next instinct is a container per run. Better — you get cgroup limits and a namespace — but a container is a normal Linux process the host kernel has agreed to treat specially. Every user's container calls into the same shared host kernel, across 300-plus syscalls plus ioctls and reachable drivers. One kernel privilege-escalation bug reachable from a syscall the user is still allowed to call, and that user is root on your host, next to everyone else's jobs. Container escapes are a known, recurring class of bug (CVE-2019-5736 let a container overwrite the host runc binary), and cgroups leak at the kernel level — a fork storm or page-cache thrash in one container still degrades its neighbors. A bare container is a real improvement and still the wrong boundary for code you never reviewed. See /blog/why-docker-is-not-a-sandbox for the full argument.

Shortcut 3: a language-level sandbox (RestrictedPython, vm2)

The cleverest-looking shortcut is a same-process sandbox: RestrictedPython to neuter dangerous builtins, or vm2 to run untrusted JavaScript in a "secure" Node context. These feel elegant because there's no VM to manage — just a library. They are also the most reliably escaped. The problem is fundamental: you're trying to build a security boundary inside a runtime that was never designed to have one, using the same interpreter the attacker is running in. Python's introspection (`__class__.__bases__.__subclasses__()`, `__globals__` walking) reaches back to dangerous objects through paths the allowlist forgot. vm2 became the canonical cautionary tale — it had more sandbox-escape CVEs than some projects have releases, and its maintainer eventually deprecated it outright with the blunt advice that it should not be used for running untrusted code. A language-level sandbox is a speed bump, not a wall.

The fix: one script run, one ephemeral microVM

Move the boundary down to hardware. When a user's script needs to run, create a fresh Firecracker microVM, write the script and its inputs into the guest, execute it under a hard timeout with pinned CPU/memory, capture the output, and destroy the VM. The next run — even the same user's next run — starts from a clean, baked snapshot. Nothing survives except the output you explicitly read back. Each microVM has its own guest kernel under KVM hardware virtualization, so the escape surface is a handful of emulated virtio devices, not the full Linux syscall table a container exposes. This is the same isolation model AWS Lambda uses to run untrusted code from thousands of customers on shared fleets.

Here's the executor your automation engine calls when a user's script step fires. Note the data flow: inputs go in as a file, the script runs against them, structured output comes back as a file, and the `with` block destroys the VM — and every secret, temp file, and stray child process — the moment `exec` returns or the TTL reaps it.

from pandastack import Sandbox
import json

def run_user_script(user_id: str, script: str, payload: dict) -> dict:
    """Run one user's automation script in its own throwaway microVM."""
    # Fresh VM per run. No shared kernel, no state from any prior run.
    with Sandbox.create(
        template="code-interpreter",
        ttl_seconds=120,                     # hard backstop: a wedged run reaps itself
        metadata={"user_id": user_id, "kind": "automation-step"},
    ) as sbx:
        # Inputs go IN as a file the script reads. No secrets in this VM.
        sbx.filesystem.write("/workspace/input.json", json.dumps(payload))
        sbx.filesystem.write("/workspace/step.py", script)

        # Run the user's code with a hard wall-clock timeout (circuit breaker).
        result = sbx.exec(
            "python3 /workspace/step.py",
            timeout_seconds=30,
        )
        if result.exit_code != 0:
            return {"ok": False, "error": result.stderr[-4000:]}

        # Output comes OUT as a known file the script agreed to write.
        raw = sbx.filesystem.read("/workspace/output.json")
        return {"ok": True, "output": json.loads(raw)}
    # VM (and every temp file + child process) is gone here

Tag the sandbox with `metadata={"user_id": ...}` so your audit log and usage billing can attribute every execution. Pass inputs as a file rather than interpolating them into the script string — you avoid injection and quoting hazards, and the contract is clean: the script reads `/workspace/input.json`, writes `/workspace/output.json`, and your host does the rest. Always set both `timeout_seconds` on the exec and `ttl_seconds` on create: the first bounds the code's wall-clock, the second bounds the VM in case the code wedges in a way `exec` doesn't catch (a detached child, a blocked syscall).

Resource limits, timeouts, and the noisy-neighbor problem

The single most common production incident with user scripts isn't a security breach — it's availability. One user's `while True` or accidental O(n^2) over a big list pegs every core, and suddenly every other user's automation is queued behind it. On a shared worker, cgroups help but leak: the kernel's page cache, lock contention, and scheduler are shared, so one greedy job still drags its neighbors down. In a per-run microVM, the vCPU count and guest RAM are hard ceilings set by the hypervisor at the VM boundary. A user script that pegs its cores pegs its cores, inside its own machine, and the box next to it never notices.

Layer your limits so each one is enforced below the thing above it:

• Wall-clock timeout — `timeout_seconds` on the exec kills the process; a user's infinite loop dies at the deadline instead of running until the heat death of the universe.
• VM TTL — `ttl_seconds` on create is the backstop for when the process forks a detached child or blocks in a syscall the exec-timeout can't reach; the whole VM reaps itself.
• CPU + memory — pinned by the template's baked snapshot (the hypervisor fixes vCPU/RAM at the VM boundary), so a runaway saturates only its own allocation.
• Output truncation — tail stdout/stderr and cap the output file you read back; "my script printed a 2GB log" should degrade to a truncated log, not an OOM on your executor.
• Concurrency per user — cap how many script runs one user can have executing at once at your dispatcher, so one user can't fan out ten thousand VMs and crowd out everyone else's fair share.

Set inner limits inside the guest too, if you want belt-and-suspenders. A small wrapper can apply an OS-level memory ceiling and its own timeout before it ever hands control to the user's code — a second wall behind the hypervisor's:

#!/usr/bin/env bash
# runner.sh — inner guard rails, run INSIDE the guest before the user's code.
# The microVM already caps total RAM/CPU; this bounds a single process too,
# so one runaway step can't eat the whole guest's allocation either.
set -euo pipefail

# Cap address space at 512 MiB and file size at 64 MiB for the child process.
ulimit -v $((512 * 1024))     # virtual memory, KiB
ulimit -f $((64 * 1024))      # max file size, KiB
ulimit -u 256                 # max user processes (fork-bomb brake)

# Hard wall-clock timeout as a second line behind the exec timeout.
exec timeout --signal=KILL 30s python3 /workspace/step.py

Controlling egress so a script can't phone home

A user script that can reach the open internet can exfiltrate whatever it can read, scan your internal network, or hammer a third party from your IP. The default should be deny: a script gets no network unless its step explicitly needs one, and even then only to an allowlist. Because each microVM sits in its own network namespace with its own egress path (PandaStack gives every sandbox a dedicated netns, veth pair, and tap from a pool of 16,384 /30 subnets per agent — no shared bridge to pivot through), egress policy is per-VM, not global. You apply the firewall to the sandbox's namespace and it binds to exactly that one run.

# Per-sandbox egress: deny-by-default, then allow only what the step needs.
# Applied to THIS run's network namespace, so it binds to one VM only.

# 1) Drop all outbound by default.
iptables -A OUTPUT -j DROP

# 2) Block the cloud metadata endpoint outright (a classic exfil target).
iptables -I OUTPUT -d 169.254.169.254 -j DROP

# 3) Allow DNS + HTTPS only to an explicit allowlist for this step.
iptables -I OUTPUT -p udp --dport 53 -j ACCEPT
iptables -I OUTPUT -p tcp -d api.trusted-partner.example --dport 443 -j ACCEPT

The important part isn't the exact rules — it's that egress is scoped to one disposable VM and defaults closed. A script that was supposed to transform a payload has no business opening a socket to an unfamiliar host, and if it tries, the packet dies in its own namespace. For the deeper treatment, see /blog/controlling-network-egress-untrusted-code. And whatever you do, never inject your platform's secrets into the guest environment — a script isolates execution, not your credentials; if the code shouldn't see a key, it shouldn't be in the VM.

The four approaches, side by side

It's worth being blunt about the trade each option makes, because the easy ones are easy for a reason and the reason is the thing that bites you:

• Isolation boundary — Shared worker: none, your process. Bare container: shared host kernel (300-plus syscalls). Language sandbox: none, the attacker's own interpreter. microVM-per-run: a separate guest kernel under hardware virtualization.
• Noisy neighbor — Shared worker: a runaway takes down every job on the box. Bare container: cgroups leak at the kernel level, so neighbors still degrade. Language sandbox: runs in your process, so a busy loop pegs your worker. microVM-per-run: RAM/CPU fixed by the hypervisor per VM; a runaway is contained to its own machine.
• Secret exposure — Shared worker: user code shares a process with all your secrets. Bare container: better, but a kernel escape reaches the host and everything on it. Language sandbox: shares your process and its env. microVM-per-run: inject nothing the code shouldn't see; a compromised run holds only its own inputs.
• Escape difficulty — Shared worker: no escape needed, it's already inside. Bare container: one reachable kernel CVE (a known, recurring class). Language sandbox: introspection escapes are routine — vm2 had more escape CVEs than releases. microVM-per-run: a VMM device-model or KVM CVE, a small and heavily audited surface.
• Teardown — Shared worker: state and leaked secrets linger between runs. Bare container: needs disciplined cleanup; leftover layers and mounts leak. Language sandbox: globals and monkeypatches bleed into the next run. microVM-per-run: destroy the VM and everything in it is gone, deterministically.
• Setup cost — Shared worker: trivial (and that's the trap). Bare container: moderate. Language sandbox: trivial (and that's the trap). microVM-per-run: an SDK call and a p50 179ms create — real, bounded, and worth it.

"But a VM per run is too slow"

This is the reflex, and it used to be right — nobody wants a multi-second VM boot for a webhook transform that runs thousands of times a minute. Per-run VM isolation is practical now because of snapshot-restore: instead of cold-booting a kernel, every create restores a baked snapshot of an already-booted machine. An end-to-end create is p50 179ms and p99 about 203ms. A true cold boot — around 3s — happens only on the very first spawn of a template, after which every create takes the fast path. The isolation you'd want for untrusted code stopped costing what it historically cost.

For scripts that need a specific pre-warmed runtime — a heavy dependency graph, a loaded library set, a configured interpreter — snapshot one set-up VM once and fork it per run instead of re-installing each time. A same-host fork is 400–750ms and shares the parent's memory copy-on-write, so a hundred forks off one warm parent don't cost a hundred full copies of RAM; a cross-host fork is 1.2–3.5s. If a script step needs to talk to a durable store, a managed database provisions in 30–90s once and persists across runs. The point isn't that any single number is magic — it's that per-run isolation now lands in the same latency bracket as dispatching to a warm worker, without the shared blast radius that made the warm worker cheap. For the broader decision, start at /blog/how-to-sandbox-untrusted-code.

Where PandaStack fits

PandaStack is built for exactly this shape of problem: run someone else's code, contain it to one disposable machine, tear it down. Each script run is its own Firecracker microVM — its own guest kernel isolated by KVM, not a shared-kernel container. Inputs go in and outputs come back through a simple filesystem API; hard timeouts and VM TTLs bound runaway code; per-VM CPU/RAM ceilings kill the noisy-neighbor problem; per-sandbox network namespaces (16,384 /30 subnets per agent) make egress policy scoped and default-closed. Creates land at ~179ms p50 (~203ms p99) with no warm pool, so a clean VM per run is affordable rather than aspirational.

The part that matters if you're weighing build-vs-buy: the core is open source and Apache-2.0, so you can self-host it on your own Linux KVM hosts and keep the user-code boundary on infrastructure you control and audit — or use the hosted API and skip running agents yourself. None of this makes a microVM an absolute boundary (VMM device-model logic bugs, KVM escape CVEs, and microarchitectural side channels are real and yours to reason about), but it does mean the boundary under your users' scripts is a hypervisor, not a text box wired directly into your worker's process. If you're currently running user scripts in an `exec()`, a bare container, or — please, no — vm2, that's the upgrade.