Running Customer UDFs Safely in a SaaS with microVMs

Somewhere in your product roadmap there is a feature that says "let customers write their own logic." A custom validation hook. A Zapier/n8n-style code step. A Retool or Airtable script. A data-transformation UDF that maps their weird CSV onto your schema. It's one of the highest-leverage features a SaaS can ship — and it quietly turns your platform into a code-execution host. The instant a customer types code into a text box and clicks Save, you are running untrusted code in your infrastructure. The only question left is where that code runs, and how much of your company it can take down when it misbehaves.

I'm Ajay, I built PandaStack, a Firecracker microVM platform. This post is the argument for running each customer UDF in its own microVM — the threat model (yes, it's untrusted even though it's "your customer"), the cold-start economics that make per-invocation VMs actually viable, how to get input in and structured output out, and where the resource, time, and network limits go. With code.

It's untrusted — even though it's "your customer"

The first thing engineers get wrong is the trust label. "It's our paying customer's code, not some anonymous internet rando" feels reassuring, and it's irrelevant. The trust boundary is not the person who wrote the code; it's the code itself, running on shared infrastructure that also serves every other tenant. Even a completely well-meaning customer's UDF is a threat, for three reasons.

• Accidents outnumber attacks. An honest customer writes an infinite loop, a `while True: data.append(x)` that eats 40 GB, or a fork bomb by mistake. In-process, that's not their outage — it's yours, for every tenant on that box.
• Their account is one credential-stuff away from being someone else's. "Trusted customer" is a property of an identity, not of the code that identity can push. A compromised customer account is an attacker with a code-execution primitive inside your walls.
• The code can read what the process can read. If the UDF runs in your app process, it inherits your database handles, your service-account tokens, your other tenants' data in memory. "Transform this row" becomes "dump every env var and POST it out" with one line the customer didn't even have to write maliciously — a dependency they pulled in did.

Three ways to run it: eval, container-per-tenant, microVM-per-invoke

There are effectively three architectures for running customer-supplied code, and they sit on a clear isolation-vs-effort curve. Here's the honest comparison.

• Isolation boundary — eval/vm2 in-process: a language sandbox in your own process (no OS boundary). Container-per-tenant: Linux namespaces + cgroups on a shared host kernel. microVM-per-invoke: a hardware-virtualized guest with its own kernel under KVM.
• Blast radius of a break — eval/vm2: your entire process, its memory, every secret and DB handle it holds. Container: the host kernel and every neighbouring container if the kernel is exploited. microVM: one disposable VM; escaping means breaking the hypervisor, a far smaller and more-audited surface.
• Track record on untrusted code — eval/vm2: vm2 shipped more sandbox-escape CVEs than release notes and is now archived; "sandboxed JS in-process" is a category error. Container: real container escapes are a known, recurring class of bug. microVM: the model AWS Lambda and Fargate use precisely because it holds under untrusted multi-tenancy.
• Cold start — eval/vm2: microseconds. Container: tens of ms to seconds (image pull, runtime init). microVM: ~3s on a true cold boot, but ~179ms p50 (~49ms of that the snapshot restore) when you restore a baked snapshot per invocation.
• Resource + time limits — eval/vm2: cooperative at best; a tight loop ignores your timeout. Container: cgroups give real CPU/memory caps. microVM: hard vCPU/RAM ceilings baked into the guest, plus a wall-clock kill from outside.
• Operational cost — eval/vm2: trivial to add, expensive to get breached. Container: moderate; you own image builds, a warm pool, and escape-hardening. microVM: a platform concern, but if you buy it (or run one), per-invoke isolation is a create() call.

The reason teams keep reaching for `eval()` or a language-level sandbox is latency: spinning up a whole VM per invocation sounds absurd when the UDF runs in 4ms. That intuition was correct for a decade. It stopped being correct when snapshot-restore made a fresh microVM cost the same order of magnitude as a container's cold start.

Cold-start economics: why per-invoke VMs are now viable

A UDF invocation is bursty and short. A customer's validation hook fires when a form submits; their transform runs when a sync lands. You cannot keep a warm VM per customer sitting idle — with thousands of tenants that's a fortune in idle RAM. So the whole model hinges on one number: how fast can you get a clean, isolated execution environment on demand?

The old answer — cold-boot a VM — was 1–3 seconds, which is why nobody did per-invoke VMs. The modern answer is snapshot-restore: boot the environment once, snapshot the running machine to disk, and restore that snapshot per invocation instead of booting. There is no warm pool of idle VMs; every create restores a baked snapshot on demand. On PandaStack the numbers are:

• Create via snapshot restore: ~179ms p50, ~203ms p99 — of which the raw restore is ~49ms; the rest is network setup and a readiness probe.
• First cold boot (before a snapshot exists): ~3s — you pay this once per template, not per invocation.
• Fork from a warmed, post-setup state: 400–750ms same-host, 1.2–3.5s cross-host — useful when each invocation should start from an expensive-to-build baseline.
• Density headroom: 16,384 pre-allocated network slots per host, so the ceiling is memory/CPU, not networking — and idle costs ~nothing because you don't keep VMs warm.

So the economics flip: a fresh, hardware-isolated VM per invocation is ~180ms of overhead, in the same ballpark as pulling and starting a container — except you get a separate guest kernel instead of a shared one. For a workflow step or validation hook that already tolerates a network round-trip, that overhead is invisible next to the safety it buys. Compare vendor claims against their own docs, but the structural point holds: snapshot-restore is what makes VM-per-invocation stop being a joke.

Passing input in, getting structured output out

The execution contract for a UDF is simple: hand it the input row (or event payload) as JSON, run the customer's function against it, and read back a structured result — plus stdout/stderr for debugging and an exit code for success. The clean pattern is: write the input JSON to a file in the guest, write a small harness that imports the customer's code and calls it, exec with a timeout, then read a `result.json` the harness produced. Don't scrape stdout for the actual result — reserve stdout for logs and use a result file for data.

import json
from pandastack import Sandbox

# The customer's UDF, pulled from your DB. Untrusted by definition.
user_udf = """
def transform(row):
    # customer logic: normalize a messy record onto our schema
    return {
        "email": row["Email"].strip().lower(),
        "full_name": f"{row['first']} {row['last']}".title(),
        "active": row.get("status") == "A",
    }
"""

# A harness YOU control: reads input.json, calls transform(), writes result.json.
harness = """
import json, sys
with open("/workspace/input.json") as f:
    payload = json.load(f)
try:
    from udf import transform
    out = transform(payload)
    with open("/workspace/result.json", "w") as f:
        json.dump({"ok": True, "data": out}, f)
except Exception as e:
    with open("/workspace/result.json", "w") as f:
        json.dump({"ok": False, "error": str(e)}, f)
    sys.exit(1)
"""

input_row = {"Email": " [email protected] ", "first": "ada", "last": "lovelace", "status": "A"}

with Sandbox.create(template="code-interpreter", ttl_seconds=60) as sbx:
    sbx.filesystem.write("/workspace/udf.py", user_udf)
    sbx.filesystem.write("/workspace/harness.py", harness)
    sbx.filesystem.write("/workspace/input.json", json.dumps(input_row))

    result = sbx.exec("python3 /workspace/harness.py", timeout_seconds=5)

    if result.exit_code == 0:
        out = json.loads(sbx.filesystem.read("/workspace/result.json"))
        print("transformed:", out["data"])
    else:
        print("UDF failed:", result.stderr or "(timeout or crash)")
# VM is destroyed here — no state survives to the next invocation

A few things are load-bearing here. The harness is your code, not the customer's — it defines the calling convention, catches exceptions so a customer's raised error becomes a structured `{"ok": false}` instead of an opaque crash, and never lets customer code choose the result path. The `timeout_seconds=5` on exec is your wall-clock circuit breaker: a customer's `while True` gets killed from outside the guest, no cooperation required. And `ttl_seconds=60` on create is the backstop so a VM you forget to kill reaps itself. The `with` block kills the sandbox on exit, so every invocation starts from the pristine baked snapshot with zero leakage from the last customer's run.

Resource, time, and network limits

Isolation stops a UDF from touching your data. Limits stop it from starving your fleet. Three independent controls, each of which the customer's code cannot cooperate its way out of:

• Time — pass timeout_seconds on every exec. A microVM timeout is enforced by the host, not the guest, so an infinite loop or a `sleep(99999)` is killed on the wall clock regardless of what the code does.
• Memory and CPU — the guest's vCPU and RAM are fixed by the template snapshot (Firecracker can't resize at restore), so a UDF that tries to allocate 40 GB hits the guest's ceiling and OOMs inside its own VM. Your host and every other tenant never notice.
• Filesystem — the rootfs is copy-on-write and disposable. A `rm -rf /` or a runaway log that fills the disk dies with the VM at the end of the invocation; nothing is shared back.

The one that trips teams up is network egress. By default a guest can reach the internet, which for a data-transformation UDF is usually exactly wrong — that's the exfiltration path where "normalize this row" becomes "POST this row to my server." The fix is at the network layer, not in the code: deny outbound from the guest's network namespace and allow-list only what the UDF legitimately needs (often: nothing). I go deeper on this in the post on controlling network egress from untrusted code (/blog/controlling-network-egress-untrusted-code) — the short version is that you cannot trust the running code to restrict itself, so you enforce egress rules outside it.

Warm vs cold, per-customer vs per-invocation

The safe default is a fresh VM per invocation, killed after — no state leaks between invocations or between tenants, and the ~180ms create is affordable on any async path. But two variants matter.

If a UDF needs an expensive baseline — a large dataset loaded, a heavy dependency imported, a warmed interpreter — build that state once, snapshot it, and fork per invocation instead of creating cold. A same-host fork is 400–750ms and shares memory copy-on-write, so every invocation starts from the identical post-setup state without re-paying the setup cost. If a customer's UDFs run in tight bursts and are genuinely on a latency-sensitive synchronous path, keep one warm VM per customer (per tenant, never shared across tenants) and hibernate it between bursts so idle cost stays near zero. The rule that never bends: one trust domain per VM. The moment two customers' code shares a VM, the VM boundary — the entire point — is gone. This is the same multi-tenant isolation discipline covered in the post on multi-tenant code execution (/blog/multi-tenant-code-execution).

from pandastack import Sandbox

# Build an expensive baseline ONCE (e.g. load the customer's reference dataset),
# snapshot it, then fork a fresh isolated VM per invocation from that state.
base = Sandbox.create(template="code-interpreter", persistent=True)
try:
    base.filesystem.write("/workspace/setup.py",
        "import pandas as pd\n"
        "df = pd.read_parquet('/workspace/ref.parquet') if False else pd.DataFrame()\n"
        "df.to_parquet('/workspace/ref.parquet')\n")
    base.exec("python3 /workspace/setup.py", timeout_seconds=30)
    snap = base.snapshot()  # capture the warmed, post-setup state

    # Per invocation: fork (400-750ms same-host), run one tenant's UDF, discard.
    for _ in range(3):
        with base.fork(snapshot=snap) as run:
            run.filesystem.write("/workspace/input.json", '{"x": 1}')
            r = run.exec("python3 /workspace/harness.py", timeout_seconds=5)
            print("exit:", r.exit_code)
finally:
    base.kill()

Fork gives you the best of both: the isolation of a fresh VM per invocation with the startup cost amortized against a warmed baseline. It's the right shape when the per-invocation setup is heavier than the invocation itself.

When a microVM is the wrong tool

Honesty first: not every "customer configuration" needs a VM. If what your customers write isn't really code — a filter expression, a template string, a declarative mapping — evaluate it with a purpose-built, non-Turing-complete interpreter and skip the whole problem. A restricted expression language with no I/O, no imports, and no loops is genuinely safe to run in-process because it can't do anything. The category error is only when you give customers a real programming language (arbitrary Python, JS, shell) and pretend a language-level sandbox contains it.

The line is: the moment a customer can `import`, spawn a process, open a socket, or write a loop, you're hosting untrusted code and the boundary needs to be an OS-or-hardware boundary — a microVM being the strongest of those that still boots fast enough to run per invocation. Everything short of that — `eval()`, `vm2`, a language VM in your process — is a convenience you'll eventually explain in an incident review. If you're evaluating this pattern for a broader plugin or extension surface, the post on building a secure plugin marketplace on microVMs (/blog/secure-plugin-marketplace-microvm) walks through the same isolation model at ecosystem scale.