Per-Tenant Isolation for User-Defined GraphQL Resolvers

At some point every extensible SaaS platform crosses the same line. A headless CMS lets customers write custom GraphQL resolvers. A low-code backend lets them attach serverless field logic. An API gateway lets them drop in a webhook transform. The feature is a growth lever — but the moment you ship it, you are running arbitrary tenant code inside your own request path. Tenant A's resolver must never read Tenant B's data, must never peg a CPU and starve everyone else on the node, and must never quietly open a socket to your internal Postgres. This post is about the isolation model that actually holds up under that pressure: one Firecracker microVM per tenant (or per request), with a CPU/wall-clock budget and a locked-down egress policy.

I'm Ajay, I built PandaStack. We run exactly this pattern in production, so I'll be concrete about the mechanics and honest about the trade-offs — especially the latency ones, because a resolver runs on the hot path and a slow isolation layer is a non-starter.

Shared kernel = shared fate

The naive implementations all share something. Run the resolver as a function in your Node process and it shares your event loop, your memory, and your environment variables — a `process.env` read hands the tenant your database URL. Run it in a shared-kernel container and you've narrowed the blast radius, but a container escape or a kernel bug still reaches the host and every neighbor on it. The uncomfortable truth: if two tenants share a kernel, they share fate. A vulnerability that compromises one can, in principle, compromise all of them.

A Firecracker microVM changes the category. Each tenant's resolver boots its own guest kernel under hardware virtualization (KVM), confined to a tiny set of emulated virtio devices. There is no shared kernel to attack — an escape would have to break the hypervisor itself, a far smaller and more heavily audited surface than the full Linux syscall interface a container sees. The `while (true) {}` loop pegs one VM's vCPU, hits its wall-clock budget, and gets killed. Nobody else notices.

Per-tenant vs per-request isolation

There are two shapes, and the right one depends on your trust model and your traffic pattern.

• Per-request: create a fresh microVM for each resolver invocation, run the code, return the result, destroy the VM. Nothing leaks between calls because there is no reused state. This is the safe default for spiky or low-volume tenants, and it's the strongest isolation you can buy.
• Per-tenant (warm): keep one microVM alive per tenant and route that tenant's invocations to it. State can persist between calls (a warm cache, a loaded module), and you skip the create cost on every request after the first. The boundary is still the VM, so tenants never mix — but you must never route two different tenants into the same warm VM.

The mistake to avoid is the tempting middle ground: one shared warm VM that runs everybody's resolvers to save money. That collapses the isolation boundary — the VM is the boundary, so co-tenanting inside it is exactly the shared-fate problem again, just one layer up. If you want warm-per-tenant economics, keep a small pool of per-tenant VMs and reap idle ones, don't pool across tenants.

The latency budget: why this is even viable

A resolver sits on the request hot path. If isolating it adds a full VM cold boot — a few seconds — to every field resolution, the feature is dead on arrival. This is the reason microVM-per-tenant was historically impractical and why people reached for in-process sandboxes instead.

Snapshot-restore is what makes it viable. Instead of cold-booting a VM per call, PandaStack bakes a template into a Firecracker snapshot once and restores it on demand. A create is p50 179ms (p99 ~203ms), and a warm snapshot restore is ~49ms — that's the number that matters on the hot path. For an even tighter budget, fork from a warm per-tenant snapshot: a same-host fork lands in 400–750ms with copy-on-write memory, so a per-tenant base VM that's already imported the tenant's dependencies forks cheaply per request. A cold boot is only paid once, on the first spawn (~3s), and then never again.

Egress allowlists: the resolver can't reach your internal DB

Isolation is not just about the CPU — it's about the network. A tenant's resolver often legitimately needs to fetch data (call the tenant's own upstream API, hit a public endpoint). But it must not be able to open a connection to your internal Postgres, your Redis, your metadata service, or another tenant's database VM. Because each microVM runs in its own network namespace with its own TAP device, egress is enforced at the network layer, not by trusting the code. You define an allowlist — the tenant's declared upstreams, public DNS — and everything else is dropped before it leaves the guest. The resolver can compute and it can call out to where you let it, and that's all.

This is where PandaStack's model helps in two directions. Inbound, sandbox preview URLs are tokenless and host-based — a port is reachable at `https://<port>-<id>.<suffix>` for the VM's lifetime, and the sandbox UUID is the credential, so you route to a specific tenant VM without threading a shared secret through your gateway. Outbound, the per-VM netns is the enforcement point for the allowlist. If your resolver logic is more serverless-shaped than persistent, the functions model fits the same mold: upload the tenant's code bundle, invoke it per request against a fresh VM, and never keep a warm pool that could leak state between tenants.

Running a tenant resolver: the execution loop

The loop is: create (or fork) a sandbox for the tenant, write the resolver code and its arguments in, exec it with a hard timeout, and parse a JSON result back out of stdout. Passing args as a file and returning structured JSON (rather than scraping human-readable stdout) keeps the boundary clean. Always pass `timeout_seconds` — it's your wall-clock circuit breaker against the infinite loop.

import json
from pandastack import Sandbox

# The tenant's user-defined resolver (stored in your DB, written by the customer).
tenant_resolver = """
import json, sys

def resolve(args):
    # Custom field logic the tenant wrote. Arbitrary, untrusted.
    order = args["order"]
    subtotal = sum(i["qty"] * i["price"] for i in order["items"])
    return {"total": round(subtotal * 1.08, 2), "currency": "USD"}

args = json.load(open("/workspace/args.json"))
json.dump(resolve(args), open("/workspace/result.json", "w"))
"""

# The resolver arguments GraphQL would pass in for this request.
resolver_args = {"order": {"items": [
    {"qty": 2, "price": 19.99},
    {"qty": 1, "price": 5.00},
]}}

def run_resolver(code: str, args: dict) -> dict:
    # Fresh VM per request: no state leaks between tenants or calls.
    with Sandbox.create(template="code-interpreter", ttl_seconds=60) as sbx:
        sbx.filesystem.write("/workspace/resolver.py", code)
        sbx.filesystem.write("/workspace/args.json", json.dumps(args))
        # Hard wall-clock budget — the busy-loop circuit breaker.
        r = sbx.exec("python3 /workspace/resolver.py", timeout_seconds=5)
        if r.exit_code != 0:
            raise RuntimeError(f"resolver failed: {r.stderr}")
        return json.loads(sbx.filesystem.read("/workspace/result.json"))
    # VM destroyed here — egress netns, memory, and disk all gone.

print(run_resolver(tenant_resolver, resolver_args))
# -> {'total': 48.58, 'currency': 'USD'}

`exec` returns an `ExecResult` with `stdout`, `stderr`, `exit_code`, and `duration_ms`. Read the result from a known file rather than parsing stdout so a stray `print()` in the tenant's code can't corrupt your response. If a resolver blows its `timeout_seconds`, the exec is killed and you return a GraphQL error for that field — the rest of the query is unaffected, and no other tenant's resolvers were ever in the same process to be affected in the first place.

The same pattern in JS

Nothing about this is Python-specific — a customer who writes their resolver in JavaScript runs in exactly the same isolation model. Here's a sample tenant resolver and the shape of the wrapper that runs it inside the guest:

// tenant-resolver.js — written by the customer, arbitrary & untrusted.
// Reads args from a file, writes a JSON result. Runs inside the microVM.
const fs = require("fs");

function resolve(args) {
  const { user } = args;
  // Custom field logic: derive a display name and a gravatar-ish hash.
  const display = user.name || user.email.split("@")[0];
  return { display, initials: display.slice(0, 2).toUpperCase() };
}

const args = JSON.parse(fs.readFileSync("/workspace/args.json", "utf8"));
fs.writeFileSync(
  "/workspace/result.json",
  JSON.stringify(resolve(args)),
);

// Host side (Python), running the JS resolver in the guest:
//   sbx.filesystem.write("/workspace/resolver.js", tenant_js)
//   sbx.filesystem.write("/workspace/args.json", json.dumps(args))
//   r = sbx.exec("node /workspace/resolver.js", timeout_seconds=5)
//   result = json.loads(sbx.filesystem.read("/workspace/result.json"))

The guest is just a Linux box with a kernel of its own, so whatever runtime the tenant's code needs — Node, Python, Bun, Deno — is a matter of which template you baked, not a change to the isolation story. A tenant who wants a native dependency isn't a special case; it's a `pip install` inside their VM, contained to their VM.

How the isolation options compare

• vm2 / in-process JS sandbox — fastest (no boundary crossing), but it's a software sandbox inside your Node process. vm2 in particular has a long history of escape CVEs, and a busy loop still blocks your event loop. Fine for fully trusted plugins, not for arbitrary tenant code.
• isolated-vm / V8 isolates — real memory isolation per isolate and much harder to escape than vm2, but still one shared process and one shared host kernel; a native crash or a resource-exhaustion bug takes the whole worker down, and you're limited to JS.
• Container-per-tenant — separate namespaces and cgroups, so CPU limits and OOM kills are enforceable, and any language works. But the shared host kernel means a container escape or kernel bug crosses the tenant boundary — the thing you're trying to prevent.
• MicroVM-per-tenant (Firecracker) — separate guest kernel per tenant, hardware-virtualized. Any language, enforceable CPU/wall-clock budgets, network-layer egress control, and an escape has to break the hypervisor. The cost is startup latency, which snapshot-restore (~49ms) and fork (400–750ms same-host) bring down to hot-path-viable.

Honest limits

A microVM isolates execution, not your architecture mistakes. If you hand the resolver a database connection string, no hypervisor will save you — never inject internal credentials into the guest environment; give the resolver a scoped, tenant-specific token instead, if it needs data access at all. Cross-host forks are slower (1.2–3.5s) than same-host, so pin a tenant's warm base and its forks to the same agent to stay in the fast lane. And there's a real capacity ceiling — a single agent tops out around 16,384 sandboxes (its /30 subnet space), with memory as the practical binding constraint well before that. For a headless CMS with thousands of tenants each firing resolvers, that's a scheduling and pooling problem worth designing for up front.

But within those bounds, the deal is a good one: you let customers write real code in your request path — the extensibility that sells the platform — without betting every other tenant's uptime and data on that code behaving. The busy loop pegs one VM. The rogue fetch hits a dropped packet. And the tenant who tries to read someone else's data finds themselves alone in a disposable Linux box with a kernel that has never heard of anyone else.