Sandboxing User-Written Webhook Transformations

It usually starts as a support ticket. A customer's destination expects `user_id` where your webhook sends `userId`, and their vendor won't budge, and yours won't either, and somebody on your team says the reasonable thing: "what if they could just write a tiny function?" So you ship a textarea labeled "Transform (optional)", with a placeholder that reads `return payload;`. It takes an afternoon. It closes eleven tickets. It is, genuinely, a good feature — this is what Zapier's code steps, Segment's functions, and Svix's transformations all are, and customers love them because the alternative is a support thread that ends in "unfortunately that's not supported."

And then, quietly, your company changes shape. You are now running arbitrary user-authored code, on your infrastructure, adjacent to data belonging to other people, at whatever volume your webhook ingest happens to run at. Congratulations: you are a code-execution company. Nobody scheduled a threat-modeling session for this, because from the product side it was a text field. I'm Ajay; I built PandaStack, which is a Firecracker microVM sandbox platform, and a meaningful share of the people who show up already have this feature in production and are trying to work out how bad the thing they shipped is. This post is the honest version of that conversation: why the obvious implementations don't hold, what the threat model actually is (it's worse than "user code is untrusted"), and how to build it so the isolation is real and the latency budget still closes.

There's an accelerant, too. Increasingly the transform isn't written by a careful integration engineer — it's generated by an LLM from a natural-language description of the mapping. That's a great product experience, and it makes your code-execution surface both higher-volume and lower-review. Nobody is reading these before they run.

The three implementations that do not hold

Almost everyone lands on one of three designs, in roughly this order, as each one breaks.

1. eval, in the ingest process

This is the version that took an afternoon. `eval` in your Node or Python ingest worker is not a security control — it's a load-bearing act of faith. But set security aside for a second, because you don't even need malice to lose here. A transform containing `while (true) {}`, or an accidentally catastrophic regex over an attacker-supplied string, does not fail — it stops. Single-threaded event loop, one wedged tick, and your entire ingest pipeline for every tenant stops accepting events. The first outage this feature causes is almost never an exploit. It's a customer who wrote a loop wrong.

The security part is bad separately, and in a way that's hard to walk back. In-process eval inherits your process's imports, your open file descriptors, your environment variables — which is where your database URL and your third-party API keys live — and your network position inside the VPC. A transform is three lines away from being an exfiltration tool holding production credentials with internal network access, and the person writing those three lines may not even be your customer.

2. node:vm, or a "sandboxed" interpreter

Then someone finds `vm.runInNewContext`, and it looks like the answer, because it is called a sandbox and it takes a timeout argument. It is not the answer, and you don't have to take my word for it: Node's own documentation states that the `vm` module is not a security mechanism and must not be used to run untrusted code. That is upstream telling you, in the manual, that the thing named sandbox is not a sandbox.

// DO NOT DO THIS. node:vm is explicitly NOT a security boundary --
// Node's own docs say so. This is here so you recognize it in your
// codebase, not so you copy it.
import vm from "node:vm";

export function runTransform(userCode: string, payload: unknown) {
  const sandbox = { payload, result: null as unknown };
  vm.createContext(sandbox);

  // The timeout option looks reassuring. It does not cover async work,
  // and it does nothing at all about the escape classes below.
  vm.runInNewContext(userCode, sandbox, { timeout: 1000 });
  return sandbox.result;
}

// Escape class 1 -- walk out through the prototype chain of anything you
// passed in, reach a host constructor, and you are back in the outer realm:
//     this.constructor.constructor("return process")().env
//
// Escape class 2 -- prototype pollution. The payload is attacker-supplied
// JSON. Mutating Object.prototype inside the context changes behavior for
// code that runs after you, in the same process:
//     ({}).__proto__.isAdmin = true
//
// Escape class 3 -- everything you handed in on purpose. A logger, a fetch,
// a Buffer, a metrics client: each one is a rope, and the untrusted context
// is holding the other end of it.

The structural problem is that `vm` gives you a fresh *context*, not a fresh *process*, and certainly not a fresh *kernel*. Everything shares one heap and one set of intrinsics, so the boundary is a JavaScript-language boundary — and JavaScript-language boundaries are famously porous. The escape usually routes through the prototype chain of some object you handed across, back to a host constructor, and from there to `process.env`. Every so often somebody publishes a fresh one-liner. You will not win this by patching, because you'd be maintaining a security boundary that the runtime's own authors have publicly declined to maintain as one.

The same skepticism applies to third-party "secure interpreter" packages and isolate-based runtimes. Some are genuinely well-built; some are a weekend project with a confident README. The test isn't whether the docs use the word sandbox — it's whether the project states a threat model that includes hostile code, and whether its issue tracker suggests it survives contact with people trying. Verify that against their docs, not mine.

3. A shared pool of containers

This is the mature-looking one, and it is a real improvement: process isolation, a memory cgroup, a filesystem view, a kill switch that actually works. Plenty of teams stop here and it's defensible for a lot of threat models. Two things about it are worth being clear-eyed on, though.

First, the kernel is shared. Every container on the box calls into the same kernel through a very large syscall surface, so a kernel privilege-escalation bug is a cross-tenant bug rather than a container bug. That's not hand-waving — it's the entire reason the gVisor/Kata/Firecracker product category exists. Second, and this is the one that actually bites in practice without any CVE involved: pooling is reuse. If a worker handles tenant A's transform and then tenant B's, whatever A left behind is sitting there when B arrives. A file in `/tmp`. A monkey-patched global in a warm runtime. A mutated module cache. A still-open socket. An environment variable someone exported. "We reset it between runs" is a claim about a cleanup routine's completeness, and cleanup routines are never complete, because the set of things that can be left behind grows every time somebody adds a dependency.

The threat model nobody writes down

Here's what makes webhook transforms specifically nastier than the generic "run untrusted code" problem. I'd want this on the whiteboard before any design discussion:

• The code is untrusted. Obvious enough. Your customer wrote it, or an LLM wrote it from your customer's prompt, and neither version was reviewed by you.
• The input is also untrusted. The payload is a webhook body. It arrives from the internet, in whatever shape the sender felt like sending.
• The input is attacker-controllable by a third party. Anyone who can reach the webhook URL — a partner system, a compromised upstream, sometimes just whoever learns the URL — chooses the exact bytes your customer's code will process.
• And the two compose. A transform that is perfectly benign on normal traffic can become an exfiltration primitive when fed a payload crafted specifically to make it behave differently.
• The blast radius is other tenants' data, because the thing the transform runs next to is your ingest pipeline, which is carrying everybody's events.

That fourth point gets underweighted constantly. Consider a transform that does something completely reasonable: building a destination URL from a field on the event, or dynamically indexing into a config object using a key taken from the payload, or — extremely common in LLM-generated transforms — calling `JSON.parse` on a string field and spreading the result into an object. Every one of those is fine on the events your customer tested with. Every one of those is a lever a third party can pull by controlling the payload. The `__proto__` key in a spread is not an exotic attack; it's a Tuesday.

So the isolation cannot be "we trust the code because our customer wrote it," and it cannot be "we validate the payload because we know its shape." It has to be a boundary that holds when a hostile input meets sloppy code — a situation you should assume is happening continuously, because at any real webhook volume it is.

The useful mental reframe: you are not running your customer's code. You are running the product of your customer's code and a stranger's input, and you don't get to review either one.

What good looks like

Strip out the vendor talk and the requirements are pretty mechanical. You want a boundary the guest can't reason its way past, a hard stop on time and memory, no network unless explicitly granted, nothing in the environment worth stealing, and a data path that is exactly one payload in and one payload out.

1. One microVM per execution — or per tenant, kept warm, with an ephemeral fork per event. The guest runs its own kernel under hardware virtualization, so an escape has to get through the hypervisor's deliberately tiny device surface rather than a general-purpose syscall table.
2. A hard wall-clock timeout enforced from outside the guest. Not a timeout inside the runtime, which the code can simply outlast; a supervisor that kills the machine. Infinite loops have to be a boring outcome that affects exactly one event.
3. Memory bounds that come from the machine rather than from a flag the runtime is politely asked to respect. A microVM has the RAM it has; allocating past it kills the guest, not your host.
4. Egress denied by default. This is the control that stops your other tenants' payloads from being POSTed to a pastebin. If a transform genuinely needs to call one enrichment API, that's an allowlisted destination the customer configures — not an open internet connection handed out as a default.
5. No ambient credentials. The guest environment should hold nothing: no database URL, no cloud metadata reachability, no service account, no internal DNS worth resolving. Pass the payload in, take the transformed payload out, and that is the whole interface.
6. Deterministic resource accounting. You need per-execution wall time and CPU anyway — to bill it, and to notice the tenant whose transform got 40× slower after their last edit.

The credentials point deserves emphasis, because it's the cheapest one to get right and the most common one to get wrong. The classic failure: the transform runs in a worker that already had `DATABASE_URL` and a payments API key in its environment, because that worker also does five other jobs. Your sandbox boundary can be flawless and it won't matter, because you handed the keys across it yourself. Cloud instance metadata endpoints are the same story with extra steps — if the guest can reach the metadata IP, you have shipped a credential vending machine to untrusted code.

The latency budget problem

Here's the crux, and it's why people end up on the shared-pool design even when they know better. Webhook ingest is latency-sensitive and high-volume. Senders retry when you're slow, some of them have short timeouts and unforgiving backoff, and your p99 is somebody else's incident. "A fresh VM per event" is the architecturally correct answer, and it has historically been an unaffordable one, because a VM meant a multi-second boot and a few hundred megabytes of setup. So teams reuse — and reuse is exactly where the isolation quietly leaks back out.

Snapshot-restore changes the arithmetic enough that the correct answer becomes practical. PandaStack keeps no warm pool of idle VMs; every create restores a baked Firecracker snapshot on demand. The restore step itself is around 49ms, end-to-end create is p50 179ms with p99 around 203ms, and the roughly 3-second figure applies only to the first-ever cold boot of a template, before its snapshot exists. A same-host fork of an existing sandbox runs 400–750ms; cross-host, 1.2–3.5s. On the capacity side each agent pre-allocates 16,384 /30 subnets, so per-sandbox networking isn't the resource that runs out first.

Put those numbers next to your real budget rather than an imagined one. If your ingest already does a durable write, a queue publish, and a destination HTTP call, a sub-200ms create is one term in a sum, not the whole thing. If you're delivering synchronously against a 500ms SLA to the sender, it's most of your budget and you need a different shape. That's a measurement you can make this afternoon, and it should drive the decision more than anybody's architecture diagram.

Three shapes, and a decision rule

• Isolation between events — Fresh microVM per event: total; every execution starts from an identical baked snapshot with zero residue from any prior run. Warm per-tenant sandbox: strong across tenants, weak between that tenant's own events, since files, globals, and stray processes persist unless you clean them.
• Per-event latency cost — Fresh microVM per event: one create, p50 179ms and p99 around 203ms. Warm per-tenant sandbox: an exec into a machine that already exists, so the create leaves the hot path entirely.
• Cross-tenant blast radius — Fresh microVM per event: one machine, one payload, deleted immediately afterward. Warm per-tenant sandbox: still confined to one tenant, but a foothold there survives across many of that tenant's events.
• Resource accounting — Fresh microVM per event: trivial, because the machine's lifetime is the execution. Warm per-tenant sandbox: you have to attribute CPU and memory per exec inside a shared long-lived guest, which is real work.
• Failure containment — Fresh microVM per event: a wedged transform costs you exactly one event. Warm per-tenant sandbox: a wedged transform can stall that tenant's stream until the sandbox is recycled.
• Idle cost — Fresh microVM per event: zero, because nothing exists between events. Warm per-tenant sandbox: you pay while it waits, which for a long-tail tenant sending three events a day is essentially the entire bill.
• Best fit — Fresh microVM per event: async ingest behind a queue, spiky or long-tail traffic, and anything touching sensitive payloads. Warm per-tenant sandbox: a small number of high-rate tenants with a genuinely tight synchronous budget.

The rule I'd actually use: default to a fresh microVM per event, and promote a tenant to a warm sandbox only when their sustained rate makes the per-event create a real fraction of the budget you measured. When you do promote them, keep the warm machine strictly per-tenant and never shared, make an ephemeral fork the unit of execution so each event still gets a clean copy-on-write machine, and recycle the underlying sandbox on a fixed schedule so any foothold has a bounded lifetime. Batching is tempting and is really the warm model with extra coupling: N events in one guest means a poisoned event can observe the other N−1, so batch only within a single tenant and keep N small enough that one timeout doesn't take down a large batch of otherwise-fine events.

The right way, in code

The shape is: create a sandbox, write the untrusted code and the untrusted payload in as data files (never interpolate either into a shell command — that's a second injection surface you'd be adding for free), exec a small runner of your own with a hard timeout, read exactly one result off stdout, and kill the machine in a `finally` so a crash in your own worker doesn't leak a VM.

import json
from pandastack import Sandbox

# The runner is OURS, not the customer's. It loads their transform, feeds
# it the payload, and prints exactly one JSON line on stdout. Anything the
# transform prints itself is redirected to stderr, where we can hand it
# back to the customer as debug output instead of corrupting our protocol.
RUNNER = r'''
import json, sys

real_stdout = sys.stdout
sys.stdout = sys.stderr          # customer print() -> stderr, always

payload = json.load(open("/work/payload.json"))

ns = {}
exec(open("/work/transform.py").read(), ns)   # untrusted, but confined
result = ns["transform"](payload)

real_stdout.write(json.dumps(result))
'''


def run_transform(user_code: str, payload: dict, budget_s: int = 5) -> dict:
    # A fresh machine per event. ttl_seconds is the backstop: if our own
    # process dies between create and kill, the platform reaps it anyway.
    sbx = Sandbox.create(template="code-interpreter", ttl_seconds=60)
    try:
        # Both untrusted inputs cross the boundary as FILES.
        # Never interpolate user code or a payload into a shell string.
        sbx.filesystem.write("/work/transform.py", user_code)
        sbx.filesystem.write("/work/payload.json", json.dumps(payload))
        sbx.filesystem.write("/work/runner.py", RUNNER)

        # Hard wall-clock bound, enforced outside the guest.
        r = sbx.exec("python3 /work/runner.py", timeout_seconds=budget_s)

        if r.exit_code != 0:
            # A crash, an OOM kill, or a timeout. All three are the
            # customer's bug, and all three must reach the customer.
            return {
                "ok": False,
                "reason": "transform_failed",
                "exit_code": r.exit_code,
                "stderr": r.stderr[-4000:],
            }

        return {
            "ok": True,
            "result": json.loads(r.stdout),
            "logs": r.stderr[-4000:],   # give them their own print() output
        }
    finally:
        # Deleted whole. No /tmp left over for the next tenant to read.
        sbx.kill()

Three details there do more work than they look like they do. Separating stdout (your protocol) from stderr (their logs) means a customer who adds a debug `print` doesn't break your JSON parsing — a failure mode that produces bewildering tickets. Slicing `stderr` to the last few kilobytes means a transform that logs in a loop can't hand you a gigabyte to store. And `ttl_seconds` is the belt to `kill()`'s braces: your worker will eventually get OOM-killed mid-execution, and when it does you want the platform reaping the orphan rather than you finding it in next month's cost report.

The egress default belongs at the platform layer, not in this function. If sandboxes can reach the internet by default, no amount of care in the runner helps — the transform just opens a socket and sends its input somewhere. Deny by default, allowlist per customer, and log the allowlist changes, because "which customers can make outbound calls from a transform, and to where" is a question a security reviewer will eventually ask you.

Failure semantics, retries, and the transform that always times out

Isolation is only half the job. The other half is what happens when the transform doesn't work, which — at any real number of customers — is constantly. Design these as product behavior, not as exception handling you'll get to later.

A timeout is a first-class outcome, not an error you swallow. It has to be delivered to the customer as a specific, debuggable state that says "your transform exceeded its 5-second budget on event `evt_abc123`", with their stderr attached. "Transform failed" with no further detail is a support-ticket generator, and the ticket will be assigned to you, and you will spend forty minutes reproducing something the customer could have fixed in thirty seconds if you'd told them what happened.

Retries have to be idempotent, which means the retry unit is the whole pipeline stage — pull event, run transform, deliver — keyed by an event id so redelivery is detectable downstream. Retrying only the transform while holding partial state is how an event gets delivered twice with two different bodies. And distinguish "the transform threw" (deterministic; retrying produces the identical broken result) from "the sandbox couldn't be created" (infrastructure; retry with backoff and it probably works).

1. Classify the failure before you retry it. Nonzero exit from user code is deterministic; a create failure or an agent hiccup is transient. Only the second class deserves backoff and another attempt.
2. Circuit-break the deterministic class per transform version. After N consecutive failures, stop executing that transform, park its events, and tell the customer loudly. A transform that always times out should burn a bounded amount of your compute, not an unbounded amount.
3. Make the breaker reset on a new transform version, not on a timer. The customer edits their code; that's the signal that it might work now. Auto-resetting on a clock just re-burns the compute every five minutes forever.
4. Cap concurrency per tenant. One customer's slow transform must not consume the sandbox capacity that everyone else's events need — and a fresh-VM-per-event design makes this easy to enforce, because the concurrency limit is just a count of live sandboxes.
5. Keep a dead-letter path with the original untransformed payload. When a customer fixes their transform on Thursday, they want Tuesday's events, and they want them from the raw input rather than from whatever the broken version emitted.

The observability you owe the customer

This is the least glamorous section and probably the highest-leverage one. You asked your customer to write code in a box on your website. They have no debugger, no local reproduction, no way to add a breakpoint. The only thing standing between them and total helplessness is what you choose to show them.

Show them stdout, stderr, and the exit code — verbatim, per execution, with the input payload alongside, because a transform failure is meaningless without the event that triggered it. If your sandbox hands you `.stdout`, `.stderr`, and `.exit_code`, you already have everything; the only work is deciding to surface it rather than collapsing it into a boolean. Add wall-clock duration so they can watch themselves approach the timeout before they cross it, and ship a dry-run endpoint that runs a transform against a captured historical event through the exact same sandbox path, so what they test is what will run.

# Dry-run a transform against a real captured event, using the same
# sandboxed path production uses. This is the single highest-value
# endpoint you can ship alongside a custom-transform feature.
curl -sS -X POST https://api.example.com/v1/transforms/test \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
        "transform_id": "tr_9f21",
        "event_id": "evt_abc123"
      }' | jq

# What the customer should get back -- not a boolean:
# {
#   "ok": false,
#   "reason": "timeout",
#   "duration_ms": 5000,
#   "budget_ms": 5000,
#   "exit_code": 124,
#   "stdout": "",
#   "stderr": "processing 1 of 40000...\nprocessing 2 of 40000...\n",
#   "input": { "id": "evt_abc123", "body": { "...": "..." } }
# }

That stderr tail is the whole ballgame. "Processing 1 of 40000" tells the customer instantly that they wrote a loop over a field that's much bigger in production than in their test fixture. Without it, they file a ticket that says "transforms are broken" and you both lose an afternoon.

You didn't plan to be this, but you are

The uncomfortable summary is that the moment you shipped that textarea, your security posture stopped being about your own code. It became about the worst code any of your customers will ever write, composed with the worst input any stranger will ever send, running next to everybody's data. That's not an argument against the feature — the feature is good, customers want it, and the alternative is losing deals to somebody who offers it. It's an argument for making the boundary structural instead of procedural.

Structural means the isolation is a property of the architecture rather than the correctness of a reset routine: a machine with its own kernel, created for one execution and deleted after it, with no network and no credentials, killed on a wall-clock deadline it cannot argue with. That used to be too expensive to do per event, and the whole reason the compromised designs exist is that everyone was working around the cost. Snapshot-restore has moved that cost enough — a create around 179ms at p50 — that the compromise isn't necessary for most ingest architectures anymore. Measure your actual budget, decide between fresh-per-event and warm-per-tenant on the numbers rather than the vibes, and then go make sure your ingest worker isn't handing `DATABASE_URL` across the boundary you just built.