Isolating Untrusted Webhook Handlers in MicroVMs

A webhook handler is a stranger's code you agreed to run on a schedule. If your product lets customers register their own inbound-event handlers, transforms, or plugins — the Zapier/n8n shape, or any SaaS that runs customer-authored logic on incoming events — then at some point an event arrives and you execute code you did not write, on behalf of a customer you cannot fully vet, on infrastructure shared with every other customer. That is a multi-tenant code-execution problem wearing a webhook costume, and the only honest question is: what's the wall between customer A's handler and customer B's data? This post is about making that wall a real one — a per-tenant Firecracker microVM — and making it cheap enough that you can stand one up per event.

The threat: whose code is running, and next to whose data?

Start with the premise, because it's uncomfortable and it's load-bearing. The moment your platform lets customers upload a handler — a JavaScript function that runs on inbound Stripe events, a Python transform that reshapes a payload, a plugin that enriches a record — you are a multi-tenant execution service. You run arbitrary code from thousands of tenants on shared machines to keep the economics sane. And that code is, from your perspective, hostile until proven otherwise. Not because your customers are malicious (mostly they aren't), but because you can't tell the difference between a customer's buggy handler, a customer's compromised handler, and an attacker who signed up specifically to run code on your infrastructure.

So enumerate what a handler can try to do if the wall is thin:

• Read another tenant's data: if handler execution shares a process, a filesystem, or a memory space with anything belonging to customer B, then customer A's handler is one path-traversal or one leaked environment variable away from customer B's payloads, credentials, or secrets.
• Reach the host: a shared-kernel container escape turns 'this handler misbehaved' into 'this handler owns the box, and every other tenant on it.' Container escapes are a recurring, well-documented class of vulnerability, not a hypothetical.
• Exfiltrate outbound: a handler that can open arbitrary network connections can POST your customer's data to an attacker's server, or scan and hit your internal services — the SSRF classic, where 'fetch this URL' becomes 'fetch http://169.254.169.254/ and read the cloud metadata credentials.'
• Burn resources: an infinite loop, a fork bomb, a memory balloon, or a crypto miner. The noisy-neighbor problem, except the neighbor is actively trying to be noisy and you're paying the CPU bill.

Why a shared-kernel container is the wrong boundary

The reflexive answer is 'run each handler in a container.' It's cheap, it's fast, and for trusted first-party code it's fine. For arbitrary customer code it's the wrong boundary, and the reason is architectural, not a matter of tuning. A container is a set of Linux namespaces and cgroups around a process that still shares the host's single kernel with every other container on the machine. That shared kernel is a large, complex attack surface — millions of lines of privileged C — and it's exactly the surface a malicious handler probes to escape.

You can harden containers a long way: seccomp filters, a user namespace, a read-only rootfs, dropped capabilities, gVisor or Kata in front. Those are real improvements and you should use them. But they're all mitigations layered on a boundary that was never designed to contain adversarial code — you're narrowing a shared kernel's attack surface, not removing the sharing. For code you genuinely don't trust, the right boundary is a separate kernel: a virtualization boundary the CPU enforces in hardware, so a compromised handler is trapped inside a guest that can't see the host's kernel at all. That's what a microVM gives you, and it's why AWS built Firecracker to run Lambda rather than trusting containers between tenants.

Containers isolate processes from each other. MicroVMs isolate kernels from each other. When the code is a stranger's, you want the second kind of wall.

The pattern: event in, microVM, exec the handler, capture, tear down

The control loop is the same whether the trigger is an inbound webhook, a queue message, or a scheduled event. Your ingest layer receives the event, then does five things:

1. Route to isolation: pick (or create) a microVM scoped to this tenant and handler. For ephemeral-per-event, that's a fresh VM restored from a snapshot; for a hot tenant, it's a persistent per-tenant VM you keep around.
2. Deliver the payload: write the event JSON into the guest filesystem (or pass it on stdin). The handler code — which you pre-loaded when the customer registered it — reads it there.
3. Exec the handler with a hard timeout: run the customer's function against the payload. Every exec returns stdout, stderr, and an exit code, so you can tell success from a crash from a timeout.
4. Capture the result and side-effects: read the handler's response off the filesystem or stdout. This is the only thing that escapes the VM — the response you deliberately extract, plus any outbound calls the handler was allowed to make.
5. Destroy or hibernate: for ephemeral, delete the VM — nothing survives. For persistent, hibernate it (snapshot + pause) so the next event for that tenant wakes it fast.

Because the VM is disposable (or resettable), you don't need defensive cleanup between events. If a handler corrupts its environment, mines a block, or leaks a file descriptor, it does so inside a VM that's about to be deleted or restored from a clean snapshot. The blast radius is one event for one tenant.

A per-event handler in Python

Here's the whole loop with the Python SDK: create a tenant-scoped microVM, deliver the inbound payload, exec the customer's handler with a timeout, read the response back out, and tear the VM down. Set PANDASTACK_API_KEY in the environment first. The `metadata` tag lets you attribute the VM to a tenant for auditing, and `ttl_seconds` is a backstop so a handler that hangs kills its own VM.

import json
from pandastack import PandaStack

ps = PandaStack()  # reads PANDASTACK_API_KEY, base url https://api.pandastack.ai

def run_handler(tenant_id: str, handler_code: str, event: dict) -> dict:
    # 1. Fresh, hardware-isolated microVM scoped to THIS tenant. Every create
    #    restores a baked snapshot on demand (~179ms p50), so a VM-per-event
    #    is cheap. The metadata tag attributes the VM for audit + egress rules.
    sb = ps.sandboxes.create(
        template="base",
        ttl_seconds=60,                       # hard cap: VM self-destructs after 60s
        metadata={"tenant": tenant_id, "kind": "webhook-handler"},
    )
    try:
        # 2. Deliver: the customer's handler + the inbound event payload go in
        #    as plain files. Nothing about this tenant touches another's VM.
        sb.filesystem.write("/handler.py", handler_code)
        sb.filesystem.write("/event.json", json.dumps(event))

        # 3. Exec the untrusted handler with a hard timeout. It reads the event,
        #    writes its response to /out.json. A loop-forever handler is killed.
        result = sb.exec(
            "python3 /handler.py < /event.json > /out.json",
            timeout_seconds=15,
        )

        # 4. Capture the response + side-effects. This is the ONLY thing that
        #    escapes the VM — everything else dies with it.
        if result.exit_code != 0:
            return {"ok": False, "stderr": result.stderr[-2000:]}
        response = sb.filesystem.read("/out.json")
        return {"ok": True, "response": json.loads(response)}

    finally:
        # 5. Destroy. No state survives the event. That's the isolation.
        sb.delete()

Note the two safety rails, because they're not decoration. The `timeout_seconds` on `exec` is a circuit breaker for a handler that loops forever; the `ttl_seconds` on create is a backstop so a VM you forget to reap reaps itself. Untrusted code gets a wall, a clock, and a hard cap — then it gets thrown away. If you'd rather stream a long-running handler's output live instead of blocking, the SDK exposes a streaming exec that emits stdout, stderr, and exit events.

Egress control: so a handler can't exfiltrate or SSRF you

Isolation stops a handler from reading its neighbors. Egress control stops it from phoning home. These are different problems and you need both. A handler that's perfectly isolated from other tenants can still POST your customer's payload to an attacker's server, or — worse — turn around and hit your own internal network: the database that thinks anything on the VPC is trusted, the admin service on a private port, the cloud metadata endpoint at 169.254.169.254 that hands out credentials to anyone who asks. That's server-side request forgery, and a customer-authored handler is a gift-wrapped SSRF primitive if you let it reach the network freely.

The microVM boundary helps here because every sandbox gets its own network namespace with its own TAP device and its own NAT rules — PandaStack pre-allocates 16,384 /30 subnets per agent, one per sandbox, so a handler's network is genuinely its own segment, not a shared bridge. That per-VM network is the enforcement point. Set your posture there:

• Default-deny outbound: a transform that only reshapes a payload needs no network at all. Give it none. The safest handler is one that can't open a socket.
• Block the internal ranges and metadata endpoint: if a handler does need egress, deny RFC1918 (10/8, 172.16/12, 192.168/16), the link-local 169.254.0.0/16 (which covers the cloud metadata IP), and anything else on your private plane. A handler should never be able to reach your database or your control plane.
• Allowlist, don't blocklist: for handlers that call out to a customer's own endpoint, prefer an explicit allowlist of destinations over trying to enumerate everything evil. Blocklists lose; allowlists fail closed.
• Rate-limit and cap: bound how much a single handler can send, so a compromised one can't turn into an exfiltration firehose or a DDoS amplifier on your bill.
• Set the timeout and TTL: an egress-heavy handler that's also slow is a resource-exhaustion attack. The hard clock is part of your network defense, not just your CPU defense.

Making a VM-per-event cheap: snapshot-restore

The usual objection to 'a fresh VM per event' is cost. A full VM per webhook sounds absurd — VMs boot in tens of seconds and hold gigabytes. That intuition is correct for cold-booting a conventional VM and wrong for a microVM snapshot restore, which is a different operation entirely. A cold boot does real work: the kernel initializes, userspace comes up, your runtime loads. A restore skips all of it — Firecracker maps a frozen, already-booted VM's memory and device state back in and resumes it mid-instruction. It's closer to unpausing than to starting a computer.

On PandaStack that restore-based create runs at a p50 of 179ms and a p99 around 203ms (the restore-and-resume core of that is roughly 49ms), versus a genuine cold boot of about 3 seconds that happens exactly once per template and is then baked into a snapshot and amortized away. Memory is copy-on-write (MAP_PRIVATE) and the rootfs is a reflink clone, so the Nth identical handler VM doesn't copy gigabytes — it shares the baked pages until it writes. If you'd rather branch a live per-tenant VM to fan out N handlers from one warm state, a same-host fork is 400–750ms (cross-host 1.2–3.5s). At those numbers, a VM-per-event stops being a luxury and becomes the obvious default: you get a real kernel boundary per event without the boot tax that made per-event VMs look impossible.

Ephemeral-per-event vs. persistent-per-tenant vs. shared container

There are three shapes for where a handler runs, and the right one depends on the tenant's traffic and your trust posture. It's worth laying them side by side rather than picking one dogmatically:

• Isolation strength — Ephemeral-per-event: strongest; a clean guest kernel per event, nothing survives. Persistent-per-tenant: strong; a real kernel boundary per tenant, but state persists within the tenant across events. Shared container: weakest; a shared host kernel is the only wall, wrong for adversarial code.
• Idle cost — Ephemeral-per-event: near zero; a tenant that isn't firing holds no compute. Persistent-per-tenant: you pay to keep the VM warm (mitigate by hibernating idle tenants — snapshot + pause — and waking on the next event). Shared container: low, but you pay in blast radius what you save in cost.
• Latency floor — Ephemeral-per-event: the ~179ms restore per event (plus your handler). Persistent-per-tenant: near-zero once warm, hibernate-wake if it was parked. Shared container: near-zero, at the cost of the boundary you actually needed.
• State between events — Ephemeral-per-event: none by construction; every event is a clean slate. Persistent-per-tenant: preserved (warm caches, DB connections, in-memory context) — a feature for stateful handlers, a risk if a handler is compromised. Shared container: shared across tenants unless you're extremely careful, which is the failure mode.
• Best fit — Ephemeral-per-event: bursty, spiky, long-tail traffic and the highest-trust-sensitivity handlers. Persistent-per-tenant: a hot tenant firing constantly, or handlers that need warm state. Shared container: only trusted first-party code, never arbitrary customer code.

In practice most platforms blend the first two: ephemeral-per-event as the default (near-zero idle cost, maximal isolation), and a persistent-per-tenant VM promoted only for the handful of tenants whose event volume makes the per-event restore floor add up — with those persistent VMs hibernated when idle so you're not renting an empty room. The shared container tier is for your own trusted code, not the customer's.

When you don't need a microVM per handler

This is real infrastructure, not a free lunch — you own routing, snapshot hygiene, egress policy, and the plumbing to get payloads in and responses out. Be honest about whether your threat model calls for it.

• The 'handlers' are declarative config, not code. If a customer 'handler' is a JSON mapping or a field-selector — no arbitrary loops, no network, no filesystem — you can evaluate it in-process safely; there's nothing Turing-complete to contain.
• The code is all first-party and trusted. If only your own engineers write handlers, a hardened container is probably enough; the multi-tenant-adversary argument evaporates.
• A managed FaaS already fits. If your handlers map cleanly onto Lambda/Cloud Functions and you don't need byte-level control of the runtime or the egress plane, use one — those platforms run on microVMs under the hood anyway, and you skip operating the fleet.

Reach for per-handler microVMs when you run genuinely arbitrary customer code, when a handler can make network calls (so SSRF and exfiltration are in scope), when compliance or contracts demand a hardware isolation boundary between tenants, or when your event volume makes managed per-invocation pricing painful. In those cases snapshot-restore gives you what containers can't — a real kernel boundary per event — without the speed penalty that used to make it impractical. Match the tool to the threat model: a webhook handler is a stranger's code, and you should treat it exactly that well.