Per-Tenant Webhook Signing Secrets in microVMs

If your product sends webhooks, you sign them. Every serious webhook API — Stripe, GitHub, Shopify — ships a per-customer signing secret and an HMAC signature header so the receiver can prove the payload came from you and wasn't tampered with in flight. The problem shows up the moment you're multi-tenant: you now hold one signing secret per customer, and if all of them live in a single delivery process, that process is a single address space holding thousands of forge-anyone-you-like keys. Add the feature customers always eventually ask for — "let me run a small transform on the payload before you deliver it" — and you're now executing tenant-supplied code in the exact process that holds every tenant's signing secret. That is a bad place to be.

This post is about moving the signing step, and any per-tenant transform code, out of your shared delivery worker and into its own ephemeral Firecracker microVM: the blast-radius argument, why an in-process JS sandbox doesn't close the gap, how ephemeral VM-per-delivery keeps a secret scoped to exactly one signature, and where a managed per-tenant database fits if you need to track delivery state. I'm Ajay, I built PandaStack — I'll be honest about where this is worth it and where it's overkill.

Why one process holding every secret is the liability

A signing secret is a bearer credential to impersonate you. Whoever holds tenant A's webhook secret can craft a payload, compute a valid HMAC, and deliver it to tenant A's endpoint with a signature that passes verification — a forged event that looks byte-for-byte legitimate. In a shared delivery worker, every one of those secrets is resident in the same heap at the same time. Shared address space is a group chat your customers' secrets did not consent to join.

• Forge-anyone blast radius: a single read primitive — a heap-scraping bug, a log line that accidentally serializes the secrets map, a dependency with a memory-disclosure CVE — exposes not one secret but all of them at once. The blast radius of a leak is your entire customer base, not one account.
• Tenant code has a key ring: the instant you let customers run a transform (a JS function, a JSONata expression with escape hatches, a webhook-of-a-webhook that shapes the payload), that code executes next to the secrets. One `process.env`, one clever traversal, one prompt-injected payload, and a tenant reads a neighbor's signing key and can now forge events as them.
• Logs and crash dumps: secrets in a long-lived process end up in heap dumps, core files, and APM traces. A shared worker maximizes the number of places all secrets can co-locate with an error handler that wasn't written to redact them.
• Confused-deputy signing: a bug in routing that hands the wrong secret to the wrong delivery signs tenant B's payload with tenant A's key. In a shared map keyed by tenant id, that's a one-character mistake away.

Why an in-process sandbox doesn't close the gap

The reflex fix for "tenants run transforms next to secrets" is a JS sandbox: vm2, isolated-vm, a Worker thread, a locked-down expression language. These narrow the surface, and they're worth using — but they don't solve the secret-isolation problem, for two reasons. First, escapes from in-process sandboxes are a recurring class of CVE; vm2 was deprecated precisely because its escapes proved unfixable. The boundary is a language runtime you share with the untrusted code, and a language runtime is not a security boundary you want to bet every customer's signing key on. Second, even a perfect sandbox doesn't change where the secrets live: they're still in the parent process's memory, in the same OS process, one successful escape away. You've made the wall taller without moving the vault out of the room.

If the transform code is genuinely untrusted, the boundary has to be lower than the language runtime — a separate kernel and a separate address space, so that even a total compromise of the guest yields only the one secret you deliberately put in that guest.

The model: one ephemeral microVM per delivery

Stop signing in a process you care about. For each delivery (or each tenant's batch of deliveries), spin up a throwaway Firecracker microVM, inject only that tenant's signing secret and just the payloads for that delivery, run the tenant's transform and the HMAC signing inside the VM, read back the signed request, and destroy the VM. Each microVM boots its own guest kernel under hardware virtualization — the same VMM AWS Lambda runs functions on — so a runaway transform, a memory-scraping attempt, or a malicious `os.system(...)` is contained to one disposable guest whose entire secret inventory is a single key. The secret for tenant A never shares an address space with tenant B's code, because they were never in the same VM, or even the same kernel.

The reason this pattern used to be impractical is boot time — nobody wants a multi-second VM boot per webhook. Snapshot-restore removes that tax. On PandaStack a create is a p50 of 179ms (p99 ~203ms) because every create restores a baked snapshot on demand — the restore step itself is ~49ms — rather than cold-booting. The first cold boot of a fresh template is ~3s, but that happens once at bake time, not per delivery. So the isolation costs you tens of milliseconds, which is noise next to the HTTP round-trip to the customer's endpoint you were going to pay anyway.

Transforming and signing inside a sandbox

Here's the core loop for one tenant's delivery: create a sandbox, write in the tenant's transform code and the raw event, inject that tenant's signing secret as an env var scoped to the one exec, run the transform + HMAC sign inside the guest, and read back the signed body and signature. Set PANDASTACK_API_KEY in your environment and the SDK picks it up. The context manager kills the non-persistent VM on exit, so the secret and the VM die together even if signing throws.

import os, json
from pandastack import Sandbox

# Tenant-supplied transform — you did NOT write this. It reshapes the event
# before delivery (rename fields, drop PII, add a tenant-specific envelope).
tenant_transform = """
const raw = JSON.parse(require('fs').readFileSync('/work/event.json', 'utf8'));
// Whatever the customer configured. Untrusted by assumption.
const shaped = { ...raw, delivered_for: 'acme', schema: 'v2' };
require('fs').writeFileSync('/work/shaped.json', JSON.stringify(shaped));
"""

# The signing step runs AFTER the transform, over the shaped bytes, using the
# per-tenant secret that only exists inside this VM for these few seconds.
sign_script = """
const crypto = require('crypto');
const fs = require('fs');
const body = fs.readFileSync('/work/shaped.json');           // exact bytes we send
const secret = process.env.TENANT_WEBHOOK_SECRET;            // scoped to THIS tenant
const ts = Math.floor(Date.now() / 1000).toString();
const signed_payload = ts + '.' + body.toString('utf8');
const sig = crypto.createHmac('sha256', secret).update(signed_payload).digest('hex');
// Stripe-style header: t=<ts>,v1=<hex>. Emit only the signature, never the key.
fs.writeFileSync('/work/out.json', JSON.stringify({
  header: `t=${ts},v1=${sig}`,
  body: body.toString('utf8'),
}));
console.log('signed ' + body.length + ' bytes');
"""

raw_event = {"type": "invoice.paid", "id": "evt_123", "amount": 4200}
tenant_secret = os.environ["ACME_WEBHOOK_SECRET"]  # from YOUR secret store, per tenant

# One throwaway VM for one tenant's delivery.
with Sandbox.create(template="base", ttl_seconds=120) as sbx:
    sbx.filesystem.write("/work/event.json", json.dumps(raw_event))
    sbx.filesystem.write("/work/transform.js", tenant_transform)
    sbx.filesystem.write("/work/sign.js", sign_script)

    # Timeout is the circuit breaker for a runaway tenant transform.
    t = sbx.exec("node /work/transform.js", timeout_seconds=10)
    if t.exit_code != 0:
        raise RuntimeError(f"transform failed: {t.stderr}")

    # The secret lives ONLY in this exec, ONLY in this VM, and dies with it.
    s = sbx.exec(
        "env TENANT_WEBHOOK_SECRET=$SECRET node /work/sign.js",
        env={"SECRET": tenant_secret},
        timeout_seconds=10,
    )
    if s.exit_code != 0:
        raise RuntimeError(f"sign failed: {s.stderr}")

    signed = json.loads(sbx.filesystem.read("/work/out.json"))
    # Deliver from your worker with the header the guest computed. The header
    # left the VM; the secret never did.
    print("deliver with header:", signed["header"])
# VM (and the tenant secret it held) is destroyed here

`exec` returns a result with `stdout`, `stderr`, `exit_code`, and `duration_ms`. Two habits keep you safe: always pass `timeout_seconds` (tenant transforms loop more often than you'd like — the timeout is your circuit breaker), and always pass `ttl_seconds` on create so a VM you forget to kill reaps itself. Notice what leaves the VM: the signature header and the body, never the secret. The signing key is write-only from your platform's perspective — it goes in, a signature comes out, and the address space that held it is gone.

That HMAC step is deliberately boring and standard. The whole point is that it runs in a guest that holds exactly one key. Here's the same signature computed inside the sandbox in a couple of other runtimes, so you can match whatever your transform stack already speaks:

# Inside the guest. Stripe-style signed payload: "<ts>.<body>".
# TENANT_WEBHOOK_SECRET is injected per-exec and never persisted.
TS=$(date +%s)
BODY=$(cat /work/shaped.json)
SIGNED_PAYLOAD="${TS}.${BODY}"

# HMAC-SHA256, hex. -hmac reads the key from argv here for brevity; in
# production pass it via env/stdin so it never lands in the guest's process
# list. Either way it exists only in this ephemeral VM.
SIG=$(printf '%s' "$SIGNED_PAYLOAD" \
  | openssl dgst -sha256 -hmac "$TENANT_WEBHOOK_SECRET" -hex \
  | sed 's/^.*= //')

echo "X-Webhook-Signature: t=${TS},v1=${SIG}"

# --- Python equivalent, same guest ---
# import hmac, hashlib, os, time
# secret = os.environ["TENANT_WEBHOOK_SECRET"].encode()
# body = open("/work/shaped.json", "rb").read()
# ts = str(int(time.time())).encode()
# sig = hmac.new(secret, ts + b"." + body, hashlib.sha256).hexdigest()
# print(f"t={ts.decode()},v1={sig}")

That inverts the trust model in your favor. The signing secret for tenant A is only ever present in tenant A's VM, for the seconds it takes to sign, alongside only tenant A's code and payload. There is no shared process that holds all keys at once, so there is no single memory space an escape — or a heap dump, or an over-eager log line — could scrape them from. A leak becomes a one-tenant, one-delivery incident instead of a fleet-wide key compromise.

Shared worker vs container-per-tenant vs microVM-per-delivery

The three architectures trade secret isolation against cost and operational weight. The honest summary, from the one axis that matters here — how many secrets can a single compromise reach:

• Secret blast radius — Shared worker: every tenant's signing key in one heap; one read primitive exposes all of them. Container-per-tenant: keys split per container, but the shared host kernel sees every container's memory, so a kernel-level compromise still reaches all. microVM-per-delivery: exactly one secret exists per guest, in a separate kernel — a full guest compromise yields one key.
• Untrusted transform code — Shared worker: runs in your process, next to the keys; an in-process sandbox escape reads them. Container-per-tenant: contained by namespaces, but a container escape or shared-kernel bug crosses to neighbors. microVM-per-delivery: hardware-virtualized guest kernel; an escape has to defeat the hypervisor, a far smaller and more audited surface.
• Confused-deputy signing — Shared worker: one routing bug signs the wrong tenant's event with the wrong key. Container-per-tenant: less likely, keys are partitioned by container. microVM-per-delivery: structurally impossible within a VM — only one key is present, so there's no wrong key to pick.
• Secret lifetime — Shared worker: all keys resident for the process lifetime, leaking into dumps and traces. Container-per-tenant: resident for the container's lifetime. microVM-per-delivery: resident for the seconds of one signing, then the address space is destroyed.
• Cost & density — Shared worker: cheapest per delivery, most expensive per breach. Container-per-tenant: idle containers cost money unless you scale them to zero yourself. microVM-per-delivery: snapshot-restore + scale-to-zero means you pay only for the ~seconds of the signing.
• Operational weight — Shared worker: simplest, until the first key-leak disclosure. Container-per-tenant: orchestrator, image registry, per-tenant lifecycle to run. microVM-per-delivery: one API call per delivery; the platform owns the lifecycle.

The container-per-tenant middle ground is real and reasonable for semi-trusted code. But note what it doesn't buy: it shares the host kernel, so it doesn't close the untrusted-transform escape surface the way a microVM does, and its main win — partitioning keys per container — is exactly the property the microVM gives you for free and per-delivery, with a real hardware boundary underneath. If the transform is genuinely arbitrary, the microVM is the boundary that matches the threat.

Tracking delivery state without re-centralizing it

Webhook delivery isn't fire-and-forget: you retry with backoff, dedupe on idempotency keys, and record attempt history so a customer can replay from a dashboard. That's state, and the temptation is to put it back in one shared database that every signing VM writes to — quietly re-centralizing the thing you just decentralized. You don't have to. PandaStack can give each tenant a managed PostgreSQL database of its own — a dedicated Firecracker VM with a durable volume, created in 30–90s (it blocks until Postgres is genuinely ready) — so a tenant's delivery log, retry cursor, and idempotency set live in a database only that tenant's deliveries touch. The ephemeral signing VM stays stateless and secret-scoped; the durable delivery state lives in a per-tenant database. See /blog/per-tenant-database-isolation for the deeper mechanics.

Why this doesn't blow up your bill

The instinct is that a VM per delivery must be expensive. It isn't, because of two properties. First, there's no warm pool: with snapshot-restore, a signing VM exists only while a delivery is in flight, so idle cost is essentially zero — you scale to zero between deliveries instead of paying for containers that sit around holding keys. Second, restore is cheap (~49ms restore step, 179ms p50 create), so the create overhead is a rounding error against the outbound HTTP call to the customer's endpoint.

For a tenant that delivers in bursts, batch the drain: one VM that lives the few seconds it takes to sign and ship a tenant's queued deliveries, then dies, costs you a few VM-seconds — not a container kept warm for the other 86,390 seconds of the day. If you want every signing VM to start from a known-good, dependency-warmed state, bake it into the snapshot once; you can also fork a configured sandbox (same-host fork is 400–750ms, cross-host 1.2–3.5s, sharing memory copy-on-write) if you want per-delivery divergence from a common base.

When this is the wrong tool

Don't reach for a VM-per-delivery if there's no untrusted code and no real secret-isolation requirement. If you sign with a single platform-wide key (not per-tenant), or your "transform" is a fixed, data-only mapping you wrote — no custom helpers, no eval, no tenant-supplied logic — then there's nothing untrusted to isolate, and signing in your worker behind a well-guarded secret manager is simpler and faster. The microVM earns its keep specifically when you hold a distinct secret per tenant AND either run tenant-supplied transform code near it, or have a compliance requirement that one tenant's key can never be co-resident with another's.

And remember the boundary is the VM, not your intentions. A sandbox isolates the tenant's transform from your infrastructure and from other tenants' keys, but it can't protect a secret you over-provision into it. Inject only the one signing key that delivery needs, set a timeout and a TTL, let the VM die, and keep the key out of your delivery-state store. Within those lines, per-tenant signing on microVMs gives your customers the payload flexibility they want without making one leaked key everyone's breach.