Per-Tenant Email Sending Isolation with microVMs

Every multi-tenant SaaS that sends email eventually hands its customers a loaded gun. It starts innocently: tenants want to customize their receipts, so you let them edit a Handlebars template. Then they want conditional logic, so you add helpers. Then a big customer wants to shape the outbound payload with a webhook, or run a small JS transform at send time to localize currency and merge in CRM fields. Now, on every send, your notification worker is executing code your customers wrote — inside the same process that sends everyone else's email. That's the moment the blast radius stops being theoretical.

This post is about running that per-tenant send-time logic in its own ephemeral Firecracker microVM instead of your shared worker: the isolation problem, per-tenant egress and sender-reputation isolation, the ephemeral-VM-per-batch pattern, and why snapshot-restore plus scale-to-zero makes it cost about what a container would. I'm Ajay, I built PandaStack — I'll be straight about where this is worth the complexity and where it isn't.

Why the shared worker is a liability

The moment tenant-defined code runs in your send worker, four distinct failure modes show up, and they compound. A shared process is a shared fate.

• Runaway compute: a tenant's template with an accidental infinite loop, or a JS transform that allocates 8GB, doesn't fail politely — it pins a worker core or OOM-kills the process that was mid-flight sending fifty other tenants' password resets. One tenant's infinite Handlebars loop shouldn't melt everyone's Tuesday.
• Noisy neighbor: even well-behaved custom code steals CPU and memory from the shared pool. A tenant doing heavy per-recipient personalization silently degrades everyone's send latency, and you can't cap them without capping the whole worker.
• Secret leakage: your worker process holds every tenant's ESP API keys, signing secrets, and DB credentials in memory. Tenant-supplied code — especially a JS transform or a template helper with too much reach — is one `process.env` or one clever prompt-injected payload away from reading credentials that belong to a different customer.
• Deliverability coupling: all tenants share your worker's outbound IPs. One spammy or compromised tenant tanks the sender reputation of the IPs everyone else sends from, and suddenly a legitimate customer's invoices land in spam because of a neighbor they've never heard of.

The model: one ephemeral microVM per send batch

The fix is to stop running tenant code in a process you care about. Instead, for each send batch, spin up a throwaway Firecracker microVM, hand it the tenant's template and transform code plus just the data for that batch, let it render and send, and destroy it. Each microVM boots its own guest kernel under hardware virtualization (the same VMM AWS Lambda uses), so a runaway loop, a memory bomb, or a malicious `os.system(...)` is contained to one disposable VM — not your fleet. The same isolation story we cover for databases in /blog/per-tenant-database-isolation applies just as cleanly to the send path.

The reason this pattern was historically impractical is boot time: nobody wants to pay a multi-second VM boot per email batch. Firecracker plus snapshot-restore removes that tax. On PandaStack a create is a p50 of 179ms (p99 ~203ms) because every create restores a baked snapshot on demand — the restore step itself is ~49ms — rather than cold-booting. The first cold boot of a fresh template is ~3s, but that happens once at bake time, not per send. So an ephemeral-VM-per-batch pattern costs you tens of milliseconds of overhead, which is noise next to the ESP round-trip you were going to pay anyway.

Per-tenant egress and sender-reputation isolation

Isolation isn't only about compute — for email, egress is half the point. Deliverability is governed by the reputation of the IP and domain that mail leaves from. When every tenant sends through one shared pool, reputation is a shared, tragedy-of-the-commons resource: one bad actor's spam complaints and bounce rate drag down the inbox placement of every honest tenant on the same IPs.

Because each microVM has its own network namespace and TAP device, you can bind a tenant's send VM to a specific egress path — a dedicated NAT IP, a per-tenant SMTP relay, or a specific ESP subaccount whose reputation is scoped to that one tenant. A tenant who burns their reputation burns only their own. On PandaStack every sandbox already runs in its own pre-allocated /30 network namespace (16,384 of them per agent), so egress control is per-VM by construction; you route the VM's outbound traffic through the relay or IP that belongs to that tenant. For the deeper mechanics of pinning and restricting outbound traffic, see /blog/controlling-network-egress-untrusted-code.

• Reputation blast radius: a compromised or spammy tenant's bounces and complaints are scoped to their own IP/subaccount, not the shared pool.
• Compliance boundary: a per-tenant egress path makes it straightforward to satisfy a customer who contractually requires their mail to leave from a dedicated IP or region.
• Rate isolation: you throttle a single tenant's send VM without touching the send rate of anyone else — the noisy-neighbor problem, solved at the network layer too.
• Clean revocation: kill the VM and the tenant's send capability, egress path, and in-memory credentials all vanish atomically.

Rendering and sending inside a sandbox

Here's the core loop for one tenant's batch: create a sandbox, write in the tenant's template and the recipient data, run a small renderer, and read back the rendered output (or let the VM send directly). Set PANDASTACK_API_KEY in your environment and the SDK picks it up. The context manager kills the non-persistent VM on exit, so cleanup is automatic even if the send throws.

import json
from pandastack import Sandbox

# Tenant-supplied Handlebars template — you did NOT write this.
tenant_template = """\
Hi {{name}},\n\nYour invoice for {{month}} is {{amount}} due {{due_date}}.\n{{#if past_due}}This account is past due — please pay to avoid interruption.{{/if}}\n"""

# The batch: only THIS tenant's recipients, nothing shared.
batch = [
    {"name": "Dana", "month": "July", "amount": "$42.00", "due_date": "2026-07-20", "past_due": False},
    {"name": "Wei",  "month": "July", "amount": "$118.50", "due_date": "2026-07-05", "past_due": True},
]

renderer = """
const fs = require('fs');
const Handlebars = require('handlebars');
const tpl = Handlebars.compile(fs.readFileSync('/work/template.hbs', 'utf8'));
const rows = JSON.parse(fs.readFileSync('/work/batch.json', 'utf8'));
const out = rows.map(r => tpl(r));
fs.writeFileSync('/work/rendered.json', JSON.stringify(out));
console.log('rendered ' + out.length + ' messages');
"""

# One throwaway VM for one tenant's batch. Bind egress to the tenant's relay
# via metadata your agent uses to pick the outbound route.
with Sandbox.create(template="base", ttl_seconds=120) as sbx:
    sbx.filesystem.write("/work/template.hbs", tenant_template)
    sbx.filesystem.write("/work/batch.json", json.dumps(batch))
    sbx.filesystem.write("/work/render.js", renderer)

    # Timeout is the circuit breaker for a runaway template loop.
    result = sbx.exec("node /work/render.js", timeout_seconds=15)
    if result.exit_code != 0:
        raise RuntimeError(f"render failed: {result.stderr}")

    rendered = json.loads(sbx.filesystem.read("/work/rendered.json"))
    print(f"{result.stdout.strip()} -> handing {len(rendered)} to the ESP")
# VM (and any secrets it held) is destroyed here

`exec` returns a result with `stdout`, `stderr`, `exit_code`, and `duration_ms`. Two habits keep you safe: always pass `timeout_seconds` (tenant-written templates loop more often than you'd like — the timeout is your circuit breaker), and always pass `ttl_seconds` on create so a VM you forget to kill reaps itself. Notice what the VM does not have: it never saw another tenant's data, and it only holds the one credential you chose to inject for this send.

If you'd rather the VM send directly instead of handing rendered output back — the cleaner design, because the ESP API key then only ever lives inside the disposable guest — inject that tenant's ESP credential as an env var on exec and let the guest make the outbound call itself:

from pandastack import Sandbox

send_script = """
import os, json, urllib.request

key = os.environ["TENANT_ESP_KEY"]         # scoped to THIS tenant only
msgs = json.load(open("/work/rendered.json"))

for body in msgs:
    req = urllib.request.Request(
        "https://api.tenant-esp.example/v1/send",
        data=json.dumps({"html": body}).encode(),
        headers={"Authorization": f"Bearer {key}", "Content-Type": "application/json"},
    )
    with urllib.request.urlopen(req, timeout=10) as r:
        print("sent:", r.status)
"""

with Sandbox.create(template="base", ttl_seconds=120) as sbx:
    sbx.filesystem.write("/work/rendered.json", rendered_json)  # from the render step
    sbx.filesystem.write("/work/send.py", send_script)
    # The key lives only in this exec, only in this VM, and dies with it.
    result = sbx.exec(
        "env TENANT_ESP_KEY=$TENANT_ESP_KEY python3 /work/send.py",
        timeout_seconds=60,
    )
    print(result.stdout)

That inverts the trust model in your favor. The ESP key for tenant A is only ever present in tenant A's VM, for the seconds it takes to send, and the VM's egress is already scoped to tenant A's IP. There is no shared process that holds all keys at once, which means there is no single memory space an escape could scrape them from.

Shared worker vs container-per-tenant vs microVM-per-send

The three architectures trade isolation against cost and operational weight. The honest summary:

• Isolation strength — Shared worker: none; tenant code runs in your process on your kernel. Container-per-tenant: process/namespace isolation on a shared host kernel (a kernel bug or escape reaches neighbors). microVM-per-send: hardware-virtualized, separate guest kernel per send — the strongest boundary.
• Runaway-code blast radius — Shared worker: takes down the worker and everyone mid-flight. Container-per-tenant: cgroup limits cap CPU/mem but a shared-kernel exploit still spreads. microVM-per-send: capped to one disposable VM, killed on TTL or timeout.
• Secret exposure — Shared worker: every tenant's keys in one memory space. Container-per-tenant: keys per container, but the host kernel sees all. microVM-per-send: only the one tenant's credential, only in that guest, gone when it dies.
• Egress / sender reputation — Shared worker: one shared IP pool, shared fate. Container-per-tenant: doable with per-container netns, but you're building it. microVM-per-send: per-VM network namespace by construction, bind each to a tenant IP/relay.
• Cost & density — Shared worker: cheapest per send, most expensive per incident. Container-per-tenant: idle containers cost money unless you scale them to zero yourself. microVM-per-send: snapshot-restore + scale-to-zero means you pay only for the ~seconds of the send.
• Operational weight — Shared worker: simplest, until the first incident. Container-per-tenant: orchestrator, image registry, per-tenant lifecycle to manage. microVM-per-send: one API call per batch; the platform owns the lifecycle.

The container-per-tenant middle ground is real and popular, but note what it's actually buying: it shares the host kernel, so it does not close the untrusted-code escape surface the way a microVM does, and per-tenant egress isolation is something you assemble yourself. It's a reasonable stop if your tenant code is semi-trusted. If it's genuinely arbitrary, the microVM is the boundary that matches the threat.

Why this doesn't blow up your bill

The instinct is that a VM per send must be expensive. It isn't, because of two properties. First, there's no warm pool: with snapshot-restore, a VM exists only while a send is in flight, so idle cost is essentially zero — you scale to zero between batches instead of paying for containers that sit around waiting. Second, restore is cheap (~49ms restore step, 179ms p50 create), so the create overhead is a rounding error against the send itself.

For a tenant that sends in bursts (nightly digests, a marketing blast) the arithmetic is friendly: a VM that lives for the ten seconds it takes to drain the batch, then dies, costs you ten VM-seconds — not a container you kept warm for the other 86,390 seconds of the day. If you need every send to start from a known-good, dependency-warmed state, bake it into the snapshot once; you can also fork a configured sandbox (same-host fork is 400–750ms, cross-host 1.2–3.5s, sharing memory copy-on-write) if you want per-send divergence from a common base.

When this is the wrong tool

Don't reach for a VM-per-send if your tenants can't run arbitrary code. If "customization" means picking from a fixed set of templates you wrote, with values substituted by a data-only, non-Turing-complete renderer (no custom helpers, no eval, no webhooks that run tenant code), then there's no untrusted code to isolate and a shared worker is simpler and faster. The microVM earns its keep specifically when tenants supply logic — Handlebars helpers, JS transforms, send-time webhooks that shape the payload — or when per-tenant egress reputation is a real business requirement.

And remember the boundary is the VM, not your intentions: a sandbox isolates the tenant's code from your infrastructure and from other tenants, but it can't protect a secret you hand into it. Inject only the one credential that send needs, scope the egress, set a timeout and a TTL, and let the VM die. Within those lines, per-tenant email isolation on microVMs gives you the flexibility your customers want without making one tenant's bad template everyone's incident.