Per-Tenant Workflow Workers in Isolated microVMs

Every durable-execution system — Temporal, Airflow, Prefect, Inngest, Hatchet, or the homegrown Postgres-queue-plus-cron half of us actually run — has the same shape: a central engine that owns workflow state, and workers that poll for tasks and execute them. The engine half is usually well-designed and battle-tested. The worker half, in most multi-tenant products, is one process that imports every tenant's activity code, holds every tenant's API keys in the same heap, and runs them in threads with a timeout and a hopeful comment.

That gets sharper the closer you get to a Zapier-style product, where tenants define their own steps and you run them. At that point "customer code" is not a metaphor: you're executing arbitrary user-authored logic inside a process that also holds a hundred other customers' OAuth tokens. I'm Ajay; I built PandaStack, so treat this as opinionated. The argument is narrow: keep your workflow engine exactly as it is, and move only the activity execution into a per-tenant Firecracker microVM.

How shared worker pools actually fail

These four incidents recur in every multi-tenant orchestration system I've seen, roughly in order of severity.

Noisy neighbour: one retry loop, everyone's backlog

Workflow engines retry by design — that's the selling point. So when one tenant's activity starts failing fast against a broken upstream, the engine re-enqueues it and your workers burn every free slot re-running it. A retry loop is a self-amplifying noisy neighbour: the faster it fails, the more capacity it eats. Add one step that loads a 4 GB CSV into a dict and the pool starts OOM-killing unrelated tenants' tasks. Per-tenant queues help with fairness; they do nothing about a shared heap and a shared kernel.

Credential blast radius: every key in one heap

To run Tenant A's "post to Slack" step and Tenant B's "charge a card" step, a shared worker needs both tenants' credentials resident. However careful your secrets manager is, a fetched token then lives in process memory alongside everyone else's, reachable from any code in that process — including a tenant step that walks the interpreter's object graph, or a dependency that helpfully logs its env. The queue is often worse: a worker that can poll one tenant's queue can usually poll them all, because queue credentials are per-worker, not per-tenant.

Poison pills: the task that kills the worker, forever

The nastiest failure is the one the retry machinery turns into a loop. A task crashes the worker hard — a segfault in a native dependency, an OOM kill, a stack overflow in recursive user code. The engine never sees a completion, so after the visibility timeout it hands the task to another worker, which also dies. Repeat until the fleet is crash-looping on one poisoned payload: a distributed denial of service that you built, run, and pay for.

Uneven tenants: the whale and the plankton

Multi-tenant load is never uniform. One customer runs three workflows a day; another fans out 200,000 activities at 02:00 UTC. In a shared pool, the whale's midnight batch is indistinguishable from a DDoS as far as everyone else's latency is concerned. So you build weighted fair queuing, per-tenant caps, and a separate "big customer" deployment — which is to say you build per-tenant isolation anyway, badly, on a deadline.

The worker-per-tenant microVM model

The change is smaller than it sounds. You keep the engine, the queues, the retry policy, the history, the UI. What moves is one function: the thing that takes an activity task and runs the tenant's code. The worker becomes a thin dispatcher that leases a task, spins up a Firecracker microVM for that tenant, hands it one task's input and that tenant's credentials, waits for a result, and destroys the VM.

A Firecracker microVM is a real VM: its own guest kernel, memory, virtual disk, and network namespace, confined by KVM hardware virtualization — the model AWS Lambda uses for untrusted code from millions of customers. A container, by contrast, is a polite suggestion to a kernel that a hundred other tenants are also using. Once each attempt runs in its own guest, the four failure modes stop being things you defend against:

• Retry storms become self-limiting. A tenant's runaway retries consume that tenant's VM budget, capped by one integer in the dispatcher. Other tenants' tasks are on other CPUs in other kernels and never notice.
• Credential blast radius collapses to one tenant, one task. The guest is created fresh, gets only the secrets this activity needs, and is destroyed after. There's no other tenant's token to steal because none was ever on that machine.
• Poison pills die with the VM. A segfault, an OOM, a fork bomb, an `rm -rf /` — all confined to a throwaway guest. The retry gets a brand-new kernel instead of a corrupted worker.
• Uneven tenants become a scheduling problem, not an architecture problem. The whale gets 200 concurrent VMs, the plankton gets 2, and fair queuing is a config value, not a subsystem.

The historical objection is latency: a VM per activity is absurd if a VM takes ten seconds to boot. Snapshot-restore removes that objection. On PandaStack every create restores a pre-baked snapshot rather than cold-booting — p50 179ms, p99 ~203ms end to end, with the restore step itself around 49ms. A first-ever cold boot, before a snapshot exists, is about 3 seconds, paid once per template. Per-activity overhead is roughly one mediocre HTTP round trip, on top of steps that already take seconds to minutes.

Where the durable state lives (hint: not in the VM)

This is the design point people get wrong, so let me be blunt: do not move your workflow engine into the microVMs. The engine's database — history, event log, timers, task queues — stays central, shared, replicated, backed up, operated by you. It's trusted first-party infrastructure; it doesn't run customer code, so it doesn't need isolation. It needs consistency, and central state is how you get consistency.

The microVM boundary goes around activity execution only. The dispatcher talks to the engine — poll, heartbeat, complete, fail — with credentials the guest never sees; the guest talks only to what the activity legitimately needs. That asymmetry is the whole security story: even a tenant who fully compromises their guest cannot poll another tenant's queue, read workflow history, or forge a completion, because the task token and queue credentials live on the host side.

Short activities, long activities, and bursty tenants

Short activities: one VM per attempt

For anything measured in seconds — call an API, transform a payload, render a template — create a VM, run it, throw it away. Disposability is a feature: every attempt starts from an identical, known-good snapshot, so there's no state to corrupt between retries. If the create overhead bothers you, batch a tenant's queued short steps into one VM lifetime.

Long activities: heartbeat from the host, not the guest

A 40-minute export needs the engine to know it's still alive, or the visibility timeout will re-dispatch it. Heartbeat from the dispatcher, driven by liveness signals out of the guest — a progress file you poll, or output on the exec stream. Don't hand the guest the task token so it can heartbeat itself; that's exactly the credential you're keeping out of tenant-controlled code. If the guest goes silent, the engine re-dispatches and the wedged VM gets killed rather than nursed.

Bursty tenants: hibernate instead of tearing down

Some tenants have expensive warm state — a big dependency tree, a JIT, a headless browser, a model in RAM — and a schedule that fires every 15 minutes. Rebuilding that VM each time is wasteful; leaving it running between bursts is worse. Hibernate is the middle path: snapshot memory and disk, stop the VM, wake it on the next task. An idle tenant costs storage, not compute, and wakes with warm state intact. Copy-on-write memory (guests restore with pages mapped MAP_PRIVATE) also means idle tenants on one template share most of their pages.

Secrets, retries, and idempotency when the VM is disposable

Two habits make this model safe rather than merely isolated. First, secrets are injected per-VM at dispatch time and never baked into a template. The dispatcher fetches only the credentials this activity is scoped to, writes them into the guest, and the entrypoint loads them into the environment and shreds the file. If the tenant's step leaks its environment into a log, it leaks its own credentials — a support ticket, not a breach.

Second, treat every attempt as potentially duplicated. Disposable VMs make at-least-once delivery more visible, not more dangerous: an activity can die at any point and the engine retries on a fresh guest. Pass a stable idempotency key derived from the run and activity ID, and make side-effecting steps use it. The one genuinely new-feeling failure is the ambiguous timeout — the VM killed after the side effect but before the result came back — which is the same ambiguity a shared worker has when it OOMs mid-activity.

Implementation: one activity, one microVM

The host-side executor with the Python SDK: create a per-tenant sandbox, write the task input and that tenant's scoped credentials into the guest, run the step under a hard timeout, tear the VM down on the way out — including when the activity throws.

import json
from pandastack import Sandbox


class ActivityFailed(Exception):
    def __init__(self, msg: str, retryable: bool):
        super().__init__(msg)
        self.retryable = retryable


def run_activity(task: dict, secrets: dict) -> dict:
    """Execute ONE tenant's activity attempt in its own Firecracker microVM.

    `task` comes from the workflow engine; `secrets` is already scoped to
    this tenant. The task token stays here on the host -- the guest never
    sees it, so tenant code can never complete or poll someone else's work.
    """
    timeout = int(task.get("start_to_close_seconds", 300))

    with Sandbox.create(
        template="base",
        ttl_seconds=timeout + 60,  # backstop: the VM dies even if we crash
        metadata={
            "tenant": task["tenant_id"],
            "workflow_run": task["run_id"],
            "activity": task["activity"],
        },
    ) as sbx:
        # 1. Exactly one task's input, and only this tenant's credentials.
        sbx.filesystem.write(
            "/run/task/input.json", json.dumps(task["input"]).encode()
        )
        sbx.filesystem.write(
            "/run/task/secrets.json", json.dumps(secrets).encode()
        )
        # Idempotency key so a retried attempt doesn't double-charge anyone.
        sbx.filesystem.write(
            "/run/task/idempotency_key",
            f"{task['run_id']}:{task['activity_id']}".encode(),
        )

        # 2. Run the tenant's step. Worst case it segfaults, fork bombs, or
        #    rm -rf's the guest -- all of which kill this VM and nothing
        #    else. The engine just sees one failed attempt.
        res = sbx.exec(
            f"/opt/worker/run-activity.sh {task['activity']}",
            timeout_seconds=timeout,
        )

        if res.exit_code != 0:
            # 124 = timeout(1), 137 = SIGKILL/OOM. Both are worth retrying
            # on a clean guest; a user-code error usually is not.
            retryable = res.exit_code in (124, 137)
            raise ActivityFailed(res.stderr[-4000:], retryable=retryable)

        marker = "---PANDASTACK-RESULT---"
        return json.loads(res.stdout.split(marker, 1)[1])
    # VM destroyed here. Secrets, temp files, and whatever the tenant's
    # code left lying around go with it.

Note what's absent: no thread pool, no `signal.alarm`, no reclaiming a worker after user code misbehaves. Cleanup is "delete the machine" — the only primitive that reliably works against adversarial code.

The guest entrypoint and the dispatch loop

Inside the guest, the entrypoint only loads the injected secrets, wall-clock-caps the tenant's step, and emits a parseable result. It holds no queue credentials and has no idea other tenants exist.

#!/usr/bin/env bash
# /opt/worker/run-activity.sh -- runs INSIDE one tenant's microVM.
# The engine's DB, task queues, and history live outside this machine.
set -euo pipefail

ACTIVITY="$1"
IN=/run/task/input.json
OUT=/run/task/output.json
SECRETS=/run/task/secrets.json

# Secrets are injected per-VM by the dispatcher, never baked into the
# template. Load them into the env, then shred the file so a curious
# activity can't read them back off disk mid-run.
set -a
. <(jq -r 'to_entries[] | "\(.key)=\(.value)"' "$SECRETS")
set +a
shred -u "$SECRETS"

export PANDASTACK_IDEMPOTENCY_KEY="$(cat /run/task/idempotency_key)"

# Drop privileges before touching tenant code, and cap wall clock. If it
# wedges, timeout(1) exits 124 and the dispatcher retries on a fresh
# guest -- there is no long-lived worker process left to poison.
timeout --signal=TERM --kill-after=10s "${ACTIVITY_TIMEOUT:-300}s" \
  setpriv --reuid=activity --regid=activity --clear-groups \
  python3 -u "/opt/worker/activities/${ACTIVITY}.py" <"$IN" >"$OUT"

# Everything before this marker is the tenant's own stdout, which the
# dispatcher ships to that tenant's logs verbatim.
echo "---PANDASTACK-RESULT---"
cat "$OUT"

And the loop that ties it back to the engine — the piece that replaces your worker's `RegisterActivity` plumbing. It leases tasks, enforces a per-tenant concurrency cap, and translates VM outcomes into engine outcomes.

// dispatcher.go -- host side. Leases activity tasks from the workflow
// engine and runs each attempt in a per-tenant microVM.
func (d *Dispatcher) Run(ctx context.Context) error {
	for {
		task, err := d.engine.PollActivityTask(ctx, d.queue)
		if err != nil {
			if ctx.Err() != nil {
				return ctx.Err()
			}
			d.log.Warn("poll failed", "err", err)
			time.Sleep(time.Second)
			continue
		}

		// Per-tenant concurrency cap: a retry storm burns one tenant's
		// slots, not the fleet. Whale gets 200, plankton gets 2.
		slot := d.slots(task.TenantID)
		select {
		case slot <- struct{}{}:
		case <-ctx.Done():
			return ctx.Err()
		}

		go func(t Task) {
			defer func() { <-slot }()

			// Heartbeat from HERE, not from the guest: the task token
			// must never enter tenant-controlled code.
			stop := d.heartbeat(ctx, t)
			defer stop()

			secrets, err := d.vault.ScopedFor(ctx, t.TenantID, t.Activity)
			if err != nil {
				d.engine.FailActivity(ctx, t.Token, err, true /*retryable*/)
				return
			}

			out, err := d.vms.RunActivity(ctx, t, secrets) // one VM, disposable
			if err != nil {
				d.engine.FailActivity(ctx, t.Token, err, IsRetryable(err))
				return
			}
			d.engine.CompleteActivity(ctx, t.Token, out)
		}(task)
	}
}

Per-tenant observability, for free

A side effect nobody plans for: once each tenant's execution has its own machine, per-tenant telemetry stops being an instrumentation project. In a shared pool, attributing CPU seconds or memory to a tenant means threading a tenant ID through every metric and hoping nobody forgets — and the numbers are estimates anyway. With a VM per tenant, the VM is the attribution boundary, so usage billing, quotas, and "which customer caused this spike" become queries. Logs follow: a tenant's stdout is one VM's output, tagged with the metadata you set at create time.

Shared worker vs per-tenant container vs per-tenant microVM

Three ways to run tenant activity code, softest boundary to hardest. If you're comparing specific orchestrators or container runtimes, verify their isolation and concurrency behaviour against their own docs — it varies by version and configuration.

• Isolation strength — Shared worker: none between tenants; threads share an address space and a kernel. Per-tenant container: namespaces and cgroups, but every tenant sits on the same host kernel, so a kernel bug crosses tenants. Per-tenant microVM: own guest kernel under KVM; an escape needs a hypervisor break.
• Credential blast radius — Shared worker: every tenant's tokens in one heap, plus queue credentials that reach all tenants. Per-tenant container: scoped per container, but secrets often ride in the image or a shared mount. Per-tenant microVM: injected per VM, shredded after load, destroyed with the guest.
• Noisy neighbour and retry storms — Shared worker: one retry loop or memory hog starves the whole pool. Per-tenant container: cgroups cap CPU and RAM, but page cache, kernel locks, and IO are shared. Per-tenant microVM: a hard vCPU/RAM boundary per guest, plus a per-tenant concurrency cap.
• Cold-start cost — Shared worker: effectively zero, the process is already up. Per-tenant container: image pull plus runtime init, and a cold image isn't cheap. Per-tenant microVM: p50 179ms create via snapshot-restore (p99 ~203ms), or hibernate/wake for warm state; the one-time cold boot before a snapshot exists is about 3 seconds.
• Per-tenant runtime customization — Shared worker: one runtime, one dependency set, one version; every tenant shares your requirements.txt. Per-tenant container: per-image customization, at the cost of an image per tenant. Per-tenant microVM: each tenant gets a whole machine, so runtimes, versions, and system packages differ freely.
• Operational complexity — Shared worker: one deployment, until it's the incident. Per-tenant container: an orchestrator, a registry, and per-tenant build pipelines. Per-tenant microVM: many VM lifecycles to manage — the real cost, and the reason to use a platform that handles scheduling, networking, and teardown rather than wiring up Firecracker yourself.

Capacity is rarely the blocker people expect. A single PandaStack agent pre-allocates 16,384 /30 subnets, so per-sandbox networking isn't the ceiling; host memory and CPU are, and copy-on-write plus hibernate push that ceiling a long way out. A tenant that needs its own database next to its workers runs on the same substrate — a managed Postgres instance is its own VM with a durable volume, created in 30–90s.

The question isn't whether your worker pool can run untrusted code. It's what happens the day it does — "one VM died" or "we rotated every customer's OAuth token over a weekend."

When not to do this

Be honest about the trade, because this model isn't free. If all your activity code is first-party — you wrote it, you review it, tenants supply only data and configuration — a shared worker pool is simpler, denser, and lower-overhead; keep it, and let per-tenant queues handle fairness. The same goes for very high-frequency, very short steps: if you run millions of 20ms activities, a ~179ms create per attempt is the wrong shape — batch that tenant's work into a longer-lived per-tenant VM instead.

It earns its keep the moment tenant-authored code, tenant-supplied dependencies, or tenant-controlled inputs reach your execution path — when you ship a "define your own step" feature, run customer scripts in a data pipeline, or let an AI agent write the activity body. At that point the shared-worker model asks you to defend one address space against code you didn't write, forever, without a single mistake. Swapping that for a hardware boundary and a disposable machine trades a permanent security invariant for a capacity problem you can scale.