Per-Tenant Queue Consumers in Isolated microVMs

Every SaaS backend of a certain age has one: a worker fleet. A dozen boxes running Sidekiq, Celery, BullMQ, Oban, or a hand-rolled loop over SELECT ... FOR UPDATE SKIP LOCKED, all chewing on one shared queue. It is the cheapest possible design and for a long time it is the correct one. Every tenant's jobs go into the same pipe, come out on the same worker processes, and share a memory space, a filesystem, a kernel, and — this is the part people notice last — a set of credentials.

The shared worker fleet is fine right up until the first customer discovers the bulk import button. I'm Ajay, I build PandaStack, so read this as opinionated: I think the shared-fleet-plus-fairness-heuristics design is a local maximum, and the exit from it is a microVM per tenant (or per tenant-batch) rather than another round of weighted queues. The argument only works because a snapshot-restored VM boots in about the time of a mediocre HTTP round trip. Before that was true, this post would have been fantasy.

The four things that push teams off the shared fleet

Nobody rewrites a working worker fleet for aesthetic reasons. In my experience it is always one of four incidents, and they arrive in roughly this order as a product matures.

1. Noisy neighbours: one tenant enqueues 200,000 jobs

A customer clicks Import CSV on a 200,000-row file, or their integration retries a webhook fan-out, or their nightly sync finally succeeds after failing for a week and flushes a backlog. Two hundred thousand jobs land in a FIFO queue. Your fleet has, say, 200 worker threads. Every other tenant's job is now behind 200,000 jobs, and "a report is ready in 4 seconds" becomes "a report is ready in about six hours." That is classic head-of-line blocking: the queue is fair in the worst possible way, first-come-first-served, and one tenant came first two hundred thousand times.

The standard fix is per-tenant queues plus a fair scheduler — round-robin across tenant queues, or weighted by plan tier. Do it; it genuinely fixes ordering. What it does not fix is capacity. Round-robin means the whale's jobs are interleaved with everyone else's, not that they consume less CPU. If that tenant's jobs are image transcodes that peg a core for 30 seconds each, they still occupy most of your fleet's CPU-seconds, and everyone else's latency still degrades — just less catastrophically. Fair queuing distributes a scarce resource; it does not create one. And it does nothing at all about the page cache, the memory allocator, or the shared Postgres connection pool that the whale is also monopolising.

2. Poison jobs: the crash that takes N tenants down with it

A worker process is usually running several jobs concurrently — threads in Sidekiq, prefork children with concurrency in Celery, an event loop with parallel handlers in BullMQ. When one of those jobs dies in a way the runtime cannot catch, it does not die alone. A malformed PDF that decompresses into 40 GiB triggers the OOM killer, and the kernel does not reap the guilty job, it reaps the fattest process — which is the worker holding nine other tenants' in-flight jobs. A segfault in a native image library, a stack overflow in recursive user-supplied JSON, a C extension that corrupts the heap: same shape, same casualties.

Then the retry machinery converts an incident into a loop. The killed worker never acked, so after the visibility timeout the broker redelivers the poison job to another worker, which also dies, taking another nine tenants' jobs with it. You have built a self-inflicted denial of service and you are paying for the compute. Dead-letter queues and max-retry counts bound it, but only after the damage; the in-flight jobs that died as collateral were fine, and now they are retried too.

3. Credential blast radius: every tenant's tokens in one heap

To run Tenant A's "push to their Salesforce" job and Tenant B's "charge a card in their Stripe account" job, the same process must hold both tenants' credentials in memory. However disciplined your secrets manager is, the moment a token is fetched it lives in the process heap alongside everyone else's, reachable by any code running in that process. That includes a dependency that helpfully logs its environment on startup, an exception reporter that captures locals, a heap dump taken during an OOM investigation, and any job that can read /proc/self/environ.

The broker credentials are usually worse. A worker authenticates to Redis or SQS or Kafka as "the worker," not as a tenant, so a job that can reach the broker connection can generally read, enqueue, or delete any tenant's messages. The isolation you think you have is enforced entirely by the fact that your own code does not currently choose to do that.

4. Tenant-supplied code: your worker is now an RCE service

This is the one that ends the argument. The day you ship "write a custom transform for your import," "define a JavaScript step in your automation," or "let the AI generate a mapping function," your worker fleet becomes a remote code execution service with extra steps. Every mitigation people reach for at that point — a restricted interpreter, a thread with a timeout, an AST allowlist, a seccomp filter bolted onto a process that also holds a hundred customers' OAuth tokens — is a defence in depth story built on a foundation of no depth. A timeout cannot un-read memory that has already been read.

The shape of the fix: a machine per tenant, not a thread per job

The change is smaller than a rewrite. Keep the broker. Keep the job definitions, the retry policy, the dead-letter queue, the dashboard. What changes is one function: the thing that takes a job and executes it. Instead of running the handler in-process, a thin dispatcher leases a batch of one tenant's jobs, creates a Firecracker microVM for that tenant, injects only that tenant's environment, runs the handler inside the guest, collects results, and lets the VM die.

A Firecracker microVM is a real virtual machine — its own guest kernel, its own memory, its own virtual block device, its own network namespace, confined by KVM hardware virtualization. That is the same boundary AWS Lambda uses for untrusted code from millions of accounts. Once the boundary is per tenant rather than per thread, the four failure modes change category:

• Noisy neighbours become a capacity dial. A tenant's burst consumes that tenant's VM budget — an integer in the dispatcher. Other tenants' jobs run in other guests on other cores; there is no shared queue position to be stuck behind, and no shared allocator to exhaust.
• Poison jobs become a contained crash. An OOM kills that guest, because the VM's memory ceiling is the VM's, not the host's. A segfault, a fork bomb, an rm -rf of the root filesystem: all of them destroy a machine that was going to be destroyed in a few seconds anyway. Nobody else's in-flight work dies with it.
• Credential blast radius collapses to one tenant. The guest is created fresh and receives only the credentials this tenant's jobs need. There is no other tenant's token on that machine to steal, because none was ever put there. The broker credentials stay host-side with the dispatcher, so guest code cannot read, enqueue, or delete anyone's messages — including its own tenant's.
• Tenant-supplied code becomes tractable. You are no longer trying to make an interpreter safe; you are running hostile code on a disposable machine whose only outbound network path is the one you allowed. Per-tenant egress rules are enforced at the VM's network namespace, so a transform step that tries to exfiltrate to a pastebin gets a connection refused rather than a 200.

The historical objection is start cost, and it was a good objection. A VM per job is absurd if a VM takes ten seconds to appear. On PandaStack every create restores a pre-baked snapshot instead of cold-booting: p50 179ms, p99 around 203ms end to end, with the snapshot-restore step itself near 49ms. The first-ever boot of a template, before a snapshot exists, is about 3 seconds and is paid once. At 179ms, per-job isolation stops being economically silly for anything that takes longer than a second — which is most of what is in your queue, because fast things do not get enqueued.

The other half of the economics is idle. A shared fleet is sized for peak and paid for at all times; a tenant who enqueues nothing for six days still costs you a slice of a reserved instance. A per-tenant VM that scales to zero costs storage while it sleeps and compute only while it runs. For the long tail of small tenants — which in most SaaS is the overwhelming majority of accounts and a rounding error of the work — that difference is the whole business case.

Architecture: dispatcher outside, handler inside

The dispatcher is the only component that talks to the broker. It runs on trusted infrastructure, holds the broker credentials and the secrets-manager client, and never executes tenant code. Its loop is: pick a tenant that has work and has budget, lease a batch of that tenant's jobs, boot a sandbox, write the batch and the tenant's scoped environment into the guest, exec the handler, read the results file back, ack or retry each job accordingly, and walk away — the TTL reaps the machine whether or not the dispatcher survives to clean up.

That last detail matters more than it looks. Teardown that depends on your own cleanup code running is teardown that fails exactly when you need it: during a deploy, an OOM, a panic, a node eviction. A TTL set at create time is enforced by the platform, not by your goodwill.

import json
from pandastack import Sandbox

BATCH_SIZE = 25          # jobs per VM lifetime -- see "sizing the batch"
JOB_BUDGET_SECONDS = 300


def drain_tenant(tenant_id: str, broker, vault) -> None:
    """Run ONE tenant's next batch of queued jobs in its own microVM.

    The broker connection and the vault client stay here, on the host.
    Nothing inside the guest can read another tenant's messages, ack a
    job, or reach the secrets manager -- it has no credentials for any
    of them and no route to them.
    """
    batch = broker.lease(tenant_id, limit=BATCH_SIZE, invisible_for=600)
    if not batch:
        return

    # Only this tenant's integration tokens. Fetched per batch, scoped by
    # the vault, never baked into the template, never shared with a peer.
    env = vault.scoped_env(tenant_id)

    # ttl_seconds is the backstop: if this dispatcher dies mid-batch, the
    # platform still destroys the VM. Cleanup you have to remember to run
    # is cleanup that does not happen during an incident.
    sbx = Sandbox.create(
        template="base",
        ttl_seconds=JOB_BUDGET_SECONDS + 120,
        metadata={"tenant": tenant_id, "batch": batch.id},
    )

    sbx.filesystem.write(
        "/run/jobs/batch.json",
        json.dumps([j.payload for j in batch.jobs]).encode(),
    )
    sbx.filesystem.write("/run/jobs/env.json", json.dumps(env).encode())

    # The handler is the same code you run today -- it just runs on a
    # kernel of its own. Worst case it OOMs, segfaults or fork-bombs,
    # and the only casualty is a machine with a 7-minute lifespan.
    res = sbx.exec(
        "/opt/worker/run-batch.sh /run/jobs/batch.json",
        timeout_seconds=JOB_BUDGET_SECONDS,
    )

    if res.exit_code != 0:
        # 137 = SIGKILL/OOM, 124 = timeout(1). The whole batch goes back
        # for redelivery on a fresh guest; nobody else's work was touched.
        broker.nack_all(batch, reason=res.stderr[-4000:])
        return

    # Per-job outcomes, so one poison payload does not re-run 24 healthy
    # siblings. Bisect on retry if you want to isolate it automatically.
    results = json.loads(sbx.filesystem.read("/run/jobs/results.json"))
    for job in batch.jobs:
        outcome = results.get(job.id, {"status": "missing"})
        if outcome["status"] == "ok":
            broker.ack(job)
        else:
            broker.nack(job, reason=outcome.get("error", "no result"))

    # No delete call, no finally block, no reaper cron. The TTL owns the
    # machine's lifetime and the secrets die with it.

Notice what is not in that function: no thread pool, no signal.alarm, no attempt to reclaim a worker after user code misbehaved, no cleanup of /tmp. The recovery primitive is "delete the machine," which is the only one that reliably works against code that is actively hostile or merely catastrophically buggy.

Sizing the batch: per tenant, not per message

The obvious mistake is one VM per message. For a 40ms job, a 179ms create is a 4x overhead and you have built the world's most secure way to waste money. Batch instead: lease N of a tenant's jobs and run them in one VM lifetime. The isolation boundary stays where it matters — between tenants — while the create cost is amortised across the batch. Pick N so that the batch's expected runtime is comfortably longer than the create, and short enough that a redelivery after a crash does not re-run an hour of work.

1. Sub-second jobs, high volume: batch aggressively (hundreds per VM) or keep a per-tenant VM warm across batches and feed it work over the exec channel. The boundary is still per tenant.
2. Seconds-to-minutes jobs: batch modestly (tens). This is the sweet spot — the create cost disappears into the noise and a redelivery costs you one batch.
3. Long, expensive jobs: one VM per job is correct. A 20-minute video transcode does not care about 179ms, and per-job isolation gives you exact attribution of CPU and memory for billing.
4. Bursty tenants with expensive warm state — a big dependency tree, a JIT, a headless browser: hibernate the VM between bursts instead of destroying it. Idle costs storage rather than compute, and it wakes with the warm state intact.

What you actually give up

The honest cost is warm process state, and it is not trivial. A long-lived worker accumulates things worth having: a filled connection pool to Postgres and Redis, a warmed JIT, a loaded model, a memoised template cache, a DNS cache, a TLS session cache. A fresh VM has none of that. If your handler's first act is to open five TLS connections and load a 300 MB model, you have moved the cost from "once per deploy" to "once per batch," and if your batches are small, that is a regression.

There are three mitigations and you will want all of them. Bake the expensive warm state into the snapshot, so the restored guest comes up with the model already resident and the caches already populated — that is what snapshot-restore is actually good for. Size batches so the warm-up amortises. And for tenants with genuinely continuous load, keep their VM alive across batches rather than recreating it; the boundary you care about is per tenant, and nothing says the machine must be per batch.

Database connections deserve a specific warning. Ten shared workers with a pool of 20 each is 200 connections. Two hundred per-tenant VMs each opening 5 connections is 1,000, and Postgres will have opinions. Put a pooler in front, give the guests short-lived pooled connections rather than direct ones, or move the tenant's data access behind a host-side API the guest calls — which you may want anyway, since it is another credential the guest then never holds.

Long-poll patterns get more expensive too. If your consumer's normal state is blocking on a five-second BRPOP against Redis, a VM per tenant means paying for many mostly-idle machines to sit in a syscall. The fix is to invert it: the dispatcher long-polls on the host side, cheaply, and only creates a VM when there is actual work. Do not lift a blocking consumer loop into the guest and then be surprised by the bill.

Four ways to run a multi-tenant queue

Softest boundary to hardest. If you are evaluating a specific broker or container runtime, check its isolation and concurrency semantics against its own docs — the details vary a lot by version and configuration.

• Noisy neighbour — Shared fleet, one queue: total head-of-line blocking; one tenant's 200k jobs are everyone's latency. Per-tenant queues on a shared fleet: ordering is fair, CPU is not — the whale still eats the fleet's capacity, page cache and connection pool. Container per job: cgroups cap CPU and RAM per job, but IO, page cache and kernel locks are shared, and the node is still oversubscribed by one tenant. microVM per tenant-batch: a hard vCPU and RAM boundary per guest plus a per-tenant concurrency cap, so a burst consumes that tenant's budget and stops.
• Poison-job blast radius — Shared fleet: an OOM or segfault kills the worker and every concurrent job in it, then redelivery repeats the murder on the next worker. Per-tenant queues on a shared fleet: identical — the queue split changes ordering, not process boundaries. Container per job: contained to that container, provided you actually set memory limits; a kernel-level crash still affects the node. microVM per tenant-batch: contained to a throwaway guest with its own kernel; the retry starts from a pristine snapshot.
• Credential isolation — Shared fleet: every tenant's tokens in one heap, plus broker credentials that reach all tenants' messages. Per-tenant queues on a shared fleet: no better; the process is still shared. Container per job: scoped per container, but secrets frequently ride in the image, a shared mount, or the orchestrator's env, and the node's metadata endpoint is one curl away. microVM per tenant-batch: fetched per batch, injected into one guest, destroyed with it; broker and vault credentials never leave the host.
• Untrusted-code safety — Shared fleet: none. Running tenant-authored transforms in a shared worker is remote code execution as a feature. Per-tenant queues on a shared fleet: none, for the same reason. Container per job: a real boundary against accidents and a soft one against attackers — namespaces and seccomp on a kernel shared with every other tenant on the node. microVM per tenant-batch: own guest kernel under KVM, so an escape requires a hypervisor break, and per-VM egress rules bound what a successful one could reach.
• Start cost — Shared fleet: effectively zero; the process is already warm with pools and caches. Per-tenant queues on a shared fleet: also zero, same processes. Container per job: milliseconds if the image is cached and the node has room; seconds to tens of seconds on a cold image pull or a scale-up. microVM per tenant-batch: p50 179ms create via snapshot-restore, p99 around 203ms, about 3 seconds for the one-time cold boot before a snapshot exists — amortised across the batch, and avoidable entirely by keeping a busy tenant's VM warm.

Capacity is rarely the wall people expect. A single PandaStack agent pre-allocates 16,384 /30 subnets, so per-sandbox networking is not the ceiling — host memory and CPU are, and copy-on-write memory plus scale-to-zero push that a long way out. If a tenant also needs its own datastore next to its workers, the same substrate runs it: a managed Postgres instance is its own VM with a durable volume, created in 30–90s.

How to get there without a rewrite weekend

Do not flip the whole fleet. The migration that works is per job class, starting with the one that hurts. Pick the queue with the worst noisy-neighbour record or the most tenant-controlled input — usually imports, exports, or the custom-transform feature — and route only that class through the dispatcher. Leave everything else on the shared fleet, where it is fine.

1. Make the handler runnable as a standalone process that reads a batch file and writes a results file. If it currently reaches into framework globals or the broker connection, that coupling is the actual migration work, and it is worth doing regardless.
2. Move secrets from process-wide config to a per-tenant scoped fetch. This is the security win even if you never ship the VMs, and it is the step that reveals which credentials were quietly shared.
3. Run the dispatcher alongside the existing fleet for one job class, with a per-tenant concurrency cap set low. Compare end-to-end latency and cost against the shared path for a week.
4. Tune the batch size against real job durations, then move the next job class. Keep the shared fleet for first-party, high-frequency, low-risk work — it is still the cheapest thing that could possibly work for that.

And be willing to not do this. If every handler is code you wrote and reviewed, tenants supply only data, and your jobs are uniform and short, a shared fleet with per-tenant queues and sane rate limits is simpler, denser and cheaper. Keep it. The model earns its keep the moment tenant-authored code, tenant-supplied dependencies, or genuinely adversarial inputs reach your execution path — because at that point the shared fleet is asking you to defend one address space, containing every customer's credentials, against code you did not write, forever, without a single mistake.

The question is not whether your worker fleet can survive a bad job. It is what the postmortem says: "one disposable VM died and the job retried" or "we rotated every customer's integration token over a weekend."