Isolating Batch Jobs and Queue Workers with MicroVMs

Almost every backend eventually grows the same organ: a worker that pulls jobs off a queue and runs them. SQS, Kafka, Redis, RabbitMQ, a Postgres table with a FOR UPDATE SKIP LOCKED — the transport changes, the shape doesn't. A loop dequeues a message, does some work, acks, repeats. It's the workhorse of resize-this-image, render-that-PDF, run-this-tenant's-transform, execute-this-scheduled-report. And for a long time you run all of those jobs in the same long-lived worker process, on the same box, sharing the same filesystem and the same kernel — because that's the obvious, cheap way to do it. This post is about the moment that stops being fine, and the pattern that fixes it: one throwaway microVM per job.

The trigger for that moment is usually one of two things. Either the jobs start running code you didn't write — tenant-supplied transforms, customer plugins, a model's generated snippet — and now your worker is an arbitrary-code-execution service wearing a queue costume. Or the jobs are your own code but heavy and adversarial-by-accident: a PDF that decompresses into 40 GiB, a regex that goes exponential, a job that leaks a file descriptor every run until the box tips over. In both cases the shared worker is the wrong container, and the reason is the same: it gives every job the entire machine to poison.

How a shared worker fails: the whole box, all at once

The shared worker is fast and simple until a single job misbehaves, and then the failure isn't scoped to that job — it's scoped to the machine. That's the property that makes shared workers dangerous at scale: one bad job in a shared worker is a murder-suicide pact with every other job on the box. Walk the concrete failure modes, because they're not hypothetical; they're the recurring 3 a.m. pages of anyone who has operated a job fleet.

• The OOM that kills everyone: one job allocates a runaway buffer — a decompression bomb, an unbounded accumulate-into-a-list, a model loaded twice — and the Linux OOM killer fires. It doesn't politely fail the guilty job; it reaps whatever process the kernel decides costs the most memory, which is frequently the worker itself, taking down every other tenant's job running in that process.
• The fork bomb / CPU pig: a job spawns processes without bound, or spins a hot loop, and every other job co-located on that worker starves. cgroups help if you configured them per-job (most shared workers don't — the cgroup is the whole worker), but a fork bomb inside a shared namespace is still contending for the same PID space and scheduler.
• The state that poisons the next job: a job writes a 2 GiB temp file to /tmp and crashes before cleanup; a job leaves a global mutated, a thread running, a socket half-open, an environment variable set. The next job to land on that worker inherits the mess. This is the flaky-CI problem, except it's production and the contamination is silent.
• The leaked file descriptor: a job opens files or sockets and doesn't close them. Over thousands of jobs the worker hits its fd ulimit and starts failing every subsequent job with 'too many open files' — a slow poison that looks like a mystery outage hours after the guilty job finished.
• The escape: if the jobs are untrusted (tenant code, model output), a shared-kernel worker is one container escape or one kernel exploit away from a job reading every other tenant's data on that host. A subprocess or a container is not a security boundary against code that's actively hostile.

The pattern: one throwaway microVM per job

The fix inverts the model. Instead of one long-lived worker that runs every job, you keep the worker as a thin orchestrator that runs no job code at all — it only pulls messages and hands each one to a fresh, disposable microVM. The job runs inside the VM; the VM is destroyed when the job finishes. Nothing about the job survives, and nothing about the job can reach the orchestrator, the host, or any other job.

A microVM is the right boundary here — not a container — because it's a real virtualization boundary the CPU enforces in hardware. On PandaStack every sandbox boots its own guest kernel under Firecracker (the same VMM AWS built to run Lambda and Fargate between tenants), with its own memory, its own filesystem, and its own network namespace. A fork bomb hits the walls of a 2 GiB guest you're about to delete. An OOM kills that guest's kernel, not your fleet. A leaked fd dies with the VM. And because the guest kernel is separate, a container-escape-class exploit has nowhere to escape to.

A shared worker asks every job to be a good citizen. A microVM-per-job assumes every job is a bad one and makes that assumption cheap — because the worst a job can do is wreck a VM that's already scheduled for demolition.

The control loop for each job is five steps and nothing else:

1. Dequeue: pull one message off the queue (SQS/Kafka/Redis/whatever). The orchestrator does this and only this — it never runs the job's code.
2. Create a sandbox: a fresh microVM restored from a baked template snapshot. Give it hard CPU/memory bounds (baked into the template) and a ttl_seconds self-destruct backstop.
3. Deliver the job: write the job's input — and, if the code is job-supplied, the code itself — into the guest filesystem. The job reads it there.
4. Exec with a hard timeout: run the job against its input. Every exec returns stdout, stderr, and an exit code, so success, crash, and timeout are all distinguishable.
5. Read the result and destroy: pull the one result file back out of the VM, then delete the VM. The result is the only thing that escapes. Ack the message on success; leave it for redelivery on failure.

Because the VM is disposable, you write zero cleanup code between jobs. There's no 'reset the worker' step, no scrubbing /tmp, no killing stray threads — the reset is `delete()`, and it's total. A job that corrupts its environment corrupts a VM that no longer exists by the time the next job starts.

A queue worker loop in Python

Here's the whole pattern with the Python SDK: a worker that pulls jobs off a queue, runs each in its own throwaway microVM with a hard time budget, reads a structured result back out, and tears the VM down before acking. Set PANDASTACK_API_KEY in the environment first. `queue` is whatever you use — the shape below maps cleanly onto boto3 SQS, a Kafka consumer, or a Redis BLPOP loop. The point is that the orchestrator never executes a byte of job code.

import json
from pandastack import PandaStack

ps = PandaStack()  # reads PANDASTACK_API_KEY, base url https://api.pandastack.ai

JOB_BUDGET_SECONDS = 120


def run_job(job: dict) -> dict:
    """One job = one throwaway microVM. Hard CPU/mem (from the template) + a hard
    clock. Whatever the job does, it does inside a VM about to be deleted."""
    # 1. Fresh, hardware-isolated VM. Every create restores a baked snapshot
    #    (~179ms p50), so a VM-per-job is cheap. ttl_seconds is a self-destruct
    #    backstop so a hung job can't outlive its budget even if the worker dies.
    sbx = ps.sandboxes.create(
        template="code-interpreter",              # CPU/RAM are fixed by the template snapshot
        ttl_seconds=JOB_BUDGET_SECONDS + 30,
        metadata={"job_id": job["id"], "tenant": job.get("tenant", "internal")},
    )
    try:
        # 2. Deliver the job's input (and its code, if job-supplied) as plain files.
        #    Nothing about this job touches another job's VM or the orchestrator.
        sbx.filesystem.write("/work/input.json", json.dumps(job["payload"]).encode())
        sbx.filesystem.write("/work/run.py", job["code"].encode())

        # 3. Exec with a hard timeout. This is the ONLY place the job's code runs.
        #    A loop-forever job is killed; a fork bomb hits the VM's walls, not yours.
        r = sbx.exec("cd /work && python3 run.py", timeout_seconds=JOB_BUDGET_SECONDS)
        if r.exit_code != 0:
            return {"id": job["id"], "ok": False, "error": r.stderr[-1000:]}

        # 4. Read the structured result back out. This is the only thing that escapes.
        out = sbx.filesystem.read("/work/result.json")
        return {"id": job["id"], "ok": True, "result": json.loads(out)}
    except Exception as e:
        # OOM / timeout / crash lands here. One job fails; the fleet does not.
        return {"id": job["id"], "ok": False, "error": str(e)}
    finally:
        # 5. Always destroy. The leaked fd, the 2 GiB temp file, the stray
        #    process — all of it dies here. The next job starts pristine.
        sbx.delete()


def worker(queue):
    """Thin orchestrator: dequeue, isolate, ack. It NEVER runs job code itself."""
    for msg in queue:                          # SQS receive / Kafka poll / Redis BLPOP
        job = json.loads(msg.body)
        outcome = run_job(job)
        if outcome["ok"]:
            msg.ack()                          # only ack once the result is safely out
        else:
            msg.nack()                         # leave it for redelivery / DLQ

Three safety rails are doing the real work here, and none of them is optional. The `timeout_seconds` on `exec` is the circuit breaker for a job that loops forever. The `ttl_seconds` on create is the backstop so a VM self-destructs even if the worker process crashes mid-job and never reaches the `finally`. And the ack-after-result ordering means a job whose VM died still leaves its message on the queue for redelivery — you never lose a job because its sandbox misbehaved. Untrusted or heavy, every job gets a wall, a clock, and a hard cap, then gets thrown away.

Guaranteed clean teardown: the property you actually bought

The headline benefit of per-job VMs isn't security theater — it's that every job starts from an identical, known-clean state and leaves nothing behind. This is the property that shared workers can never give you no matter how carefully you write cleanup code, because cleanup code runs only when the job cooperates, and the jobs that break your fleet are precisely the ones that don't cooperate: they crash before cleanup, they leak resources you didn't know to release, they mutate state you didn't know was shared.

With a microVM, teardown is not code you wrote and might have gotten wrong — it's `delete()` returning the VM's memory and disk to the host. There is no partial cleanup, no 'we forgot to close that,' no accumulating drift over a million jobs. Every job in a batch of a million runs against byte-for-byte the same baked snapshot. That reproducibility is worth as much operationally as the isolation is worth for security: the 700,001st job behaves exactly like the first, because the environment literally is the first, restored fresh.

• No temp-file poisoning: a job's /tmp is its own VM's /tmp, gone on delete. The next job's disk is a fresh reflink clone of the pristine template.
• No fd / process leaks: the guest kernel and everything running under it is destroyed. There is nothing to leak into.
• No environment drift: env vars, mutated globals, half-finished migrations, cached bad state — all scoped to a VM that ceases to exist.
• No cleanup code to maintain: you delete the entire machine, so you never write (or forget) the try/finally that scrubs shared state.

Scaling many jobs cheaply: snapshot-restore, fork, and idle-cost ≈ 0

The reflexive objection to 'a VM per job' is cost — a VM per job sounds like renting a datacenter to run a cron. That intuition is calibrated on cold-booting conventional VMs, which take tens of seconds and hold a warm pool of idle capacity you pay for around the clock. A microVM snapshot restore breaks both assumptions, which is the whole reason per-job isolation is affordable rather than aspirational.

• Create is a restore, not a boot: a PandaStack create runs at ~179ms p50 and ~203ms p99, because the heavy step is a ~49ms snapshot-restore against a baked template — Firecracker maps a frozen, already-booted VM back in and resumes it mid-instruction. The genuine ~3-second cold boot happens once per template, gets baked into a snapshot, and is amortized away forever.
• No warm pool to pay for: there are no idle VMs kept hot waiting for jobs. A VM exists for the seconds a job runs, then it's gone. Between bursts your per-job compute cost is effectively zero — you pay for work, not for standby. That's idle-cost ≈ 0, and it's the single biggest structural cost advantage over a fleet of always-on workers.
• Copy-on-write memory and disk: restoring the Nth identical VM doesn't copy gigabytes. Guest memory is MAP_PRIVATE (pages copy only on write) and the rootfs is an XFS reflink clone, so many concurrent identical job VMs share the baked pages until each writes. Concurrency is bounded by real host RAM/CPU, not by copying overhead.
• Enormous address-space headroom: an agent pre-allocates 16,384 /30 subnets, so a per-job VM pool has room to spare before networking is a bottleneck. The real ceiling is memory and CPU, which you govern by capping worker concurrency.

There's a second scaling lever when jobs share a warm starting state — say every job in a batch needs the same 4 GiB model loaded, or the same big dataset warmed into memory. Instead of restoring a cold template per job, you boot one VM, warm it, then fork it per job. A same-host fork is 400–750ms (cross-host 1.2–3.5s) and it's copy-on-write from the parent's live memory, so each forked job starts with the warm state already in RAM and pays only for the pages it dirties. You get the warm-start latency of a shared worker with the clean-teardown isolation of a fresh VM.

Warm-starting jobs with a fork-per-job pool

When every job needs the same expensive preamble — load a model, open a dataset, warm a cache — paying for that preamble once and forking from it is dramatically cheaper than redoing it per job. Boot a parent VM, run the preamble, then fork the parent per job. Each fork inherits the parent's warm memory copy-on-write, runs its job, and gets deleted. The parent is never touched by a job, so it stays pristine and forkable for the life of the batch.

import json
from pandastack import PandaStack

ps = PandaStack()  # reads PANDASTACK_API_KEY

# 1. Boot ONE parent VM and pay the expensive warm-up exactly once.
#    The parent never runs a job, so it stays clean and forkable.
parent = ps.sandboxes.create(template="code-interpreter", ttl_seconds=3600)
parent.filesystem.write("/work/warmup.py", WARMUP_CODE.encode())
parent.exec("cd /work && python3 warmup.py", timeout_seconds=300)  # load model / dataset


def run_job_forked(job: dict) -> dict:
    # 2. Fork the warm parent per job: same-host fork is 400-750ms and copies
    #    the parent's warm memory copy-on-write, so the model is ALREADY in RAM.
    child = parent.fork(ttl_seconds=180)
    try:
        child.filesystem.write("/work/input.json", json.dumps(job["payload"]).encode())
        # The job runs against warm state, but in its OWN throwaway VM. It cannot
        # corrupt the parent or any sibling job — a fork is a private CoW branch.
        r = child.exec("cd /work && python3 job.py", timeout_seconds=120)
        if r.exit_code != 0:
            return {"id": job["id"], "ok": False, "error": r.stderr[-1000:]}
        return {"id": job["id"], "ok": True,
                "result": json.loads(child.filesystem.read("/work/result.json"))}
    finally:
        child.delete()  # only the fork dies; the warm parent lives on for the next job


# for job in batch: run_job_forked(job)   # each job: warm start, clean teardown
# parent.delete()                          # tear the parent down when the batch is done

The isolation story survives forking intact: a fork is a private copy-on-write branch of the parent, so a job can scribble all over its guest — mutate the loaded model's weights, corrupt the cache, wedge the process — and none of it touches the parent or any sibling, because copy-on-write means the moment a fork writes a page, it gets its own private copy. You get shared warm state for reads and total isolation for writes. When the batch is done, delete the parent and the whole tree is gone.

Shared worker vs. microVM per job, concern by concern

Laid side by side, the difference isn't a matter of degree — it's which layer the failure is contained at. A shared worker contains a bad job at the machine boundary (i.e., it doesn't); a microVM contains it at the job boundary.

• Blast radius of a bad job — Shared worker: the whole box; an OOM or fork bomb takes down every co-located job. MicroVM per job: exactly one job; the guilty VM dies, the rest are untouched.
• State between jobs — Shared worker: leaks by default; /tmp, globals, fds, and stray processes carry over unless you scrub perfectly every time. MicroVM per job: none by construction; every job restores the same clean snapshot.
• Memory / CPU limits — Shared worker: soft and shared; a per-job cgroup is extra work most shared workers skip, and a fork bomb still contends for the shared scheduler and PID space. MicroVM per job: hard and hypervisor-enforced; the guest gets a fixed RAM/vCPU it physically cannot exceed.
• Noisy neighbor — Shared worker: unavoidable; jobs contend for the same CPU, memory, and I/O. MicroVM per job: eliminated; each job has its own guest kernel and its own resource envelope.
• Untrusted / tenant code — Shared worker: unsafe; a shared kernel is one escape away from cross-tenant data access. MicroVM per job: safe; a separate guest kernel behind a hardware virtualization boundary is the wall AWS built Firecracker to provide.
• Cleanup code — Shared worker: you write it, maintain it, and it fails to run exactly when a job crashes. MicroVM per job: none; teardown is delete(), which is total and can't be skipped.
• Idle cost — Shared worker: you pay to keep workers warm whether or not jobs are flowing. MicroVM per job: ≈ 0; a VM exists only while a job runs, no warm pool.
• Latency floor — Shared worker: near-zero once the worker is up. MicroVM per job: the ~179ms restore per job (or a 400–750ms fork if you need warm state), plus your job.

The one column where the shared worker wins is that last one: latency. A per-job restore adds ~179ms that an already-warm shared worker doesn't pay. For most batch and queue work that's noise against a job that runs for seconds to minutes; for ultra-high-rate, ultra-short jobs it can matter, and that's exactly where the fork-per-job pool (warm parent, cheap CoW children) or a per-tenant persistent VM earns its keep.

When a shared worker is still fine

Per-job microVMs are real infrastructure — you own routing jobs in, reading results out, template hygiene, and concurrency governance. Don't reach for them when the threat model doesn't call for it.

• The jobs are all your own trusted code and well-behaved. If only your engineers write the job code, it has bounded memory, and it cleans up after itself, a shared worker with per-job cgroups is probably enough — the multi-tenant-adversary and resource-poisoning arguments are weaker.
• Jobs are sub-millisecond and enormous in volume. If a job is a trivial in-memory transform that runs in under a millisecond, a ~179ms VM create dwarfs the work. Batch such jobs into chunks and run a chunk per VM, or keep them on a shared worker.
• A managed queue-consumer FaaS already fits. If your jobs map cleanly onto Lambda/Cloud Run jobs and you don't need byte-level control of the runtime or the isolation plane, use one — those run on microVMs under the hood and you skip operating the fleet.
• The work is I/O-bound fan-out with no untrusted code. If a 'job' is just 'call this API and write the row,' with no code execution and no poisoning risk, async concurrency on one worker is simpler and cheaper.

Reach for per-job microVMs when jobs run code you can't fully trust, when one job's OOM or leak has been taking down unrelated jobs, when tenants must be isolated from each other by a hardware boundary, or when you're tired of writing cleanup code that fails exactly when a job crashes. In those cases snapshot-restore makes strong per-job isolation cheap enough to be the default — and a bad job stops being a fleet-wide incident and becomes a single `ok: False` in your results, exactly where it belongs.