Isolating Multi-Tenant GenAI Image Jobs in MicroVMs

A GenAI image platform is one of the most permissive things you can operate. To be useful it lets tenants bring their own pipelines: a custom ComfyUI-style node graph, a handful of pip-installed "community" nodes, a LoRA someone trained on a weekend, a checkpoint downloaded from a model hub. Each of those is code or a blob you did not write, uploaded by someone you've never met, and your platform's whole job is to execute it and hand back a PNG. If all of that runs on one shared box, you have quietly signed up to run arbitrary code from every tenant next to every other tenant's prompts, outputs, and credentials.

I'm Ajay; I built PandaStack. This post is about the containment boundary for that workload: give each generation job its own Firecracker microVM, so the untrusted graph and the untrusted checkpoint load inside a disposable guest with its own kernel, not inside the process that holds everyone else's data. I'll be upfront about the part PandaStack does not do — heavy GPU diffusion still wants a GPU host — and precise about the part it does: the isolation and orchestration boundary, plus the CPU-side pre/post-processing where a poisoned file does its damage.

The checkpoint is not a file, it's a program

The dangerous assumption is that a model checkpoint is inert data you load. Some formats are — `.safetensors` was designed so that loading weights is not code execution, and that name is a promise. But a huge amount of what real users upload is a PyTorch pickle: `.ckpt`, `.pt`, `.pth`, `.bin`. Unpickling is Turing-complete by design, so a crafted checkpoint can invoke `os.system`, import anything, and open a socket the instant a node calls `torch.load`. "Load this model" is a remote-code-execution primitive wearing a file extension.

And it isn't only the weights. A ComfyUI graph is an execution plan, and "custom nodes" are Python modules you `pip install` and import into the same interpreter that runs everyone's jobs. The pre-processing (decode the uploaded image, resize, strip EXIF) and post-processing (upscale, watermark, re-encode) are parsers chewing on attacker-controlled bytes — a malformed PNG or a decompression-bomb image is a native-code bug or an out-of-memory event waiting to happen. Every one of those is a place where a shared box leaks.

The model: one microVM per generation job

The structural fix is to stop loading untrusted graphs and checkpoints inside your trusted process at all. Your control plane — the request router, the queue, the part with the storage and database credentials — stays on a trusted host and drives disposable sandboxes over an API. For each job you create a fresh Firecracker microVM, write the tenant's checkpoint and graph into the guest, run the load-and-generate step inside the guest, read the resulting PNG back over the wire, and destroy the VM. When the job is done, the entire blast radius goes with it: no leftover files, no warmed-up interpreter holding another tenant's pickle, no state to bleed into the next prompt.

This is only practical because the isolation is nearly free at the boundary. A cold Firecracker boot is a couple of seconds, but PandaStack creates sandboxes by restoring a baked snapshot on demand — the restore step is around 49ms, and end-to-end create is p50 179ms (p99 ~203ms); a full cold boot (~3s) only happens once at bake time. So a fresh, hardware-isolated machine per job costs about a fifth of a second, not a fifth of an hour. Each agent also has 16,384 pre-allocated /30 network subnets, so fanning out one VM per concurrent job isn't a slot-capacity problem — memory and CPU are the real ceiling.

Here's the per-job shape with the Python SDK. The tenant's untrusted checkpoint and graph are written into the guest, loaded there, and the generated PNG is pulled back through the filesystem API. Your host never unpickles a stranger's checkpoint.

from pandastack import Sandbox

def generate_for_tenant(tenant_id: str, checkpoint: bytes, graph: str,
                        init_image: bytes) -> bytes | None:
    # A fresh, hardware-isolated guest for THIS job. ~179ms p50 to create.
    sbx = Sandbox.create(
        template="code-interpreter",
        ttl_seconds=300,               # hard ceiling: a runaway graph can't run forever
        metadata={"tenant": tenant_id},
    )
    try:
        # Push the UNTRUSTED artifacts INTO the guest. Never load a checkpoint on
        # the host -- torch.load() on a pickle is arbitrary code execution.
        sbx.filesystem.write("/work/model.ckpt", checkpoint)
        sbx.filesystem.write("/work/graph.json", graph)
        sbx.filesystem.write("/work/init.png", init_image)

        # Load + run the pipeline INSIDE the guest. If this checkpoint's
        # __reduce__ fires an os.system(), it fires in a disposable VM that
        # holds no other tenant's data.
        result = sbx.exec(
            "cd /work && python3 run_pipeline.py graph.json model.ckpt init.png",
            timeout_seconds=240,
        )
        if result.exit_code != 0:
            return None            # generation failed; nothing to read back

        # Pull the generated PNG back out as raw bytes.
        return sbx.filesystem.read("/work/out.png")
    finally:
        sbx.delete()               # blast radius destroyed with the VM

Per-job resource ceilings and noisy neighbors

Isolation isn't only about secrets; it's about capacity. Image jobs spike — a 4K upscale, a batch of 16 samples, a graph with a runaway loop, or an image decompression bomb can each pin CPU and eat RAM. On a shared box, one tenant's spike steals from everyone. A microVM has a fixed vCPU and RAM budget baked into its snapshot, so a tenant's runaway job saturates its own guest and no one else's. Pair that with a `ttl_seconds` cap and a job that never terminates becomes a guest that gets reclaimed, not a 3am page. The decompression-bomb image that would OOM a shared worker hits this guest's own OOM boundary and dies there.

• Isolation strength — Shared multi-tenant box: none; every tenant's graph, custom nodes, and unpickled checkpoint share one address space with everyone's prompts and output-bucket credentials. MicroVM per job: hardware-virtualized guest with its own kernel — a compromised checkpoint or malicious node stays inside one disposable VM.
• Untrusted checkpoint load — Shared box: torch.load runs a stranger's pickle in your trusted process, one uploaded file from cross-tenant RCE. MicroVM: the load happens in a throwaway guest that holds no other tenant, so __reduce__ fires into a void.
• Noisy neighbor / resource spikes — Shared box: one tenant's 4K upscale, 16-image batch, or runaway node graph starves everyone. MicroVM: fixed vCPU/RAM per guest plus ttl_seconds; a spike saturates only itself and is then reclaimed.
• State bleed between jobs — Shared box: temp files, cached tensors, and leaked buffers persist across users; prompt A's output can linger for prompt B. MicroVM: the guest is destroyed on delete/ttl — there is no next job for anything to leak into.
• Custom-node / pip supply chain — Shared box: a typosquatted community node executes in the shared interpreter with the whole host's reach. MicroVM: it executes inside the guest with only that guest's filesystem and network namespace.

For a tenant who reuses the same base checkpoint across many jobs, you don't have to reload it every time. Do the expensive load once, snapshot the guest with the model resident, and fork that warm snapshot per concurrent job — a same-host fork lands in roughly 400–750ms and shares the loaded weights copy-on-write, so ten concurrent generations share the memory pages until each diverges (a cross-host fork is 1.2–3.5s when the warm snapshot lives on another agent). The trust rule still holds: keep that warm snapshot per-tenant, never a shared baseline that folds multiple tenants' checkpoints into one image.

# --- Warm ONCE per tenant: load the base checkpoint, then snapshot it. ---
warm = Sandbox.create(template="code-interpreter", ttl_seconds=1800)
warm.filesystem.write("/work/model.ckpt", tenant_checkpoint)
warm.exec("cd /work && python3 load_checkpoint.py model.ckpt",
          timeout_seconds=600)
snap = warm.snapshot()          # capture the guest with the model resident
warm.hibernate()

# --- Burst? Fork the warm snapshot per concurrent job (~400-750ms same-host,
# copy-on-write, so N forks don't re-load N copies of the checkpoint). ---
def render(graph: str, init_image: bytes) -> bytes | None:
    with snap.fork(ttl_seconds=180) as sbx:
        sbx.filesystem.write("/work/graph.json", graph)
        sbx.filesystem.write("/work/init.png", init_image)
        r = sbx.exec("cd /work && python3 run_pipeline.py graph.json init.png",
                     timeout_seconds=150)
        return sbx.filesystem.read("/work/out.png") if r.exit_code == 0 else None

Where the boundary actually is (and isn't)

The honest architecture is a split. The untrusted, attacker-controlled steps — deserializing the checkpoint, executing the graph and its custom nodes, decoding and re-encoding images — run inside a per-job microVM. The GPU-heavy diffusion step, if you have one, runs on a GPU host that this sandbox drives as a downstream service; PandaStack orchestrates the job and contains the code, it doesn't pretend to do the matmuls. And the sandbox isolates execution, not your secrets: never inject storage keys, signing tokens, or DB credentials the running graph shouldn't see into the guest environment. Egress is real too — if your threat model includes a checkpoint exfiltrating another tenant's prior output, restrict the guest's outbound network at the network layer rather than trusting the pipeline to behave.

None of this makes generation slower in the way that matters. The pipeline still runs at pipeline speed; what changes is where it runs and who it can reach. The tenant's untrusted checkpoint and node graph execute in a disposable guest holding none of your other customers' data, your router reasons over the returned PNG on a trusted host, and at ~179ms per create plus ~400ms per warm fork, the containment is close to free. For the general untrusted-code pattern behind this, see /blog/how-to-sandbox-untrusted-code; for the per-tenant model-serving cousin, /blog/microvm-per-tenant-ml-inference-isolation; and for the snapshot-and-fork mechanics, /blog/snapshot-and-fork-explained.