Per-Tenant ML Inference Isolation With MicroVMs

"Bring your own model" is one of the most reasonable-sounding features a platform can ship and one of the most dangerous. A customer uploads a set of weights and a little Python — a preprocessor to reshape their inputs, a postprocessor to format the outputs — and your platform runs inference on their behalf. It reads great in the changelog. What it actually means, at the systems level, is that you have agreed to execute arbitrary code and deserialize arbitrary binary blobs, both authored by people you've never met, on machines that also hold other customers' data.

The two obvious ways to build it are both wrong. You can run every tenant's model in one long-lived Python worker process, which is fast and cheap and means any tenant's preprocessing code shares an address space with every other tenant's connection strings, cached rows, and model weights. Or you can give each tenant a container, which is better, until you remember that a container shares the host kernel — it's a polite suggestion to the kernel about what a process should see, not a wall the process can't climb over. For code and binaries you can't audit, neither is the boundary you want.

I'm Ajay; I built PandaStack. This post is about the boundary that actually holds: one Firecracker microVM per tenant (or per request), with the loaded-model state snapshotted so you don't pay a cold model load every time, and fork so you can absorb burst without re-warming. Let me walk through why the naive versions leak, then the model that doesn't.

Loading a model is running code, not reading a file

The belief that gets platforms breached is that a model file is data. Some formats are — safetensors was designed precisely so loading weights is not code execution. But the formats people actually upload, over and over, are not safe. A PyTorch checkpoint saved with `torch.save` is a Python pickle, and unpickling is Turing-complete by design: a crafted pickle can invoke `os.system`, import anything, and run whatever it likes the instant you call `torch.load`. "Just load the model" is a remote code execution primitive dressed as a file read.

• Pickle is RCE by design — a malicious `.pt`/`.pth`/`.ckpt`/`.pkl` runs arbitrary code during deserialization via `__reduce__`. `torch.load(untrusted)` is not safe, and `weights_only=True` helps but is not a full sandbox for every format a real platform accepts.
• The preprocessing code is arbitrary too — a tenant's "reshape the tensor" script is a Python module you import and call. It runs with whatever the worker process can reach: env vars, mounted secrets, the network, the filesystem.
• Model formats drag in parsers — GGUF, ONNX, HDF5, and friends are binary formats with their own parsing surface; a malformed file can trip a native bug in the loader.
• Weights are big and hostile to memory limits — a tenant can upload a model sized to OOM the host, or a preprocessor that allocates until the kernel's OOM killer starts picking neighbors. That's a denial-of-service, no exploit required.
• The blast radius is every tenant on that worker — a shared inference process is a credential-rich, data-rich room. One tenant's `__reduce__` reads another tenant's cached rows and posts them out.

The model: one microVM per tenant

The structural fix is to stop running untrusted model code inside your trusted process at all. Move the whole load-and-infer step into a throwaway Firecracker microVM. Your control plane — the part with the database credentials and the request router — stays on a trusted host and drives disposable sandboxes over an API. The weights are written into the guest, the pickle is unpickled inside the guest, the preprocessor and the model run inside the guest, and the result comes back over the wire. When the request (or the tenant's session) is done, you destroy the VM and the entire blast radius goes with it.

The reason this is practical and not just correct is latency. A cold Firecracker boot is a couple of seconds, but PandaStack creates sandboxes by restoring a baked snapshot on demand — the restore step is around 49ms, and end-to-end create is p50 179ms (p99 ~203ms). So a fresh, hardware-isolated machine per tenant costs about a fifth of a second, not a fifth of an hour. And each agent has 16,384 pre-allocated network slots, so fanning out across many tenants at once isn't a slot-capacity problem — memory and CPU are the real ceiling.

Here's the per-request shape with the Python SDK. The untrusted weights are written into the guest, loaded there, and inference runs there; your host never unpickles a stranger's checkpoint.

from pandastack import Sandbox

def infer_for_tenant(tenant_id: str, weights: bytes, payload: dict) -> dict:
    # A fresh, hardware-isolated guest for THIS request. ~179ms p50 to create.
    sbx = Sandbox.create(
        template="code-interpreter",
        ttl_seconds=120,            # hard cap: a runaway preprocessor can't run forever
        metadata={"tenant": tenant_id},
    )
    try:
        # Push the tenant's untrusted weights INTO the guest. Never load them
        # on the host process -- torch.load() on a pickle is arbitrary code.
        sbx.filesystem.write("/work/model.pt", weights)
        sbx.filesystem.write("/work/input.json", json_dumps(payload))

        # Load + run inside the guest. If this checkpoint's __reduce__ fires an
        # os.system(), it fires in a disposable VM holding no other tenant.
        result = sbx.exec(
            "cd /work && python3 run_inference.py model.pt input.json",
            timeout_seconds=90,
        )
        return {"ok": result.exit_code == 0, "output": result.stdout}
    finally:
        sbx.delete()   # blast radius destroyed with the VM

Snapshot the warm model so cold loads aren't paid twice

Per-request isolation has one honest cost: loading a multi-gigabyte model is slow, and doing it on every request is wasteful even if it's safe. This is exactly what snapshots are for. For a tenant whose model doesn't change between requests, do the expensive load once, snapshot the guest with the model resident, and then serve subsequent requests by restoring that snapshot — the model is already loaded in the restored memory image.

The trust rule matters here: the snapshot you keep and re-restore must contain only that tenant's own model and code, never a shared baseline that mixes tenants. A per-tenant warm snapshot is a per-tenant asset. You get the cold-load cost amortized without collapsing the isolation you built the whole thing for.

# --- Warm ONCE per tenant: load the model, then snapshot that state. ---
warm = Sandbox.create(template="code-interpreter", ttl_seconds=1800)
warm.filesystem.write("/work/model.pt", tenant_weights)
warm.exec("cd /work && python3 load_model.py model.pt", timeout_seconds=600)
snap = warm.snapshot()          # capture the guest with the model resident
warm.hibernate()

# --- Serve requests by restoring the warm snapshot (model already loaded). ---
# Burst? Fork the warm snapshot per concurrent request: ~400-750ms same-host,
# copy-on-write, so N forks don't cost N copies of the weights until they write.
def serve(payload: dict) -> dict:
    with snap.fork(ttl_seconds=90) as sbx:
        sbx.filesystem.write("/work/input.json", json_dumps(payload))
        out = sbx.exec("cd /work && python3 infer.py input.json",
                       timeout_seconds=60)
        return {"output": out.stdout}

Fork is what makes burst affordable. When a tenant gets a spike, you don't re-load their model N times — you fork the warm snapshot per concurrent request. A same-host fork lands in roughly 400–750ms and shares the warm image copy-on-write, so ten concurrent inferences share the weights' memory pages until each diverges. (A cross-host fork is 1.2–3.5s when the warm snapshot lives on another agent.)

Noisy neighbors and resource limits

Isolation isn't only about secrets; it's about capacity. In a shared worker, one tenant's inference — a huge batch, a pathological input, a preprocessor that spins — steals CPU and memory from everyone else. A microVM has a fixed vCPU and RAM budget baked into its snapshot, so a tenant's runaway job saturates its own guest and no one else's. Pair that with a `ttl_seconds` cap and a runaway job is a guest that gets reclaimed, not an incident. The decompression-bomb equivalent for models — a checkpoint sized to exhaust RAM — hits the guest's own OOM boundary and dies there.

• Isolation strength — Shared Python worker: none; every tenant's preprocessing and unpickled weights share one address space with everyone's data. Container per tenant: shares the host kernel, so an escape or a kernel bug reaches the host and neighbors. MicroVM per tenant: hardware-virtualized guest with its own kernel — a compromised load stays in one disposable VM.
• Cold model load — Shared worker: paid once but the process is permanent and permanently exposed. Container: paid per container spin-up. MicroVM: warm the model once, snapshot it, restore/fork the warm image (~400-750ms fork) so the load is amortized without a persistent shared process.
• Noisy neighbor — Shared worker: one tenant's batch or spin starves all. Container: cgroup limits help but the kernel is shared. MicroVM: fixed vCPU/RAM per guest plus ttl_seconds; a runaway saturates only itself and is reclaimed.
• Cleanup — Shared worker: state and leaked buffers persist across tenants forever. Container: docker rm, if you remember and if nothing escaped. MicroVM: the guest is destroyed on delete/ttl — there's no next request for a compromise to persist into.

None of this makes inference slower in the way that matters. The model still runs at model speed; what changes is where it runs and who it can reach. The tenant's untrusted weights and code execute in a disposable guest holding none of your other customers' data, your router reasons over the results on a trusted host, and at ~179ms per create plus ~400ms per warm fork, the isolation is close to free.

For the adjacent problems: fine-tuning jobs (which run untrusted training code, not just inference) get their own treatment in /blog/microvm-per-tenant-llm-finetuning-jobs, the snapshot-and-fork mechanics are in /blog/snapshot-and-fork-explained, and the general untrusted-code model is in /blog/how-to-sandbox-untrusted-code.