Pickle Is a Virtual Machine You Have Been Downloading
I build PandaStack, an open-source Firecracker microVM platform, and the pickle conversation reliably starts in the wrong place. Someone has a product that accepts model artifacts, or a worker fleet, or an internal registry, and they want to discuss whether a particular file is malicious. The file is not the interesting question. The interesting question is that they have been running a bytecode interpreter over bytes from strangers and calling it loading.
`pickle` is not a serialisation format in the sense JSON is. It is a stack machine with a documented instruction set, and `pickle.load` is an interpreter loop over that instruction set. One of its instructions imports a name from a module; another calls whatever is on the stack with arguments from the stack. Not as an exploit — as the mechanism by which Python objects are reconstructed. So the canonical four-line payload everyone has seen is not a vulnerability demonstration. It is a conformance test.
What makes this a 2026 problem rather than a 2011 blog post is that the machine-learning ecosystem took the format and shipped it to everybody. A PyTorch `.pt` is a zip archive with a pickle inside it. A scikit-learn model distributed as a `.pkl` is a pickle. A NumPy `.npy` holding an object array is a pickle. A Hugging Face repo full of `.bin` files is a directory of pickles. An enormous number of people have downloaded a program from the internet and run it as a favour to a stranger, under the impression that they were downloading numbers.
The unpickler is an interpreter loop
The standard library is blunter about this than any security advisory. The source comments in `pickletools` open with "A pickle is a program for a virtual pickle machine", then describe the machine's two data areas — the stack, and the memo of objects already seen so a cyclic graph can reference itself. Opcodes are executed once each, first to last, until a `STOP` is reached, and whatever sits on top of the stack is handed back to you as your data. The module also ships a disassembler, which is not a tool you build for a data format.
Most opcodes are boring. `EMPTY_DICT` pushes a dict, `BININT1` pushes a small integer, `SETITEM` pops a key and value into the dict below it. A pickle of a configuration dictionary contains nothing else, and it is as safe as the equivalent JSON, because nothing in it names anything. Then there are the ones that do.
The opcodes that are not boring
- `GLOBAL` — protocol 0 to 3, the single byte `c`. Takes a module name and a qualified name as two newline-terminated strings and resolves them: import the module, walk the attribute path. One opcode, one import, one `getattr`.
- `STACK_GLOBAL` — protocol 4 and later. The same thing with both names popped off the stack instead of read inline, so they can be memoised and dotted names work properly. Functionally identical; worth knowing because a rule matching the textual `GLOBAL` form will not see it.
- `REDUCE` — `R`. Pops an argument tuple, pops a callable, calls the callable with the tuple, pushes the result. That is the entire opcode, and it is how every object whose state cannot be expressed as a plain dict gets rebuilt.
- `BUILD` — `b`. Pops a state object and applies it to the object underneath: `__setstate__(state)` if the class defines one, otherwise `__dict__.update(state)`. So a class with a `__setstate__` is handed a dictionary chosen by whoever wrote the file. This one gets a fraction of the attention `REDUCE` gets and deserves equal billing.
- `INST` (`i`, protocol 0) and `OBJ` (`o`, protocol 1) — the old combined forms that resolve a global and instantiate it in one step. They still load. Legacy opcodes are not a legacy problem; they are a current parsing surface with a lower profile in everybody's rule sets.
- `NEWOBJ` and `NEWOBJ_EX` — call `cls.__new__(cls, ...)`, the modern creation path `__reduce_ex__` emits for most objects, usually followed straight by a `BUILD`. And `EXT1`/`EXT2`/`EXT4` resolve an integer against the `copyreg` extension registry rather than naming a module at all — nothing is registered there by default, which makes them a reminder that opcode-to-callable is not always a string you can grep for.
# Real `pickletools.dis` output, CPython 3.14. Lines starting with `#` are my
# annotations; every other line is verbatim. No input here is malformed.
>>> import os, pickle, pickletools
>>> pickletools.dis(pickle.dumps({"epochs": 3}, protocol=4))
0: \x80 PROTO 4
2: \x95 FRAME 15
11: } EMPTY_DICT
12: \x94 MEMOIZE (as 0)
13: \x8c SHORT_BINUNICODE 'epochs'
21: \x94 MEMOIZE (as 1)
22: K BININT1 3
24: s SETITEM
25: . STOP
highest protocol among opcodes = 4
# Pure data: not one opcode names anything. find_class is never reached, so a
# restricted unpickler would not be consulted a single time on this stream.
>>> class Evil:
... def __reduce__(self): # the documented hook
... return (os.system, ("id > /tmp/pwned",))
...
>>> pickletools.dis(pickle.dumps(Evil(), protocol=4))
0: \x80 PROTO 4
2: \x95 FRAME 42
11: \x8c SHORT_BINUNICODE 'posix' # module name -> stack
18: \x94 MEMOIZE (as 0)
19: \x8c SHORT_BINUNICODE 'system' # attribute name -> stack
27: \x94 MEMOIZE (as 1)
28: \x93 STACK_GLOBAL # import posix; getattr(_, 'system')
29: \x94 MEMOIZE (as 2)
30: \x8c SHORT_BINUNICODE 'id > /tmp/pwned'
47: \x94 MEMOIZE (as 3)
48: \x85 TUPLE1 # the argument tuple
49: \x94 MEMOIZE (as 4)
50: R REDUCE # pop args, pop callable, CALL it
51: \x94 MEMOIZE (as 5)
52: . STOP # 53 bytes, start to finish
highest protocol among opcodes = 4
# The module is `posix`, not `os`: os.system.__module__ == 'posix' on Linux and
# macOS, so a denylist matching the string "os" does not match this. Dump the
# same object at protocol 2 and the three opcodes at offsets 11-28 collapse
# into one -- `c GLOBAL 'posix system'`, the name inline as text. The author
# picks the protocol, and the protocol picks what a scanner has to match.
Fifty-three bytes. Nothing in that disassembly is malformed and the unpickler did nothing but what the specification tells it to. The format did not fail. It worked.
__reduce__ is the extension point, which is why nobody can fix it
`__reduce__` returns a callable and an argument tuple, and the pickler writes that pair into the stream as a global reference followed by a `REDUCE`. This is the documented way an object declares how to reconstruct itself, used constantly by code that has never once thought about security: anything holding a file handle, a socket, a lock, a compiled regex, a C extension's opaque handle. `copyreg` exists so you can register reconstructors for types you do not own. `datetime` uses this machinery. NumPy uses it. Your ORM uses it.
So "make pickle safe" means "remove the mechanism every non-trivial Python object uses to serialise itself." That is not a patch, it is a different format — which is precisely what `safetensors` is, and precisely why it only covers tensors. The bug class has outlived two decades of secure-default work because the feature is load-bearing, and the standard library's response has always been the honest one: a warning at the top of the page telling you not to unpickle data you do not trust.
The ecosystem shipped it to everybody
Deserialisation code execution used to be a backend specialty that showed up in Java and PHP advisories. Then machine learning made artifact exchange a mass-market activity, and the artifact format was pickle. Where it hides:
- `.pt`, `.pth`, `.ckpt` — a PyTorch checkpoint is a zip archive containing a pickle plus the tensor storages it references. Lightning checkpoints add hyperparameters and callback state, a considerably larger object graph to reconstruct.
- `.bin` — the pre-safetensors Hugging Face convention, and still what a great many repositories serve. A repo of these is a directory of pickles.
- `.pkl` and joblib artifacts — the entire classical-ML world. A fitted scikit-learn pipeline ships as a pickled object graph; joblib is pickle with compression bolted on. There is no restricted mode to fall back to here, which people miss because PyTorch got one.
- `.npy` and `.npz` — only when they hold object arrays, and only when you pass `allow_pickle=True`. The default has been `False` since NumPy 1.16.3, a change its release notes attribute directly to CVE-2019-6446. `allow_pickle=True` is the flag equivalent of holding a door open for someone you have not looked at.
- Anything a framework calls a cache. Feature stores, tokeniser caches, memoisation files on disk, task results in queue backends. Pickle is fast and convenient, which is why it keeps winning these decisions on merit.
weights_only=True is the real fix, and it is now the default
`torch.load(..., weights_only=True)` swaps the general unpickler for a restricted one that refuses arbitrary imports and reconstructs only tensors, primitive types, dicts, and globals you explicitly allowlist through `torch.serialization.add_safe_globals`. PyTorch flipped that default from `False` to `True` in 2.6, having warned about the change from 2.4 onward; the current stable signature shows `weights_only=True`. This was the highest-leverage change anybody has made in this area and it got less credit than it deserved.
A default moving is not the same as a problem going away, though. A plain tensor state dict loads; a serialised `nn.Module` does not, and the documented workaround is `weights_only=False`, the dangerous path under a different name — so every artifact that predates the flip pushes somebody straight back to it. Tensor subclasses and serialised NumPy arrays need explicit allowlisting. And the installed base of vendored loading code passing `weights_only=False` to make a 2023 checkpoint work is still out there, in notebooks, in Dockerfiles, in a `utils/load.py` nobody has opened since its author left. The other limit is scope: `weights_only` is a PyTorch feature. It does nothing for joblib, for a scikit-learn pipeline, or for a bare `pickle.load` in somebody's feature-engineering step.
safetensors is the right answer where it applies
safetensors fixes the format instead of the loader: a JSON header describing names, dtypes, shapes and byte offsets, followed by a flat block of tensor data, memory-mappable, with no construct in the grammar for expressing "call this function". There is nothing to allowlist because there is nothing to call. If you can require it, require it, and prefer the safetensors variant of any repository publishing both.
It is not a universal answer, and pretending otherwise is how a team ends up with a policy instead of a plan. safetensors stores tensors. It does not store a fitted preprocessing object, an optimiser state with arbitrary objects in it, or the general Python object graph that a large share of circulating artifacts actually are. Announce "safetensors only" and the first legitimate `.ckpt` arrives within the week, attached to a ticket from a customer who is not wrong.
The queue is the vulnerability; the pickle is the calling convention
The model-weights story is the one that gets written about. The operationally nastier version is pickle as a wire protocol, because there the payload does not have to be hostile for you to be in trouble. The exposure is the ability to write a message; the pickle is merely how that write becomes a function call in every process reading the queue. Check all four of these against your own versions before acting on anything — defaults have moved, mostly in the right direction, and not uniformly.
- Celery. The default serializer has been JSON since Celery 4.0, and the project's security documentation says plainly that pickle is inherently insecure and should be avoided whenever clients are untrusted or unauthenticated. The remaining exposure is configuration drift: a `task_serializer` set in 2016 and never revisited, or an `accept_content` that still lists pickle for compatibility. Celery's own docs note that by default workers trust that data from the broker has not been tampered with, and offer an `auth` serializer that signs messages with a private key — signing, not encryption.
- Ray. Serialisation is a customised pickle protocol 5 plus cloudpickle, by design, because moving Python callables around a cluster is the product. There is no serializer to change. What you control is who can reach the cluster's ports, and Ray does not ship authentication enabled by default — verify that against their current security documentation rather than my summary.
- Dask. Administrative messages always go over msgpack; data movement tries Dask's own serialisers and falls back to pickle and cloudpickle, and the task serialisation scheme is fixed rather than configurable. You can pass `deserializers=` on a `Client` to exclude pickle for data, which is worth knowing and does not cover tasks.
- `multiprocessing`. The standard library's own warning is the clearest sentence in this post: `Connection.recv()` automatically unpickles the data it receives, which can be a security risk unless you can trust the process which sent the message. Hence authentication keys on `Listener`, `Client` and `BaseManager` — a shared secret, because the alternative is accepting opcodes from anyone who can open a socket.
The shape is identical in all four. Something that can write a message becomes something that can call a function in every worker reading that queue. A Redis instance with no password on a network you believe is private. A RabbitMQ user shared across six services because rotating it was going to be a project. A Dask scheduler port exposed because somebody wanted the dashboard. The compromise chain is not "found a pickle bug" — it is "found a queue" — and the blast radius is the worker fleet, which usually holds more credentials than anything else you own.
Which is why the fix here is not a sandbox. If your wire protocol is pickle, the fix is the protocol: JSON or msgpack, an `accept_content` that excludes pickle, signed messages, an authenticated broker. A microVM per worker is a fine thing to have for other reasons — noisy neighbours, poison jobs, per-tenant blast radius, the argument at Per-Tenant Queue Consumers in Isolated microVMs — but it does nothing about "anyone who can write to the queue can run code," because the code runs with exactly the privileges the worker legitimately needs.
Why the in-process mitigations are partial
The restricted unpickler is correct, and a smaller promise than it looks
`pickle.loads` offers no hooks whatsoever. `pickle.Unpickler` does: subclass it, override `find_class(module, name)`, and you decide what the stream may import. This is the mitigation the Python documentation itself recommends, it works against the C implementation, and you should ship it anywhere you must accept a pickle you did not write.
# A restricted unpickler: the CORRECT in-process mitigation. Ship it. The
# docstring is where "it is not a sandbox" stops being a slogan.
import io
import pickle
# Every (module, name) pair the stream may IMPORT. find_class is the only hook
# pickle gives you, and it is reached ONLY by an opcode that names something:
# GLOBAL, STACK_GLOBAL, INST, OBJ, or the copyreg EXT family.
ALLOWED = frozenset(
{
("numpy.core.multiarray", "_reconstruct"),
("numpy", "ndarray"),
("numpy", "dtype"),
("collections", "OrderedDict"),
}
)
class Restricted(pickle.Unpickler):
"""Refuses to import any name outside ALLOWED.
PROMISES: no opcode in the stream can resolve a name outside the set above.
posix.system, subprocess.Popen, builtins.eval, builtins.getattr -- refused,
loudly, before anything is called. Real property; kills the copy-pasted
payload class outright.
DOES NOT PROMISE: that calling an ALLOWED name with attacker-chosen
arguments is safe. REDUCE calls your allowlisted callable with a tuple off
the stream; BUILD hands your allowlisted class's __setstate__ a dict off the
stream. So the allowlist asserts every name in it behaves safely under
hostile arguments -- far stronger than "these look harmless". One entry
whose __setstate__ writes to a path out of its own state reopens the door,
and the stream walking through looks completely ordinary.
"""
def find_class(self, module, name):
if (module, name) not in ALLOWED:
# A 400 is what the client gets; this is what you alert on.
raise pickle.UnpicklingError(f"refused import: {module}.{name}")
return super().find_class(module, name)
def persistent_load(self, pid):
# Belt and braces: the base class already raises on a PERSID opcode
# with no handler. Refusing it here makes the rejection yours.
raise pickle.UnpicklingError("persistent ids not accepted")
def loads_restricted(raw: bytes):
if len(raw) > 64 * 1024 * 1024:
raise pickle.UnpicklingError("larger than the deserialize budget")
# pickle.loads() takes no hooks at all. You must go through Unpickler.
return Restricted(io.BytesIO(raw)).load()
Two properties of that code deserve saying out loud. First, `find_class` is reached only when the stream names something — a pure-data pickle calls it zero times, which I verified by printing from an override rather than assuming. So the allowlist is exercised exclusively by streams trying to do something, which is an excellent property for monitoring: every rejection is an interesting event rather than noise, and a non-zero rejection rate on a path that should only ever see plain tensors is a page, not a dashboard.
Second, the part that gets glossed over. An allowlist is a claim that every name in it is safe to call with arguments an attacker chose — far stronger than "these classes look harmless." `REDUCE` calls your allowlisted callable with a tuple from the stream. `BUILD` hands your allowlisted class's `__setstate__` a dictionary from the stream. If one entry has a `__setstate__` that takes a path out of its own state and writes to it, or a constructor that opens something, or a reconstructor that passes a dtype string straight into a C parser, the door is open again — and the stream walking through it looks entirely ordinary, because it is using the allowlist as designed. PyTorch's `add_safe_globals` is the same bargain made explicit. Which is fine; just make sure the review question is "what does this do with attacker-controlled input," not "do I recognise this class."
Static scanners are useful and bypassable, and both halves matter
picklescan, modelscan and fickling all disassemble the opcode stream without executing it and flag dangerous imports, and the model hubs run scanning of their own. Use them. They are cheap, they catch the overwhelming majority of payloads in circulation — which are copied from a blog post and call `os.system` — and fickling will analyse and rewrite pickles rather than only pattern-match them.
They are also reading an opcode stream the attacker shapes, and the evasion surface is structural rather than incidental. The protocol decides which opcode performs the import. `GLOBAL` puts the name in the stream as inline text; `STACK_GLOBAL` puts it there as two string pushes. The legacy `INST` and `OBJ` forms are less commonly modelled. `posix.system` is not the string `os.system`. The extension opcodes name an integer rather than a module. And a dangerous callable can be reached through a chain of innocuous attribute lookups instead of imported. Every public scanner has had bypasses published against it; maintainers fix them, and the next one is a weekend's work for somebody motivated.
Underneath that, a tool forbidden from executing the file must decide what a name means statically, in a language where a name means whatever it was last bound to. Not a solvable problem — a tractable one. A clean scan is a filter that removed the lazy attacks, not a proof, and the gap between those two words is what the rest of this post is about.
The microVM shape: unpickle somewhere disposable, hand back something inert
If the load is genuinely code execution, the only control that changes the outcome is where the code runs. So run it in a guest that exists for one artifact and is deleted afterwards. What that buys, specifically:
- Its own guest kernel. The unpickler's syscalls go to a kernel whose only job is this one file and which will not exist in a moment. A container shares your host kernel with every other tenant, so namespaces and cgroups are enforcement by the same kernel the payload is talking to — a polite suggestion, made to the thing you are trying to protect.
- A small device model, and unconditional teardown. Firecracker exposes a handful of virtio devices rather than four decades of emulated hardware, and the VM is deleted rather than reset and returned to a pool with its temp directories swept — so nothing that ran in it survives, including whatever you would have forgotten to clean up.
- Attribution, and no credential in reach. One artifact, one sandbox id, one log stream: when something in there calls `socket.socket`, you know which upload it was, because that guest handled exactly one. And the process doing the dangerous load holds no cloud role, no database password, no internal service token — not a narrowly scoped one, none. That control costs nothing and is skipped most often.
And the value that crosses back must be inert. This is where people lose the win they just paid for: they convert inside a sandbox and then hand back a pickle of the converted object, so the trusted side unpickles attacker-influenced bytes and the whole exercise was theatre with extra steps. Inert means a short, checkable list. A `safetensors` file. A `.npy` written with no object dtype and read back with `allow_pickle=False`. A JSON document you validate against a schema you own. Not a pickle. Not a Python module the guest generated. Not a `.pth` your loader will open with `weights_only=False` because the guest's report said it was fine.
The conversion gateway, which is the actual advice
Here is the part I care about more than the sandbox mechanics, because it is where the cost argument lives: you do not sandbox every load forever. You sandbox the conversion, once, at ingest.
The artifact arrives. One microVM opens it, performs the single dangerous deserialisation, re-emits the contents in a format with no call mechanism, writes a metadata report, and dies. You store the converted artifact under a digest you computed yourself. From then on, production loads the safe format with no sandbox, no scanner, no added latency, and no `weights_only` argument to get wrong, because there is no unpickler anywhere in the path.
The asymmetry is the point. An artifact is deserialised once and loaded thousands of times, so a sandbox on the load path taxes every inference while a sandbox at the gateway taxes each artifact once. It also fails in the better direction: if the conversion sandbox is unavailable, new artifacts stop being admitted, which is a queue and a dashboard; if a sandbox on your inference path is unavailable, you are down. The properties I would hold the design to:
- Bytes in, never a URL. The guest receives a file. It does not receive a reference the trusted side will dereference later, and it does not get to tell you where to fetch the next thing.
- The guest holds nothing. No credentials in the environment, no mounted secret, no instance role, no broker password. If the conversion must write a multi-gigabyte output somewhere, read the resource section below — that is the one place this gets genuinely uncomfortable and I am not going to route around it.
- The conversion script writes and exits. It does not negotiate. It does not import a module the artifact shipped to describe itself, it does not evaluate a config the artifact carried, and it has no retry loop that could be steered. It runs in the same guest as the hostile code, so it has to be a short program whose only output is files.
- The wall clock lives in the guest, and it is the only one. `timeout` for wall time, `ulimit -t` for CPU seconds, so the normal failure is a non-zero exit with usable stderr rather than a VM that vanished mid-write. Do not mistake `ttl_seconds` for a second clock: the reaper measures it from the sandbox's last activity, so a conversion grinding through twenty gigabytes is never idle and never reaped. The TTL catches a guest you forgot about — a different and also real failure.
- The output is validated by the trusted side against a schema it owns, and its size is capped before anything parses it. A four-hundred-gigabyte `report.json` is a denial of service you paid for in advance.
- One artifact, one sandbox, no batching inside a guest. The second file in a reused VM inherits the first file's consequences, and the point was that there are none.
The latency objection to a VM per artifact used to be decisive, and snapshot restore removed it. On PandaStack there is no warm pool of idle VMs: every create restores a pre-baked snapshot, landing at p50 179 ms and p99 203 ms end to end, with the restore step itself around 49 ms. Only the first-ever spawn of a template pays a cold boot of roughly three seconds, and the agent bakes the snapshot out of it. Measured against a conversion that reads several gigabytes off disk, 179 ms is not a number anyone will bring to a meeting.
import hashlib
import json
from pandastack import Sandbox
from pandastack.exceptions import CommandFailed
# Runs INSIDE the throwaway guest -- the one place where weights_only=False is
# defensible: the code is going to run, and the kernel it runs on is about to
# be deleted. Writes and exits: no retry loop, nothing the artifact can steer.
CONVERT_PY = r"""
import json, pathlib, sys
import torch
from safetensors.torch import save_file
src, out_dir = pathlib.Path(sys.argv[1]), pathlib.Path(sys.argv[2])
obj = torch.load(src, map_location="cpu", weights_only=False) # the loud part
state = obj.get("state_dict", obj) if isinstance(obj, dict) else obj.state_dict()
# Non-tensors do NOT travel. Everything that does is described, not embedded.
keep = {str(k): v.contiguous() for k, v in state.items() if torch.is_tensor(v)}
save_file(keep, out_dir / "model.safetensors") # no call mechanism, by design
(out_dir / "report.json").write_text(json.dumps(
{k: [list(v.shape), str(v.dtype)] for k, v in keep.items()}))
"""
# The real bound is in the guest: `timeout` is wall clock, `ulimit -t` CPU
# seconds. ttl_seconds is NOT a second wall clock -- the reaper measures it from
# last ACTIVITY, so a grinding conversion is never idle and never reaped.
FENCE = (
"cd /work && umask 077 && mkdir -p out && "
"ulimit -t 900; ulimit -f 33554432; ulimit -c 0; "
"exec timeout --signal=TERM --kill-after=10s 600 "
"python3 -I convert.py in/artifact.bin out"
)
def convert_one(artifact: bytes, artifact_id: str) -> tuple[bytes, dict]:
"""One artifact, one microVM, one audit record. Never reused: the second
file in a recycled guest inherits the first file's consequences."""
sbx = Sandbox.create(
# Custom template: torch + safetensors baked in, at the RAM the job
# needs. build_from_dockerfile() takes cpu/memory_mb/size_mb, and
# Firecracker cannot change either at snapshot restore -- memory_mb=
# on create() would be silently overridden here.
template="pickle-gateway",
ttl_seconds=1800, # IDLE ttl: reaps a forgotten guest, not a busy one
metadata={"job": "pickle-convert", "artifact": artifact_id},
)
try:
sbx.filesystem.write("/work/convert.py", CONVERT_PY.encode())
sbx.filesystem.write("/work/in/artifact.bin", artifact) # bytes, not a URL
logs: list[str] = []
# exec_stream, not exec: streaming widens the client HTTP timeout so a long conversion is not cut off at 30s. The in-guest fence is the real bound.
code = sbx.exec_stream(FENCE, on_stderr=logs.append, timeout_seconds=900)
if code != 0:
# 124 = timeout fired. 137 = the guest OOMed, which is nobody
# else's problem, which was the entire point.
raise CommandFailed(f"{artifact_id}: convert exited {code}",
exit_code=code, stdout="", stderr="".join(logs)[-4000:])
report = json.loads(sbx.filesystem.read("/work/out/report.json"))
safe = sbx.filesystem.read("/work/out/model.safetensors")
# The guest is a possibly compromised witness: validate against YOUR
# schema, store under YOUR digest.
return safe, {"shapes": validate(report),
"sha256": hashlib.sha256(safe).hexdigest()}
finally:
sbx.kill() # unconditional. A leaked sandbox bills by the GiB-hour,
# patiently, for as long as you fail to notice it.
Resource reality, which is the problem you will actually hit
In practice a multi-gigabyte artifact is a harder engineering problem than a hostile one, and the constraint is specific enough to state exactly. Firecracker cannot change vCPU or RAM at snapshot restore, so a sandbox's size is whatever was baked into its template's snapshot — which means `cpu=` and `memory_mb=` on `create()` are overridden for any template that has one. The first-party templates are `base` at 4 GiB and 8 vCPU, `code-interpreter` and `agent` at 2 GiB, `browser` at 4 GiB, `postgres-16` at 1 GiB, the 8 vCPU being burst capacity shared under contention by cgroup weight. Asking for 24 GiB at create time does not fail loudly. It quietly gives you the baked size, and the conversion dies at an out-of-memory you did not predict.
So choose the size at bake time instead. The SDK's `templates.build_from_dockerfile()` takes `cpu`, `memory_mb` and `size_mb`, and its docstring is explicit that those are baked into the resulting template snapshot. A conversion gateway is therefore a custom template: your Dockerfile with torch, safetensors, numpy and your scanner of choice already installed, baked once at the memory the job needs, restored on every create. That also takes a `pip install` off the ingest path, which is the other place untrusted-adjacent code execution likes to hide — see Safely running pip install from LLM-generated code. The first-party set is on templates.
Even with the right baked size, a 20 GB checkpoint is not a sizing argument. It is a streaming problem, and the shapes that work are boring:
- Convert tensor by tensor. Both the PyTorch zip container and safetensors are built for it: read one storage, write one tensor, drop the reference. Peak memory becomes the largest single tensor plus overhead rather than the whole checkpoint, which is usually an order of magnitude of headroom for free.
- Size the disk, not only the RAM. `size_mb` on the template build is the rootfs, and a conversion needs room for input and output simultaneously, plus headroom. An ENOSPC partway through a write is the worst failure mode in this post, because it produces a truncated output that looks exactly like a file.
- Do not read a multi-gigabyte result back through the SDK. `filesystem.read()` returns `bytes`, so your client buffers the entire artifact in memory. Fine for a report and a small model; not a plan for a checkpoint.
- Which is where the no-credentials rule collides with reality, and I would rather name the collision than route around it. The conversion script runs in the same guest as the hostile code, so any upload credential in that guest is a credential the payload can use. The best available answer is one useless to an attacker: write-only, scoped to the single object key you generated, valid for minutes, no list, no read, no delete — a pre-signed single-object PUT. Then verify the digest on your side before trusting a byte of it. A real reduction in exposure, not a boundary; and if the artifacts fit back through the control plane, do that instead and skip the problem.
Three places to put the dangerous load
| Dimension | Plain pickle.load in-process | Restricted Unpickler in-process | MicroVM conversion gateway |
|---|---|---|---|
| What stops a REDUCE payload | Nothing. The opcode is doing its documented job. | The import is refused before the call, if the name is outside your allowlist. | Nothing stops the call. The guest it lands in is deleted afterwards. |
| Where a phone-home originates | Your service: its DNS, its routing, its identity. | Nowhere if the allowlist holds, everywhere if it does not. | A per-sandbox netns. Metadata and sibling sandboxes fenced; the internet is not. |
| Credentials in reach | Everything the process holds, which is everything. | The same, on a bypass. | Nothing by construction, except an upload credential if you need one. |
| Cost per production load | Zero, the honest reason most teams stop here. | Near zero, plus the allowlist review you owe forever. | Zero. The sandbox ran once at ingest; production loads safetensors. |
| Diagnosing it at 3am | A stack trace from a process that did forty other things. | A clean refusal naming a module and a name, which is genuinely good. | One artifact, one sandbox id, one log stream. |
The honest limits
- The conversion script shares a guest with the payload. No clever arrangement fixes this: the code doing the load and the code being loaded are in the same address space by definition. So the conversion must be short, must write and exit, and must never take an instruction from the artifact. Everything the guest tells you afterwards is a claim from a possibly compromised witness — validate the output yourself, do not believe the report.
- A converted artifact can still be a bad model. safetensors removes code execution, not malice. Weights are not code, but weights can be backdoored, and a model that scores well on your evals and misbehaves on a trigger is an active research area this post does not touch. "Not remote code execution" and "safe to deploy" are different claims; conflating them is how a conversion gateway becomes a rubber stamp.
- Egress is open by default. The host's FORWARD chain drops sandbox-to-sandbox traffic, so no guest reaches another guest's subnet, and drops the whole of `169.254.0.0/16`, so the cloud metadata endpoint is unreachable — which closes the most-cited deserialisation-to-credential-theft path before your unpickler gets a vote. The general internet is still reachable, and so is your own VPC if the sandbox can route to it. A payload can phone home. Those deny rules are yours to add, as code that runs for every sandbox; the shape is at Controlling Network Egress for Untrusted Code.
- safetensors does not cover everything people pickle. A fitted scikit-learn pipeline is an object graph, not a tensor dict, so converting one to something inert means writing an explicit exporter for the estimators you support — coefficients, intercepts, tree arrays, preprocessing parameters as JSON or plain arrays — and refusing the rest. Real work, the right work, not a library call.
- If your wire protocol is pickle, the fix is the protocol. Repeating it because it is the commonest way this advice gets misapplied: wrapping each worker in a microVM does not stop queue-write access from becoming code execution. Change the serializer, set `accept_content`, authenticate the broker. The sandbox is for the artifacts, not for the calling convention.
- You have acquired a scheduler. Per-artifact VMs mean capacity can run out, and the fleet admits against working-set memory rather than committed, so a create refused for capacity is a real 503 you now have to handle. Plus a 5.10 guest kernel to check your toolchain against, and a conversion template to rebuild every time torch moves.
The summary
`pickle` is a stack machine with a documented instruction set, and `GLOBAL` plus `REDUCE` exist to import a name and call it. Not a defect anybody can fix, because `__reduce__` is the extension point every non-trivial Python object relies on. Do all the cheap correct things: `weights_only=True`, which PyTorch made the default in 2.6; `safetensors` wherever the data is tensors; `allow_pickle=False`, which NumPy has defaulted to since 1.16.3; a restricted `Unpickler` where you must accept a pickle at all, understanding that its allowlist is a claim about attacker-chosen arguments rather than about familiar class names; a static scanner as a filter, not a proof. Then fix any wire protocol that is pickle, because that one is a protocol problem and no amount of sandboxing touches it.
Then build the gateway. One microVM per untrusted artifact, no credential in reach, a wall clock inside the guest, a conversion that writes and exits, an inert output you validate yourself, and an unconditional `kill()`. Store the inert thing and load that in production forever, with nothing in the path. You cannot make unpickling safe — it is an interpreter, and it is doing its job. You can make it happen once, somewhere you are willing to lose.
Frequently asked questions
Is unpickling untrusted data actually dangerous, or is that warning theoretical?
It is dangerous by design, and the design is documented. Pickle is a stack-based virtual machine, and two of its opcodes exist to import a name and call it: `GLOBAL` or `STACK_GLOBAL` resolves a module and attribute, and `REDUCE` pops an argument tuple plus a callable and invokes it. A payload that runs a shell command is fifty-three bytes of entirely well-formed pickle, and `pickletools.dis` will disassemble it without executing anything. There is no parser bug involved and no version where this is patched, because `__reduce__` is the documented way Python objects say how to reconstruct themselves — used by `datetime`, by NumPy, by `copyreg`, by anything holding a handle to something unpicklable. Removing the mechanism would break every library that pickles a non-trivial object, which is why the standard library's answer has always been a warning rather than a fix. One detail worth internalising: a denylist is a bad fit even before you consider evasion, because `os.system` serialises as module `posix`, not module `os`.
PyTorch made weights_only=True the default. Is the checkpoint problem solved?
It is the single biggest improvement anybody has shipped here, and it is not the end of the matter. The flip landed in PyTorch 2.6 after warnings from 2.4 onward, and the current stable signature shows `weights_only=True`. With it on, a plain tensor state dict loads and a `__reduce__` payload fails loudly instead of running. The gaps are real, though. A serialised `nn.Module` does not load under the restricted unpickler, and the documented workaround is `weights_only=False` — the dangerous path by another name — so artifacts predating the flip push people straight back to it. Tensor subclasses and serialised NumPy arrays need explicit allowlisting through `add_safe_globals`, which is the same bargain as any allowlist: you are asserting that each added name is safe when called with arguments an attacker chose. And `weights_only` is a PyTorch feature, so it does nothing for joblib, scikit-learn pipelines, `.npy` files loaded with `allow_pickle=True`, or a bare `pickle.load` in a feature pipeline. Pin your version, set the argument explicitly at every call site rather than relying on the default, and prefer safetensors when a publisher offers both.
Can a restricted Unpickler with a find_class allowlist replace a sandbox?
It can for a narrow schema you defined, and it cannot for arbitrary third-party artifacts. The mechanism is sound: subclass `pickle.Unpickler`, override `find_class(module, name)`, refuse anything outside your set, and no opcode in the stream can resolve a name you did not approve. Ship that. The limit is what the allowlist actually asserts. `REDUCE` calls your allowlisted callable with an argument tuple from the stream, and `BUILD` hands your allowlisted class's `__setstate__` a dictionary from the stream, so an allowlist is a claim that every entry behaves safely under attacker-chosen arguments — much stronger than "I recognise these classes." One entry whose `__setstate__` writes to a path taken from its own state, or whose constructor opens a resource, reopens the door, and the stream exercising it looks completely ordinary because it is using the allowlist as designed. For a message format you own, where the allowlist is four types you wrote, that review is tractable and the allowlist is a genuine boundary. For a PyTorch checkpoint from a hub, where the legitimate allowlist is dozens of framework internals you did not write, it is a mitigation, and the dangerous load belongs somewhere disposable.
Is my Celery, Ray, or Dask setup exposed just because pickle is involved?
Not automatically, and the question to ask is who can write to the queue rather than which serializer is configured. Celery has defaulted to JSON since 4.0 and its security docs call pickle inherently insecure, so the usual exposure there is configuration drift — a `task_serializer` set years ago, or an `accept_content` that still accepts pickle for compatibility — plus the documented fact that workers trust broker contents by default, which is what the optional signing serializer addresses. Ray serialises with a customised pickle protocol 5 plus cloudpickle because moving Python callables is the product, so there is no serializer to change; what you control is network reachability and authentication, and you should check their current security documentation rather than any blog's summary. Dask uses msgpack for administrative messages and pickle or cloudpickle for data, with a fixed task-serialisation scheme, and you can exclude pickle from a `Client`'s `deserializers` for data but not for tasks. In every case the compromise chain is the same: write access to the broker becomes code execution in every worker reading it. The fix is the protocol and the authentication in front of the broker. A microVM around each worker helps with noisy neighbours and poison jobs and does nothing about this, because the payload runs with exactly the privileges the worker needs anyway.
How do I convert a 20 GB checkpoint if a sandbox's RAM is baked into its template?
You pick the size at bake time and you stream the conversion. Firecracker cannot change vCPU or RAM at snapshot restore, so a sandbox's size comes from the template's baked snapshot and `memory_mb=` on `create()` is overridden for any template that has one — `base` is 4 GiB and 8 vCPU, `code-interpreter` and `agent` are 2 GiB, `browser` 4 GiB, `postgres-16` 1 GiB. Asking for more at create time gives you the baked size silently, and the conversion dies at an out-of-memory you did not plan for. The actual lever is the template: `templates.build_from_dockerfile()` takes `cpu`, `memory_mb` and `size_mb`, all baked into the resulting snapshot, so a gateway template carries torch, safetensors and your scanner pre-installed at the memory the job needs. Even then, a 20 GB artifact should not be loaded whole. Convert tensor by tensor — read one storage, write one tensor, drop the reference — so peak memory is the largest single tensor rather than the checkpoint, which both the PyTorch zip container and safetensors are built to support. Size the rootfs for input and output simultaneously, because a half-written output file looks like a file. And do not read the result back through `filesystem.read()`, which buffers the whole thing in your client's memory; have the guest PUT it to object storage with a write-only, single-key, short-lived credential, then verify the digest on your side.
Keep reading
- A checkpoint is a program — The model-ingest version of this argument, from the artifact's side rather than the format's.
- Per-tenant queue consumers in microVMs — What a worker sandbox does buy you, which is not protection from a writable broker.
- Controlling network egress for untrusted code — The deny rules you add yourself, because sandbox egress is open by default.
- Parsing untrusted XML — The other format that keeps an interpreter in its specification, with the same flag-is-not-a-boundary problem.
- Safely running pip install from LLM-generated code — Install-time code execution, which is how a conversion template gets compromised if you build it on the hot path.
Related posts
- Every In-Process Python Sandbox Leaks, and the Reason Is Structural
Removing a name from a namespace removes a name, not reachability. CPython's object graph has no partitions, and `ctypes` is a documented feature whose purpose is calling arbitrary native code. Here is where the boundary actually has to go.
- Document Conversion Is Remote Code Execution With a Progress Bar
A convert endpoint accepts an arbitrary program from a stranger and runs it with your service account and your network position. The office formats are not data. They are a feature set, and some of those features dial out.
- Per-Tenant ML Inference Isolation With MicroVMs
A bring-your-own-model inference platform runs tenant B's pickle-loaded weights and Python preprocessing in the same worker that holds tenant A's data. That's cross-tenant RCE with extra steps. One microVM per tenant fixes it.
- Extracting Untrusted Archives: Zip Bombs, Zip Slip, Symlink Escape
Archive extraction is one of the few operations where the attacker picks the amplification factor. You cannot inspect your way out of that. You can only put it somewhere you are willing to lose.
- Build a Code Interpreter with a Python Sandbox
Build a ChatGPT-style code interpreter that runs untrusted Python in its own microVM — capture stdout, read back plots, persist state across cells.
More in Code execution · See Code interpreter sandboxes on PandaStack
49ms p50 cold start. Fork, snapshot, and scale to zero.