Per-Tenant MicroVM Isolation for Thumbnail Generation

If you're building a multi-tenant SaaS product, you have a thumbnailing pipeline somewhere, whether you designed one on purpose or not. Avatar uploads. Product photos for the storefront. A CMS media library. A chat app that previews attachments. Somewhere behind an upload button, a worker takes a stranger's bytes, hands them to Pillow or libvips or ImageMagick, and gets back a resized JPEG. That worker almost certainly also handles thumbnailing for every other tenant on the platform, because that's the obvious way to build it: one pool of workers, one queue, one library, N customers.

I'm Ajay — I build PandaStack, which runs Firecracker microVMs as a service, so take the pitch angle as read. What I want to walk through here is narrower than "sandbox everything": specifically why a shared thumbnailing worker puts every tenant in the blast radius of one malicious upload from any other tenant, what a decompression bomb or a crafted-file exploit actually does to that shared worker, and the concrete shape of per-tenant (or per-job) microVM isolation that contains the damage to one disposable VM instead of your whole fleet.

The shared-worker problem, specifically for multi-tenant SaaS

Single-tenant apps get to reason about "our uploads." Multi-tenant apps don't have that luxury — every image that hits your thumbnailer was uploaded by someone else's customer, someone else's vendor, someone else's anonymous signup flow. Tenant A's malicious avatar and Tenant Z's legitimate product photo are, from the worker's point of view, two calls to the same decode function on the same process, often within milliseconds of each other.

That matters because the failure modes of image libraries aren't tenant-scoped. A decompression bomb doesn't know whose upload it is — it just allocates memory until the worker OOMs, and every job in that worker's queue dies with it, including Tenant Z's harmless PNG that happened to be next in line. A heap-corruption bug in the decoder doesn't stay inside "Tenant A's request" either; if it gets code execution, it gets the whole worker process, which typically holds the database credential, the object-storage credential, and the internal service token for every tenant the worker serves, not just the one whose file triggered it. Multi-tenancy turns a per-file bug into a per-fleet incident.

Why the decoder is the risk, not the format

ImageMagick, libvips, and Pillow (which itself wraps libjpeg-turbo, libpng, libwebp, libtiff, and friends under the hood) are enormous, mature C and C++ codebases whose entire purpose is parsing binary formats designed by other people, some of whom did not have your worker's memory safety in mind. That's not a knock on any of these projects — they're excellent, widely deployed, and actively maintained — it's just a statement about what the job is. Decoding attacker-supplied bytes into pixels is inherently adversarial input handling, and the ecosystem's history reflects that: ImageMagick's ImageTragick-era disclosures showed how a crafted file could steer the library into shelling out to an external delegate; various libwebp and libtiff advisories over the years have covered heap overflows reachable straight from a malformed file; SVG-in-image and PDF-in-image tricks have been used to coax "image" decoders into fetching external resources or invoking other interpreters entirely. I'm not going to cite specific CVE numbers here — go read your vendors' advisories for the exact versions you run, because that's the only version of this list that's actually actionable for you.

The upshot is a genre of bug, not a fixed list of bugs. Patch today's disclosed issue and you're protected against today's disclosed issue. The next one is, structurally, still out there, because the job — parse hostile bytes with a general-purpose C parser — hasn't changed.

Three failure classes, one shared worker

It's worth naming the three ways a thumbnailing job goes wrong, because a shared worker is equally exposed to all three and a per-tenant microVM neutralizes all three the same way — by making the blast radius one disposable machine instead of the process everyone's jobs run in.

• Decompression and pixel bombs — a small file that expands to an enormous allocation: a zip-bomb-style compressed payload, or a header that simply claims a 50,000-by-50,000-pixel image before a single pixel has been decoded. Nothing was exploited; the library did exactly what it was asked, and your worker's memory is gone.
• Memory-safety bugs in the decoder — a malformed TIFF tag, a crafted WebP chunk, a font-hinting bytecode path — that trips a heap overflow or use-after-free and hands the attacker code execution as your worker process, with whatever credentials that process holds.
• Confused-deputy tricks — an SVG that references an external resource, a delegate that ImageMagick can be steered into shelling out to, a PDF page embedded where an "image" was expected — that make the parser do something it was built to do, on the attacker's behalf, from inside your network.

Per-tenant microVM isolation: one bomb, one disposable VM

The fix isn't to write a bug-free decoder — nobody has managed that in thirty years of image libraries and you're not going to be the exception. The fix is to make sure that when the decoder does misbehave, what it's misbehaving inside of is a machine that contains nothing valuable and gets destroyed within seconds. Concretely: every thumbnailing job — one per tenant, or one per upload if you want the tightest possible boundary — gets a fresh Firecracker microVM. The VM receives the raw upload bytes and nothing else: no database credential, no object-storage credential, no service token, no network route to anything interesting. It runs the resize, returns the output bytes, and is destroyed. If the file was a decompression bomb, the VM dies with an OOM and your handler sees a non-zero exit code. If the file was a crafted exploit, the attacker's prize is a machine holding one file they already had, for the seconds before it's deleted.

This also solves the specific multi-tenant version of the problem that a shared worker can't: Tenant A's malicious upload can no longer take down Tenant Z's job, because there is no shared process for it to take down. Each tenant's thumbnailing job lives in its own throwaway kernel.

The pattern with the PandaStack SDK

Five steps: create a sandbox, write the uploaded bytes in, run the resize, read the thumbnail bytes out, destroy the sandbox. Nothing about the sandbox knows which tenant uploaded the file, and nothing in the sandbox can reach anything that isn't the file itself.

from pandastack import Sandbox

MAX_UPLOAD = 20 * 1024 * 1024  # 20 MiB, checked before we spend a VM

RESIZE_SCRIPT = """
import sys
from PIL import Image

Image.MAX_IMAGE_PIXELS = 64_000_000  # ~64MP hard cap: refuse pixel bombs
                                      # before PIL allocates a raster for them

src, dst, size = sys.argv[1], sys.argv[2], int(sys.argv[3])
with Image.open(src) as im:
    im.load()          # force full decode now, inside the ulimit below
    im.thumbnail((size, size))
    # Re-encode fresh; never pass the original bytes through to storage.
    im.convert("RGB").save(dst, "JPEG", quality=85, optimize=True)
"""


def thumbnail_for_tenant(tenant_id: str, upload: bytes, size: int = 512) -> bytes:
    if len(upload) > MAX_UPLOAD:
        raise ValueError("upload too large")

    # One VM per job. It never learns tenant_id -- that's metadata for our
    # own logs, not something the guest needs to do its job.
    with Sandbox.create(template="base", ttl_seconds=120) as sbx:
        sbx.filesystem.write("/work/in.bin", upload)
        sbx.filesystem.write("/work/resize.py", RESIZE_SCRIPT)

        r = sbx.exec(
            "cd /work && ulimit -v 786432 && ulimit -t 20 && "  # ~768MiB, 20s CPU
            f"timeout -s KILL 25 python3 resize.py in.bin out.jpg {size}",
            timeout_seconds=30,
        )
        if r.exit_code != 0:
            # Malicious and merely-corrupt files look identical from here.
            # Reject; don't retry on a bigger box.
            raise ValueError(f"thumbnail failed ({r.exit_code}): {r.stderr[-1500:]}")

        return sbx.filesystem.read("/work/out.jpg")  # bytes, nothing else
    # Sandbox destroyed on block exit -- the bomb, the exploit attempt, and
    # the original file all go with it.

Two things worth pointing at. `Image.MAX_IMAGE_PIXELS` plus the `ulimit -v` belt-and-braces the same failure twice on purpose: the library-level cap rejects an absurd header before PIL allocates a raster for it, and the process-level address-space limit catches anything the library-level cap didn't anticipate. And `im.load()` forces the full decode to happen inside the ulimit'd process rather than lazily on first access later — with Pillow's lazy decoding, a bomb can otherwise detonate somewhere you didn't intend to be measuring it. If you're on libvips instead of Pillow, the equivalent guardrails are `VIPS_DISC_THRESHOLD` and disabling untrusted loaders; if you're on ImageMagick, it's `-limit area` plus a locked-down `policy.xml` with delegates disabled — the decoder-specific knobs change, the ulimit-and-disposable-VM wrapper around them doesn't.

Shared worker vs. per-tenant microVM

• Isolation boundary — Shared worker: none between tenants; every job runs in the same process, same heap, same open credentials, same network position as every other tenant's job. Per-tenant microVM: a separate guest kernel and hypervisor boundary per job; Firecracker's device model is deliberately tiny (virtio-net, virtio-blk, virtio-vsock, serial, RNG) and runs behind a jailer and seccomp filter, so escaping means defeating the guest kernel and then that surface.
• Decompression-bomb blast radius — Shared worker: one bomb OOMs the process, which kills every other tenant's in-flight job in the same worker and can trigger noisy-neighbor pressure across the node. Per-tenant microVM: RAM is capped at the hypervisor boundary per VM, so the worst case is one VM dying with a bad exit code while every other tenant's job runs untouched in its own VM.
• Crash and exploit recovery — Shared worker: a crashed or compromised process takes its whole job queue down with it, and if the crash was actually a heap-corruption exploit, the attacker inherits every credential that worker held for every tenant it serves. Per-tenant microVM: the VM that crashed or got exploited is destroyed on job completion (successful or not) regardless, so recovery is "the next tenant's job spins up a fresh VM," not an incident.
• Credential exposure — Shared worker: total, if the exploit lands — database, object storage, and internal service tokens are usually all reachable from the same process. Per-tenant microVM: nothing, as long as the VM is only ever handed file bytes and never a credential, which the pattern above enforces by construction.
• Cost per job — Shared worker: effectively free marginally, since the process is already warm and amortized across every tenant's traffic. Per-tenant microVM: on PandaStack a create restores a baked snapshot rather than cold booting, p50 179ms / p99 around 203ms per create, so a fresh VM per job adds real but small latency — well under the time the resize itself takes for anything but a tiny thumbnail.

When per-tenant isolation is more than you need

If your "multi-tenant" product only ever thumbnails images your own systems generated — say, screenshots your own rendering pipeline produced — you don't have an attacker-controlled input problem and a per-tenant VM buys you very little. Isolation earns its cost when the bytes came from someone you don't control: a signup form, a public API, a customer's customer. If every tenant's upload path already runs inside its own hardware-isolated compute — a per-tenant VM or container for the whole request, not just thumbnailing — you likely already have this property and adding a second sandbox inside it is redundant. And if the actual bug you're worried about is authorization (Tenant A reading Tenant B's thumbnails from shared storage) rather than decoder exploitation, that's an access-control fix, not a sandboxing one — fix it first, since it's both more common and cheaper for an attacker to find.

Where it does earn its cost is exactly the shape described at the top: a shared worker pool, public uploads, a general-purpose C decoder, and more than one tenant's data reachable from the same process. That's most multi-tenant SaaS thumbnailing pipelines as they're actually built today, whether or not anyone chose it on purpose.