Running a Multi-Tenant Render Farm on MicroVMs: A .blend File Is a Program

I build PandaStack, an open-source Firecracker microVM platform, and one conversation keeps repeating itself. Someone runs a hosted rendering product — a Blender render API, an architectural-visualisation SaaS where clients upload scenes, a studio's farm that also takes files from outsourcing partners — and describes the workload as file processing. Customers upload a scene, we render it, we hand back frames. The question they ask is about cost per frame.

That is the wrong first question, because the premise is wrong. A scene file is not a document. It is a program with a document's file extension, and the renderer is its interpreter. Once a customer supplies both the scene and the render flags, you are operating a general-purpose code-execution service whose billing unit happens to be images, and everything about how you should build the farm follows from that sentence.

A .blend file is a program

Blender is a Python application with a 3D editor attached, and the file format reflects that honestly. A `.blend` can carry text datablocks marked to register on load. It can carry driver expressions — Python snippets evaluated to compute a channel value during dependency-graph evaluation, which is to say during the render you asked for. It can carry handlers that fire on frame change, and node graphs can contain scripted nodes. That is the feature set that makes Blender a pipeline tool; it is a problem only when the file's author is a stranger and the machine evaluating it is yours.

Blender is not unusual. A Houdini digital asset is a packaged graph that can carry Python for its own parameter callbacks — the point of an HDA is executable behaviour distributed as a file. Maya is worse in an interesting way: a `.ma` is an ASCII file of MEL commands, so the scene is literally a script the application sources, and script nodes have been a recurring malware vector there for years. Every serious scene format grew a way to carry code.

So a farm that accepts scene files accepts code. The standard advice is to disable auto-run, and that advice is correct. It is also weaker than it sounds, in two ways that only show up when you run this as a service rather than as a workstation.

First, the flag protects a process you do not fully control. Hosted products routinely let customers supply render arguments — output format, frame range, a scene name inside the file, sometimes a Python override because that is the only way to say "use my camera, my samples, my output path" against a file you did not author. Once extra arguments are customer-influenced, your hardening flag is one list concatenation from being overridden, and `-P` lives in the same argument vector. Second, a flag is a property of an invocation, not of a system. Your main render path passes it; does your thumbnail path, or the retry path someone wrote at 2am by copying a command out of a log?

The hardening flags, and why they are not the boundary

Here is roughly what a careful invocation looks like inside the guest. You should do all of this, and seeing it written out makes the gap obvious.

#!/bin/sh
# Inside the per-job microVM. Runs as a non-root user with no credentials
# in its environment and nothing interesting outside /work.
set -eu

# Keep Blender from loading the user's addons, startup scripts and prefs.
export HOME=/work/home
export BLENDER_USER_SCRIPTS=/work/empty-scripts   # exists, is empty
export BLENDER_USER_CONFIG=/work/empty-config

# Resource fences, so a pathological scene fails in userspace with an exit
# code instead of as a kernel OOM kill that takes the box with it.
ulimit -t 3600          # CPU seconds
ulimit -v 6291456       # address space, KiB
ulimit -f 8388608       # max output file size, blocks
ulimit -c 0             # no core dumps of somebody's scene
umask 077

# --factory-startup: ignore prefs and addons entirely
# --disable-autoexec: refuse embedded Python auto-execution
# --python-exit-code: make a script failure a non-zero exit, not a warning
# -noaudio: drop a subsystem we will never use
exec timeout --signal=TERM --kill-after=30s 3600 \
  blender --background \
          --factory-startup \
          --disable-autoexec \
          --python-exit-code 42 \
          -noaudio \
          "/work/scene.blend" \
          --render-output "/work/out/frame_####" \
          --render-format PNG \
          --render-frame "${FRAME}"

Every line earns its place. `--factory-startup` means an uploaded scene cannot rely on an addon you happened to install, and the `ulimit` values turn "one extra subdivision level" from a host-wide memory event into an exit code you can report to the customer. And none of it is a security boundary: it is configuration applied to a process, by a process, on a kernel that process talks to directly. The flags constrain what the renderer is willing to do on purpose; they have no opinion about what it can be made to do by accident.

The second surface: everything the job reads

A render job is not one file. Walk what gets parsed on a typical architectural-visualisation frame: PNG, JPEG, TIFF and OpenEXR for textures and HDRI environments; OpenVDB for volumetrics; Alembic and USD for cached geometry; glTF with Draco for imported props; FBX, which has an ancestry rather than a specification; TrueType for every text object; OCIO configuration for colour management. Each is a large C or C++ library parsing bytes chosen by whoever uploaded the asset.

Image and geometry parsers are the classic memory-safety neighbourhood — not a slur on maintainers who fuzz hard, but an observation about the shape of the work: read a length field, allocate, read that many bytes, interpret them as a structure, recurse. Notice what that does to the hardening story. `--disable-autoexec` is a decision the renderer makes about a datablock; a heap overflow in an EXR decoder does not consult your flags. The renderer is already running, and the attacker's goal is to take over the process running it — which on a shared farm can read every tenant's staged assets and holds the credentials to your asset store.

Why a container is the wrong boundary for this workload

I am not anti-container. But a container is a process with namespaces and cgroups on your host's single shared kernel, and the isolation is implemented by the same kernel the contained process makes syscalls into. That is fine when the code is yours and the threat model is accidental interference; it is thinner when the code was uploaded by a stranger specifically so that you would execute it. A container is a polite suggestion to the kernel, and the kernel's syscall surface is enormous next to a hypervisor's device model.

Now the render-specific part, which gets underweighted. This workload is legitimately CPU-saturating, long-running and memory-hungry: a correct, benign, paying job pins every core for forty minutes and climbs to the top of the memory graph. That is what success looks like. So the signature of a benign heavy render and the signature of a compromised node being used for something else are the same signature.

Think about what that does to your on-call. At 3am you are paged for a node at full CPU and high memory pressure with another tenant's job being OOM-killed. Noisy neighbour, or escalation? You cannot tell from outside, because the one signal you have is saturated by normal operation. You deleted your ability to distinguish a capacity problem from a security problem at the architecture stage, months before the page.

What a microVM per job actually buys

A Firecracker microVM is a hardware-virtualised guest with its own kernel. On PandaStack each sandbox boots Ubuntu 24.04 on guest kernel 5.10 under Firecracker v1.16, and the device model it gets is deliberately tiny: a handful of virtio devices, no more. That smallness is the argument. An exploit landing in a parser now has to get out of a guest kernel it fully owns and then through a hypervisor whose whole job is to be small enough to audit.

• Own guest kernel per job. A kernel-reachable bug in the parsing path compromises a kernel that is about to be deleted and has seen one customer's data.
• Own network namespace and TAP device, from a pool of 16,384 pre-allocated /30 subnets per host agent, so per-job network policy is a property of the job rather than a separate system you remembered to configure.
• A boundary that survives a malicious node in the scene graph. The threat is not an attacker with a shell; it is a driver expression or a crafted chunk header reached at frame 340 of 600.
• No state carried between tenants: no shader cache to poison, no leftover temp file holding someone's unreleased asset, and no node 7 that has been up for ninety days — which kills off the "it only fails on node 7" class of bug.
• Reproducibility as a side effect: every job restores the same baked snapshot generation, pinning the Blender build, the OCIO config and the guest kernel.

The job pipeline

1. Ingest. The customer uploads a scene archive to object storage directly, not through your API. You record a job row with the archive digest, tenant and frame range. Nothing has been parsed yet, by anything.
2. Plan without opening the file. Split work from metadata the customer declared: frames 1 to 600 becomes units of N frames, a single huge frame becomes a tile grid. Resist peeking inside to count frames — that peek is a full parse, the exact operation you are isolating.
3. Fetch assets with something that is not the renderer: a boring fetcher that applies your allowlist, caps sizes and redirect counts, and never parses image or geometry data.
4. Create one microVM per unit from a baked render template, with a TTL that is a hard ceiling rather than a suggestion.
5. Write the unit in, with no credentials. The guest gets no token for your asset store or your queue.
6. Render with the fences above, streaming logs out as they happen, so a job about to fail tells you before the TTL does.
7. Read outputs back, verify them, publish under a key derived from job and unit identity.
8. Delete the sandbox unconditionally, in a `finally`, including where the render failed or your orchestrator crashed. The TTL is the backstop for when your code forgets.
9. Stitch from object storage, never from a worker's local disk, so reassembly does not depend on a machine that may be gone.

Step 3 is the one people skip, and it quietly determines your threat model.

Fanning a frame range across sandboxes

A frame range is embarrassingly parallel, so the orchestrator is a work queue plus a pool, and all the interesting logic is in the failure handling.

import concurrent.futures as cf
import pathlib
from pandastack import Sandbox

SCENE = pathlib.Path("/stage/job-7f3a/scene.blend").read_bytes()
ASSETS = pathlib.Path("/stage/job-7f3a/assets.tar").read_bytes()
JOB = "job-7f3a"

RENDER_SH = r"""#!/bin/sh
set -eu
export HOME=/work/home BLENDER_USER_SCRIPTS=/work/empty
ulimit -t 3600; ulimit -v 6291456; ulimit -c 0; umask 077
mkdir -p /work/out /work/home /work/empty
exec blender --background --factory-startup --disable-autoexec \
  -noaudio /work/scene.blend \
  --render-output /work/out/frame_#### --render-format PNG \
  --render-frame "$1"
"""


def render_frame(frame: int, attempt: int = 1) -> str:
    """Render one frame in its own microVM. Returns the object-store key."""
    key = f"{JOB}/frames/frame_{frame:04d}.png"
    if object_store_exists(key):          # idempotency: someone already did it
        return key

    sbx = Sandbox.create(
        template="base",
        ttl_seconds=5400,                 # hard ceiling above the ulimit -t
        metadata={"job": JOB, "frame": str(frame), "attempt": str(attempt)},
    )
    try:
        sbx.filesystem.write("/work/scene.blend", SCENE)
        sbx.filesystem.write("/work/assets.tar", ASSETS)
        sbx.filesystem.write("/work/render.sh", RENDER_SH)
        sbx.exec("mkdir -p /work/out && tar -xf /work/assets.tar -C /work",
                 timeout_seconds=300, check=True)

        r = sbx.exec(f"sh /work/render.sh {frame}", timeout_seconds=4800)
        if r.exit_code != 0:
            raise RenderFailed(frame, r.exit_code, r.stderr[-4000:])

        png = sbx.filesystem.read(f"/work/out/frame_{frame:04d}.png")
        if len(png) < 1024 or png[:8] != b"\x89PNG\r\n\x1a\n":
            raise RenderFailed(frame, 0, "output is not a plausible PNG")

        object_store_put(key, png)        # write-once key
        return key
    finally:
        sbx.kill()                      # the TTL is the backstop, not the plan


def render_range(first: int, last: int, width: int = 24) -> dict[int, str]:
    done, pending = {}, list(range(first, last + 1))
    for attempt in (1, 2, 3):
        if not pending:
            break
        failed = []
        with cf.ThreadPoolExecutor(max_workers=width) as pool:
            futures = {pool.submit(render_frame, f, attempt): f for f in pending}
            for fut in cf.as_completed(futures):
                frame = futures[fut]
                try:
                    done[frame] = fut.result()
                except Exception as exc:
                    log.warning("frame %d attempt %d failed: %s",
                                frame, attempt, exc)
                    failed.append(frame)
        pending = failed
    if pending:
        raise JobFailed(JOB, sorted(pending))
    return done

Three things there are deliberate. The `ttl_seconds` ceiling sits above the in-guest `ulimit -t`, so the normal failure is a clean non-zero exit with usable stderr rather than a sandbox vanishing mid-write. `sbx.kill()` is in a `finally`, because the happy path is not where leaks come from. And the output is checked in the worker, because a renderer that exits zero and writes a truncated file happens.

`sbx.filesystem.write` is the right tool up to a comfortably-sized archive; for a scene that is tens of gigabytes, attach a durable volume or hand the guest a short-lived signed URL. On concurrency, with 16,384 pre-allocated subnets per agent you will not run out of network slots before you run out of CPU and memory. PandaStack's scheduler scores hosts at `0.6 * free_cpu + 0.3 * free_mem_gb` after hard capacity gates, with 10-second heartbeats and agents more than 30 seconds stale excluded, so a wide fan-out spreads.

Tiling one frame, and the worker that dies at 94%

Frame ranges are easy. The harder case is a single enormous still — a 16K architectural print, a poster-resolution product shot — where the unit of work is a region of one image. Most renderers will render a sub-rectangle happily; in Blender you set the render border and crop, so each sandbox renders its tile with the same camera and sampling configuration. Three rules make that survivable.

1. Tile identity is derived, never assigned. The output key is a hash of (scene digest, render-settings digest, tile rectangle, sample seed), so two workers rendering the same tile produce the same key and the second write is a no-op. That is what lets you retry aggressively without bookkeeping.
2. Tiles are written once, whole, never appended. Partial uploads are the one failure mode that turns a retry from harmless into silently-corrupt-final-image, so use a write-once key and a completion marker.
3. The deadline is yours, not the platform's. Enforce how long a tile may take inside the guest, so a slow tile fails as a tile with logs attached. A sandbox reaped by TTL tells you only that time passed.

With those in place, a worker dying at 94% of a tile is a non-event: nothing was published, the key is still absent, the next attempt re-renders that rectangle. Compare the shared-farm version, where a node died holding the only copy of nineteen finished tiles in `/tmp` and you are now reasoning about which were flushed.

Branching deserves a word, because the two primitives have confusingly similar names. `sbx.fork()` clones the rootfs by reflink and then cold-boots the child, so it inherits the disk but not the memory — its own kernel, its own PIDs, its own entropy — which makes it the roughly-3-second cold-boot shape, not a sub-second one. `sbx.fork_tree(4)` snapshots once and restores N children that do inherit live memory. Both cap at 16 children per call. For tiling fan-out `fork()` is usually the one you want: the gigabytes of scene and texture bytes are shared on disk by reflink instead of copied into every worker, and three seconds of boot is noise against a render measured in minutes. Reach for `fork_tree` only when you want the parsed scene already resident — and then remember the children inherit seeded RNG state, a consistency feature for tiles of one image and a trap for N scene variations. Mechanics in Snapshots and Forks: Copy-on-Write for Running Machines.

Assets in, frames out, and the egress question

Ingress has three shapes and you will use two. Small to medium scenes go in with `sbx.filesystem.write` straight from your orchestrator, which has the pleasant property that the render VM needs no network at all. Large recurring asset libraries belong on a durable volume mounted per job, so you pay the transfer once rather than per render. One-off multi-gigabyte scenes get a signed URL and an in-guest fetch.

Egress out is the mirror: `sbx.filesystem.read` returns the frame as bytes to a process that already holds your storage credentials, keeping those credentials out of the guest entirely. Protect that asymmetry. The highest-value property of this architecture is that the machine running a stranger's code holds nothing worth exfiltrating, and the fastest way to throw it away is to hand the guest an asset-store token because it was convenient.

That default is right for the platform's main workloads — a sandbox that cannot install a package is not much use — and exactly wrong for a render job. So ask who chooses the URL. If your pipeline fetches whatever the scene points at, from inside the render VM, the scene file picks your outbound traffic: a crafted reference can name an internal service, a metadata endpoint, or a host that answers differently on the second request. That is SSRF with a 3D file as the request builder.

The structural fix is step 3: resolve external references in a separate fetcher outside the render VM, with an allowlist, size caps, redirect limits and nothing interesting in reach, then write the results in. The render VM then needs no egress, and "no egress" is far easier to verify than "correct egress." The general version is in Controlling Network Egress for Untrusted Code.

The honest limits

GPU rendering is a different architecture

Firecracker's device model is minimal on purpose, and PCIe passthrough of a discrete GPU is not part of it. If your product is OptiX or CUDA rendering, microVM-per-job is not your substrate; device passthrough is a different design with a different isolation story, and the trade-offs are in GPU Passthrough and microVMs: The Honest Answer. What this pattern does cover is still most of real production rendering: CPU path tracing, format conversion and validation, decimation and LOD generation, UV unwrapping and texture baking, thumbnailing, denoise passes and compositing — all CPU-bound, all embarrassingly parallel, all driven by files a stranger supplied.

Guest size is a template decision, not a per-request one

Firecracker cannot change a guest's vCPU count or RAM at snapshot restore. Since every PandaStack create restores a baked snapshot, guest size is a property of the template, and a per-request `memory_mb` is overridden to match what was baked. For rendering that is a design constraint rather than a detail, because "how much RAM does this scene need" is the central question of the domain and is not knowable before you parse the scene. A 64 GiB scene is a template decision, not a request parameter.

So build template tiers the way managed databases do — small, medium and large render templates, each baked at its own size — and route jobs by declared class, with a documented policy for exceeding it. That policy should be "fails cleanly against its own `ulimit` and is offered a retry at the next tier up," not "gets OOM-killed somewhere and produces a support ticket shaped like a mystery."

The 179 ms create time is almost irrelevant here

I will be blunt, because this is where microVM pitches oversell. A PandaStack create is 179 ms at p50 and 203 ms at p99, because every create restores a baked snapshot rather than cold-booting; the first-ever spawn of a template cold-boots in about 3 seconds and bakes the snapshot for everyone after it. Those are the right headline for an agent sandbox, where something is waiting. A render job takes minutes to hours: against a forty-minute frame, the difference between a 179 ms create and a 30-second boot is a tenth of a percent of wall clock. Anyone selling you per-job isolation for rendering on cold-start latency is selling the wrong axis.

What matters economically is the other end: teardown. Per-job VMs are affordable here because the job's machine exists only while the job does. PandaStack bills $0.054 per vCPU-hour and $0.0162 per GiB-hour across all classes, accruing while the sandbox exists, with CPU charged on CPU-seconds actually burned. A deleted sandbox accrues nothing, and there is no warm pool of idle VMs underneath, because snapshot-restore-on-every-create removes the need for one. You are not paying for isolation; you are paying for the render.

Container per job vs microVM per job vs shared bare metal

Read the last row as seriously as the first. If every scene on your farm was authored by someone with a badge, the shared farm is fine and this post is overhead. The moment files arrive from outsourcing partners, contractors, a self-serve API or a free tier, the trust assumption that made it reasonable has quietly expired — and usually nobody writes that down.

The customer whose scene renders a single black frame

Somewhere in every self-serve compute product's history there is a customer whose jobs are models of good behaviour. They submit regularly, they never complain, their scenes always succeed. Each job produces a single 1920 by 1080 frame that is entirely black, in about four hours, at full utilisation on every core you will give them.

You have built, with great care, a system that executes arbitrary code at high CPU for long periods on behalf of anonymous users and then bills someone for it. That is also a precise description of a mining pool's ideal customer, except that in their business model you are the one paying — on a free tier you are literally buying the electricity. The black frame is the tell, and the joke is that it is not a bug: it is the output of a scene whose only job is to be a plausible reason for the CPU time.

The microVM boundary does not stop this — isolation is not abuse prevention — but it makes the abuse contained, attributable and cheap to end. One job is one VM, so the resource profile belongs to one tenant with no ambiguity. Past the protocol-level DROP rules that catch the lazy version it is product work: output plausibility checks, a ratio between CPU-seconds burned and pixels produced that someone actually looks at, and a payment method before the expensive tier. The remedy is then a `delete` call and a flag on an org, not forensics on a node shared by four hundred jobs.

The summary

A hosted render farm is a code-execution platform in a creative-tools costume. Scene files carry Python, MEL or a packaged graph that executes by design, and the flags that disable it are a property of each invocation rather than of your system. Beneath them is the loader layer — EXR, TIFF, PNG, Draco, OpenVDB, USD, FBX, fonts — a very large amount of C reached by bytes a stranger chose, which does not consult your flags before it overflows. And the workload's own resource signature is indistinguishable from that surface being exploited, so on a shared kernel you cannot tell a capacity page from a security page at 3am.

One microVM per job answers the whole family: own guest kernel, own network namespace, a small device model, no credentials inside, and a machine deleted when the job ends. Fan the frame range out wide, derive unit identity from content so retries are free, publish write-once so a worker dying at 94% is a non-event, and fetch assets with something that is not the renderer. Keep the pitch honest while you do it: the 179 ms create rounds to nothing against a forty-minute render, GPU rendering is a different architecture, and guest sizing is baked into the snapshot. What makes this affordable is the other end of the lifecycle — isolation you stop paying for the instant a job ends is isolation you can afford on every job, including the free ones, which is where the interesting files arrive.