Rendering Untrusted Ad Creatives in Per-Tenant MicroVMs

Every ad platform, DSP, retail-media network, and app-monetization SDK eventually builds the same service: the creative pipeline. An advertiser uploads a creative — an HTML5 banner ZIP, a rich-media unit, an MP4, a set of images, a third-party ad tag — and your system has to render it, look at it, measure it, and decide whether it is allowed to run on your inventory. Almost every team builds that on a pool of long-lived headless Chrome instances and an ffmpeg box, because that is the obvious way to build it and it works fine right up until it doesn't.

The problem is what a creative actually is. An HTML5 ad unit is not a picture of an ad. It is arbitrary JavaScript, arbitrary CSS, arbitrary fetches, and often a chain of third-party tags that resolve to code you have never seen, from a vendor you have no contract with, chosen by a buyer you onboarded through a self-serve form with a credit card. You execute it because that is the product. Malvertising is not a hypothetical threat model; it is an industry with a supply chain, tooling, and people whose full-time job is getting a payload past exactly the review pipeline you are building.

I'm Ajay, I built PandaStack — this post is about the boundary that actually holds for creative rendering: one Firecracker microVM per creative job, with the browser and the transcoder hardware-isolated, egress locked to your asset CDN, and the sandbox itself used as the detection instrument.

What a creative pipeline actually does

It is worth being concrete, because the security argument depends on how much untrusted execution hides inside what looks like a batch job. A typical creative-ingest pipeline does most of this for every upload:

• Render a preview — load the creative in a real browser at each supported size, wait for network idle, and capture a screenshot for the review queue and the advertiser's own preview UI.
• Measure weight and load time — total transferred bytes, request count, time to first render, CPU time burned. Publishers enforce these limits contractually, so the numbers have to come from an actual render, not a static analysis.
• Policy-scan the behavior — does it auto-expand, autoplay with sound, trigger a redirect without a click, load a cryptominer, request geolocation, open a popunder, or fetch a second-stage script from a domain that wasn't in the declared vendor list.
• Transcode video and image assets — ffmpeg for MP4/WebM variants and thumbnails, ImageMagick or libvips for image derivatives, all of it running native decoders over files an advertiser uploaded.
• Resolve third-party tags — a VAST wrapper or JS tag that points at another server, which points at another server; the real creative only materializes at render time.
• Store the artifacts — screenshots, transcodes, extracted metadata — into your object store, under credentials the rendering process is holding.

Read that list again as an attacker would. You have offered to execute their JavaScript, in your infrastructure, on demand, with a browser, while holding object-store credentials, and you will tell them the result. That's a very generous API.

The creative is a program, and you agreed to run it

The uncomfortable part of creative review is that the untrusted input isn't a document you parse — it's a program you deliberately execute, in the most complicated runtime ever shipped. A browser renderer is millions of lines of C++ handling adversarial input by design: a JIT compiler, an image decoder stack, a font shaper, a video pipeline, a compositor. It is one of the most-attacked pieces of software on earth, which is why it has a sandbox of its own and why that sandbox gets escaped often enough to have a standing bug-bounty price list.

The media side is no better. "ffmpeg CVE" and "ImageMagick CVE" are not incidents, they are a genre — demuxers, decoders, and delegate handlers parsing attacker-controlled containers in C, with a long history of memory-safety bugs and the occasional feature that turns out to be a file-read primitive. Your transcoder is fed MP4s uploaded by strangers who are financially motivated to get code running on your machines.

Why the shared headless-Chrome farm is the wrong boundary

The default architecture is a pool of persistent headless browsers, each handling creative after creative, tab after tab, often from different advertisers within the same minute. Sometimes it's one browser per pod, sometimes it's a browser-per-tab-pool because someone benchmarked the startup cost and decided reuse was worth it. Either way, the isolation between two advertisers is a browser context — a boundary designed to stop cookie leakage between websites, not to stop a determined party who bought their way into your renderer.

That boundary fails in several directions at once. A renderer exploit reaches a process holding page state and caches from other advertisers' creatives, and often the credentials your automation uses to upload screenshots. A creative that leaves timers, service workers, or sockets running poisons the next render on the same instance, which makes your policy scan non-deterministic in a way nobody diagnoses for months. And a memory-bomb creative — a canvas allocation loop is four lines — takes down a pod running dozens of unrelated jobs, so an advertiser can degrade your review SLA for every competitor by uploading one bad banner.

• Isolation strength — Shared headless-Chrome farm: browser contexts on one process tree, one kernel; a renderer escape reaches every advertiser's assets in that pool. Container per render: fresh process and filesystem, but the host kernel is shared, so a renderer escape plus a kernel bug reaches the host and its neighbours. MicroVM per render: hardware-virtualized guest with its own kernel — a full Chrome escape lands the attacker inside a disposable VM containing one creative.
• State bleed between advertisers — Shared farm: caches, service workers, storage, and leftover timers survive into the next creative unless every teardown is perfect. Container: clean per run if you don't reuse the container, which teams usually end up doing for speed. MicroVM: restored from an identical baked snapshot every time; there is no previous tenant.
• Memory-bomb creative — Shared farm: OOMs the pod and every job on it. Container: hits a cgroup limit, and the host OOM killer picks a victim that isn't always the offender. MicroVM: hits the guest's own fixed RAM ceiling and dies alone; the fleet doesn't notice.
• Infinite-loop creative — Shared farm: burns a worker until someone's watchdog notices, if there is one. Container: needs an external timeout you remembered to wire up. MicroVM: ttl_seconds is enforced by the platform, so a spin loop is billed for minutes, not until Tuesday.
• Egress control — Shared farm: the pool needs broad outbound reach for every vendor any creative might legitimately use, so every creative inherits all of it. Container: per-container network policy, if your platform actually enforces it. MicroVM: its own network namespace and routing — allow the asset CDN, drop the rest.
• Behavioral detection — Shared farm: network and CPU signals are mixed across concurrent creatives, so attribution is guesswork. Container: better, but shared host networking still blurs it. MicroVM: one creative per network namespace, so every packet and every CPU-second is attributable to exactly one creative.

One microVM per creative job

The structural fix is to stop rendering advertiser code inside anything you intend to keep. Your control plane — the part holding the creative database, the object-store credentials, and the review queue — stays on a trusted host and drives disposable sandboxes over an API. The creative bundle is written into the guest, the browser runs in the guest, the transcode runs in the guest, and only a screenshot plus a structured verdict comes back. When the job ends you destroy the VM, and everything the creative did goes with it: service workers, spawned processes, whatever a second-stage payload managed to install.

This is only practical because creating the VM is cheap. PandaStack doesn't cold-boot per job — every create restores a baked Firecracker snapshot on demand, with the restore step around 49ms and end-to-end create at p50 179ms (p99 ~203ms). A cold boot is ~3s and only happens the first time a template is baked. A hardware-isolated machine per creative therefore costs about a fifth of a second, which is rounding error next to waiting for network idle on a rich-media unit. Each agent also pre-allocates 16,384 network slots, so concurrency is bounded by host memory and CPU, not by network plumbing.

import json
from pandastack import Sandbox

def review_creative(creative_id: str, bundle: bytes, w: int, h: int) -> dict:
    """Render one advertiser creative in its own microVM and return a verdict.
    The host never loads advertiser JavaScript."""
    sbx = Sandbox.create(
        template="browser",
        ttl_seconds=120,                 # an infinite-loop creative is not billed forever
        metadata={"creative": creative_id, "job": "review"},
    )
    try:
        # The untrusted bundle goes INTO the guest. Nothing is unzipped host-side.
        sbx.filesystem.write("/work/creative.zip", bundle)
        sbx.exec("cd /work && mkdir -p unit && unzip -o -q creative.zip -d unit",
                 timeout_seconds=30)

        # render.py drives headless Chrome over CDP: loads index.html at the slot
        # size, records every request the creative makes, waits for network idle,
        # screenshots, then dumps weight + behaviour signals as JSON.
        r = sbx.exec(
            f"cd /work && python3 render.py --dir unit --width {w} --height {h} "
            f"--shot shot.png --report report.json --budget-ms 20000",
            timeout_seconds=90,
        )
        if r.exit_code != 0:
            # A renderer crash is now a boring job failure, not an incident.
            return {"creative": creative_id, "verdict": "error",
                    "detail": r.stderr[-2000:]}

        report = json.loads(sbx.filesystem.read("/work/report.json"))
        shot = sbx.filesystem.read("/work/shot.png")   # bytes, straight to review queue
        return decide(creative_id, report, shot)
    finally:
        sbx.kill()   # service workers, timers, and any spawned process die here

Notice what crosses the boundary in each direction. Going in: bytes an advertiser uploaded. Coming out: a PNG and a JSON report your host parses with a strict schema. The host never runs a browser, never unzips a bundle, never hands a decoder an untrusted MP4. If the creative achieves code execution, it achieves it in a guest holding one creative, no object-store credentials, and a two-minute lifetime.

Egress: the asset CDN and nothing else

Isolation that only covers the CPU is half a boundary, and for creative review the network half is arguably the more important one. Hostile creatives are usually not trying to own your infrastructure — that's a bonus. They're trying to phone home: beacon that they reached a real render, fetch a second-stage script that only serves the payload to certain geos or user agents, or exfiltrate whatever they can read. A shared browser farm with open outbound access gives them all of it.

Because each sandbox gets its own network namespace with its own routing and NAT rules, egress policy is per-creative rather than per-fleet. Default-deny, then open exactly the holes a legitimate creative needs — typically your own asset CDN, and for tag-based creatives an explicit allowlist of vendor domains the advertiser declared. Everything the creative tries that isn't on that list is not just blocked, it's evidence.

#!/usr/bin/env bash
# Egress policy for a creative-render guest: the asset CDN, and nothing else.
# Applied per-sandbox -- each microVM has its own netns, so this is not fleet-wide.
set -euo pipefail

CDN_IP="${ASSET_CDN_IP:?set the asset CDN IP}"

# Default deny outbound. Loopback and established replies stay.
iptables -P OUTPUT DROP
iptables -A OUTPUT -o lo -j ACCEPT
iptables -A OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# Cloud metadata: the first thing an exploited renderer reaches for.
iptables -A OUTPUT -d 169.254.0.0/16 -j LOG --log-prefix "CREATIVE-METADATA "
iptables -A OUTPUT -d 169.254.0.0/16 -j DROP

# No lateral movement into the VPC: your queue, your DB, your neighbours.
iptables -A OUTPUT -d 10.0.0.0/8     -j DROP
iptables -A OUTPUT -d 172.16.0.0/12  -j DROP
iptables -A OUTPUT -d 192.168.0.0/16 -j DROP

# The one legitimate hole: HTTPS to the asset CDN serving this creative.
iptables -A OUTPUT -d "$CDN_IP" -p tcp --dport 443 -j ACCEPT

# Log everything else before dropping -- the log IS the malvertising signal.
iptables -A OUTPUT -j LOG --log-prefix "CREATIVE-EGRESS-DENY " --log-level 4
iptables -S OUTPUT

Bake these rules into the template so they are in force the instant the snapshot restores, rather than racing the creative to apply them after boot. And be deliberate about DNS: if the guest resolves arbitrary names, DNS is itself a beaconing and low-bandwidth exfiltration channel that never needs your firewall's permission. Pin the CDN by IP where you can, and point the guest at a resolver you control so the query log is yours.

Detecting malvertising by watching the sandbox

Static analysis of creative JavaScript loses. It is minified, obfuscated, packed, and frequently fetches its real payload at render time based on the geo, the referrer, or a coin flip. The reliable signal is behavioural: what did this thing actually do when we let it run? And behavioural signals are only trustworthy if they are attributable to exactly one creative — which is precisely what a shared browser farm destroys, because forty concurrent renders share one network stack and one CPU accounting domain.

Give each creative its own guest and its own network namespace and the sandbox becomes an instrument. Every packet in that namespace belongs to this creative; every CPU-second in that guest was burned by it; every denied connection in the log was attempted by it. The signals worth collecting are mostly boring and mostly decisive:

1. Egress attempts outside the declared vendor list — a creative reaching for a domain the advertiser never declared is the single highest-value signal you can collect, and default-deny turns it into a log line instead of a breach.
2. Metadata-endpoint or RFC1918 probing — no legitimate banner has ever needed 169.254.169.254. One packet there is not a policy violation, it's an attack, and it should escalate rather than just fail review.
3. CPU burn and sustained utilization after load — a display banner that pins a core for twenty seconds is either a cryptominer or so badly built that publishers will complain anyway. Same verdict either way.
4. DOM churn and layout thrash — mutation counts and forced reflows over the render window catch auto-expanding units and the "invisible overlay that steals the click" pattern.
5. Navigation without user gesture — record CDP navigation events and check whether any user interaction preceded them. Auto-redirect creatives are the most common malvertising payload and they announce themselves clearly.
6. Media and popup APIs, plus weight and request count — autoplay-with-sound attempts, geolocation prompts, popunders, transferred bytes, and time to first render. Each maps directly to a published policy rule, and you need the weight numbers for publisher contracts anyway.

Collect those into a verdict document per creative and store it beside the screenshot. Reviewers get a picture and a behaviour sheet, your policy engine gets structured input, and when an advertiser disputes a rejection you have packet-level receipts instead of a vibe.

Fan out over a campaign's creative set with fork

One VM per creative is the right isolation unit, but a campaign is rarely one creative. It is a set: six sizes, four locales, three variants, plus video cuts — dozens of renders that all need the same warm browser, the same fonts, the same instrumentation.

This is what fork is for. Prepare one guest with Chrome warmed, your CDP driver loaded, fonts installed, and the policy instrumentation in place. Snapshot it while it is still creative-neutral — code only, no advertiser bytes. Then fork that snapshot once per creative in the campaign. Each fork is an independent microVM sharing the baked memory copy-on-write until it writes, so thirty renders don't cost thirty warm Chromes worth of RAM, and rootfs clones are reflinks rather than copies. A same-host fork lands in 400–750ms (cross-host is 1.2–3.5s), so a whole campaign's fan-out is seconds of setup.

from concurrent.futures import ThreadPoolExecutor
from pandastack import Sandbox

# 1. Warm ONE browser guest, then snapshot it. Creative-neutral: code and fonts only.
base = Sandbox.create(template="browser", ttl_seconds=1800)
base.exec("pip install websockets pillow", timeout_seconds=300)
base.filesystem.write("/work/render.py", RENDER_SRC)        # CDP driver + instrumentation
base.filesystem.write("/work/policy.json", POLICY_SRC)      # slot budgets, vendor allowlist
base.exec("python3 /work/render.py --warm about:blank", timeout_seconds=120)
snap = base.snapshot()
base.kill()

# 2. One fork per creative in the campaign. ~400-750ms same-host, copy-on-write.
def render_one(job) -> dict:
    fork = snap.fork(ttl_seconds=120)
    try:
        fork.filesystem.write("/work/creative.zip", job.bundle)
        fork.exec("cd /work && mkdir -p unit && unzip -o -q creative.zip -d unit",
                  timeout_seconds=30)
        r = fork.exec(
            f"cd /work && python3 render.py --dir unit "
            f"--width {job.w} --height {job.h} --shot shot.png --report report.json",
            timeout_seconds=90,
        )
        if r.exit_code != 0:
            return {"creative": job.id, "verdict": "error", "detail": r.stderr[-500:]}
        return build_verdict(
            job.id,
            fork.filesystem.read("/work/report.json"),
            fork.filesystem.read("/work/shot.png"),
        )
    finally:
        fork.kill()   # one hostile creative burns one fork, not the campaign

with ThreadPoolExecutor(max_workers=16) as pool:
    verdicts = list(pool.map(render_one, campaign_jobs))

rejected = [v for v in verdicts if v["verdict"] == "reject"]
print(f"{len(verdicts)} creatives rendered, {len(rejected)} rejected")

The isolation story survives the fan-out because every fork holds exactly one creative and the snapshot never holds any. Fan-out increases throughput; it never packs two advertisers into one guest. It also fixes your failure semantics: a creative that hangs the renderer is one fork that hits its TTL and gets killed, with the offending creative already identified, because it was the only advertiser input in that VM. Debugging a stuck render queue stops being archaeology.

The cost argument, briefly

The standard objection is that a VM per render costs more than a browser farm that's already running. The farm's real cost is that it's always on: you size it for peak creative-ingest bursts — lumpy, because campaigns launch on Mondays and quarter boundaries — pay for that capacity at 4am, and over-provision headroom specifically so one advertiser's memory-bomb can't starve everyone. Ephemeral microVMs invert it. Nothing runs between creatives, there's no warm pool to keep fed because a create is a snapshot restore (~49ms restore step, p50 179ms end to end), and density comes from copy-on-write: forks of one warmed-browser snapshot share memory pages until they diverge, and rootfs clones are reflinks. The practical ceiling is host RAM, not networking — with 16,384 pre-allocated subnets per agent you run out of memory long before you run out of network slots.

You cannot review malvertising safely in an environment you also want to keep. Render it somewhere you're happy to delete.

There is genuine operational cost here — snapshot hygiene, a job queue, plumbing artifacts back to the trusted side — and if you only render creatives your own team authored, you need none of it. But the moment strangers can upload code your infrastructure will execute and screenshot, you are running a hostile-input browser in production, and the only open question is what's in the room with it when a creative turns out to be exactly what someone paid for it to be.

For the adjacent mechanics: the general case for putting a browser in a VM instead of a container is in /blog/browser-isolation-microvm, the render-and-capture service pattern including warm snapshots is in /blog/microvm-headless-browser-screenshot-service, and the per-job network policy details are in /blog/controlling-network-egress-untrusted-code.