Running IoT and Embedded Firmware Emulation in Disposable MicroVMs

A firmware image is not a document. It is a filesystem, a kernel or a kernel's worth of modules, an init system, and a vendor's unexamined opinion about what should happen in the first three seconds of a device's life. When your pipeline "analyzes" one, what it actually does is unpack an archive format chosen in 2011, extract a filesystem image with a tool written to be forgiving rather than safe, and then boot somebody's init as root. Every one of those steps is code execution. Only the last one looks like it.

I'm Ajay; I built PandaStack. This post is about running firmware emulation jobs — the device-management SaaS that boots a customer's image before pushing it to ten thousand units, the security vendor diffing two OEM releases, the manufacturer regression-testing a build against a CVE corpus — inside disposable Firecracker microVMs. Along the way: why the unpackers are the scariest part of the pipeline and not the emulator, the honest performance story of running QEMU inside a microVM (it is TCG, and that is usually fine), how to give a phoning-home firmware a fake internet and turn its egress into a finding rather than an incident, and why snapshot-then-fork is the single highest-leverage move available for this workload.

What a firmware job actually is

Strip away the product framing and every firmware pipeline is the same five stages, executed against bytes that arrived from a customer upload form, a vendor FTP server, or a scraper pointed at a support-downloads page. Each stage has a different failure mode and a different reason to be somewhere expendable.

1. Identify — a signature scan over the blob to find where the compressed pieces start. This is mostly reads, and it is the only stage that is genuinely cheap and genuinely safe.
2. Carve and decompress — pull out the LZMA, gzip, XZ, or vendor-mangled payloads. This is where decompression bombs live, and where a header field claiming a 400 GB output causes a very fast, very rude allocation.
3. Extract the filesystem — squashfs, jffs2, cramfs, ubifs, romfs, yaffs2, and the LZMA-patched squashfs variants that only one forked tool understands. This is where path traversal and symlink escapes live.
4. Emulate — boot the extracted rootfs under a system emulator with a stand-in kernel, or run individual binaries under user-mode emulation. This is where the firmware's own intent finally gets to express itself.
5. Observe and extract findings — console output, syscall traces, attempted network egress, crashed services, hardcoded credentials found at runtime rather than by grep. This is the only stage whose output you want to keep.

Almost everyone builds this on a shared analysis host or a container on a CI runner, because stages one through three look like file processing and stage four looks like "we run QEMU, QEMU is a sandbox." Both of those beliefs are load-bearing and both of them are wrong in specific ways worth spelling out.

The unpackers are more dangerous than the emulator

People brace for the emulated firmware and relax during extraction. It should be the other way around. The emulator at least intends to be a boundary. The extraction toolchain is a pile of C from the embedded-Linux era, a couple of Perl-shaped heuristics, and a dispatcher that picks which of them to invoke based on bytes the image chose.

Extraction bombs, and why your upload limit means nothing

A firmware image is compressed by definition — that is the entire point of shipping it to a device with 8 MB of flash. So your size limit bounds the compressed artifact and tells you nothing about what comes out. A squashfs superblock carries a declared filesystem size, and a tool that trusts it will try to satisfy it. LZMA and XZ streams can be constructed with absurd expansion ratios out of nothing but zeros. And because firmware images legitimately nest — an image containing a filesystem containing a compressed update package containing another filesystem — a recursive extractor with no depth limit will happily follow that structure until the disk is gone or the process is.

The defense that actually works is not a smarter check, because determining whether an image will blow up an extractor generally requires running the extractor. The defense is to make the blowup hit a wall you chose: extract into a fixed-size filesystem so a bomb produces ENOSPC instead of a full host disk, and give the whole job a hard memory ceiling so an allocation from a lying header produces a dead process instead of a paging storm on a machine other people are using.

Path traversal and the symlink that eats your extraction root

The classic is an archive member named `../../etc/cron.d/backdoor`. GNU tar has refused those for a long time and Python's `tarfile` grew an extraction filter that rejects them, but the firmware world is not the tarball world — it is `unsquashfs`, `jefferson`, `ubireader`, `cramfsck`, and a handful of vendor forks, and "this extractor sanitizes member paths" is a per-tool question with a genuinely mixed answer. Squashfs tooling in particular has shipped directory-traversal bugs in living memory.

The subtler version needs no traversal at all. An archive contains a symlink named `etc` pointing at `/etc`, and then a regular file at `etc/passwd`. The extractor creates the link, then dutifully writes through it. Now you have written outside your extraction root using nothing but two perfectly ordinary members. The same trick with `usr/lib` and a `.so`, or with a path that resolves into the analysis tool's own plugin directory, converts "we extracted an image" into "the image chose what our next job would execute."

And please stop mounting them

The convenient shortcut — `mount -o loop -t squashfs image.sqfs /mnt` — is the one that should worry you most, because it hands attacker-controlled bytes directly to a kernel filesystem driver. Kernel filesystem parsers are not written on the assumption that the image is hostile; they are written on the assumption that the image came off a disk you own. Fuzzing filesystem drivers with malformed images is a well-trodden research area precisely because it works. On a shared host, mounting a customer-supplied jffs2 image is a kernel-surface interaction with everything else on that box.

Userspace extractors avoid the kernel driver and have their own bugs, which is a strictly better trade. But if your pipeline genuinely needs a mounted filesystem — some firmware only makes sense with real inodes and real permissions — then do the mount inside a guest whose kernel exists solely to be destroyed afterwards. That is not a workaround. That is the correct architecture for feeding a kernel parser bytes a stranger chose. I went through the general shape of this in /blog/microvm-fuzzing-harness-isolation.

A container is a polite suggestion to the kernel. A squashfs image is a set of instructions for a kernel parser. Putting those two facts in the same sentence is the whole argument.

#!/usr/bin/env bash
# Runs INSIDE the guest, on an image nobody has vouched for. Every step
# assumes the archive is hostile -- not because most vendors are, but
# because a meaningful fraction of real images are hostile by accident.
set -euo pipefail

# An unpacker has no legitimate reason to resolve a hostname, so make any
# library that tries fail fast and loudly rather than hang for 30s. This is
# only the inner ring -- the ring that actually HOLDS is the sandbox's own
# netns, egress-denied on the host, which nothing in here can renegotiate.
export http_proxy=http://127.0.0.1:1 https_proxy=http://127.0.0.1:1
export no_proxy=

# Fences BEFORE the first parse. A superblock with an absurd declared size
# gets to allocate long before you get a chance to validate anything.
ulimit -v 4194304    # 4 GiB address space -- LZMA bombs hit this wall
ulimit -t 1200       # 1200s CPU -- pathological decompressors get reaped
ulimit -u 256        # extractors that shell out, shelling out forever
ulimit -c 0          # no core dumps of a file we are about to destroy

# 1. Extract into a FIXED-SIZE filesystem. This is the only bomb defense
#    that does not require predicting the output size: the bomb gets
#    ENOSPC, the job fails loudly, and the guest disk is never involved.
truncate -s 6G /work/extract.img
mkfs.ext4 -q -F -m 0 /work/extract.img
mkdir -p /work/root
mount -o loop,nodev,nosuid,noexec /work/extract.img /work/root

# 2. Identify before extracting. A signature scan is a READ; the
#    extraction path is a dispatcher that invokes third-party tools by
#    heuristic, which is a much larger surface for a much smaller gain.
binwalk --signature --term /work/firmware.bin > /work/findings/signatures.txt

# 3. Extract as an unprivileged user, into a directory on that fixed
#    filesystem, with the extractor's own traversal guard on where it has
#    one. Note what is NOT here: no 'tar -P', no absolute paths, no
#    --keep-directory-symlink, no running any of this as root.
setpriv --reuid=1000 --regid=1000 --clear-groups \
  unsquashfs -no-xattrs -dest /work/root/squashfs-root /work/carved/rootfs.sqfs

# 4. Audit the result before ANY later stage reads it. A symlink pointing
#    out of the extraction root is not a corrupt image -- it is an image
#    that was written to make your next tool follow it somewhere useful.
find /work/root -xdev -type l -print0 |
  while IFS= read -r -d '' link; do
    target=$(readlink -- "$link")
    case "$target" in
      /*|*../*) printf 'escaping-symlink\t%s\t%s\n' "$link" "$target" >> /work/findings/extract.tsv ;;
    esac
  done

# 5. Never let the escapes reach the emulator's view of the world. Neuter
#    them in place and record that you did, because a symlink farm pointing
#    at /etc is itself a finding worth reporting to whoever shipped it.
sed -n 's/^escaping-symlink\t\([^\t]*\)\t.*/\1/p' /work/findings/extract.tsv |
  xargs -r -d '\n' rm -f --

The nesting problem: QEMU inside a microVM is TCG, and that's fine

Here is the part that everyone hand-waves, so let's be blunt about it. You want to boot an ARM or big-endian MIPS rootfs, and your analysis machine is x86. That means system emulation — `qemu-system-arm`, `qemu-system-mips`, occasionally `qemu-system-ppc` for the truly vintage. When you run that emulator on a bare Linux host, it still can't use KVM for a foreign architecture, so cross-architecture emulation is already software translation. But the reflex people have is "put it in a VM and it'll be slow because of nesting," and it's worth separating the two effects.

Firecracker does not expose nested virtualization to its guests. There is no usable `/dev/kvm` inside a Firecracker microVM, and there wouldn't be a usable one even if the device node appeared. So any emulator you run inside the guest runs in TCG — QEMU's tiny code generator, pure software translation, block-by-block. For cross-architecture firmware that changes nothing, because you were in TCG anyway. For the same-architecture case — an x86 firmware image, or an ARM image on an ARM host where you might have hoped for hardware acceleration — it is a real, honest cost, and you should size your budgets accordingly rather than pretend it away.

• Cross-architecture is a wash — an ARM or MIPS rootfs on an x86 analysis host is TCG whether or not there's a microVM in the picture. The nesting costs you nothing you weren't already paying.
• Same-architecture is a real tax — you give up hardware acceleration for the emulated guest. If your corpus is x86 UEFI images on x86 hosts, measure this before you commit; it may push you toward a different isolation shape for that specific subset.
• Emulated boots are slow in absolute terms regardless — a router firmware booting under TCG takes tens of seconds to minutes. Against that, a 179ms p50 sandbox create is noise. Provisioning latency is not the thing to optimize here.
• The right axis is width, not depth — you make the campaign fast by running four hundred images at once in four hundred guests, not by making one emulated boot 20% faster. This workload is embarrassingly parallel and the images don't need to talk to each other.
• User-mode emulation is the escape hatch — `qemu-arm-static` plus binfmt_misc plus a chroot runs individual firmware binaries far faster than full system emulation, and it's the right tool for "does this CGI handler crash on this input." It is the wrong tool for anything involving init, kernel modules, `/proc` contents the binary inspects, or ioctls on device nodes that don't exist. Know which question you're asking.

There's a second reason the nesting is worth accepting, and it's the one that decides the argument. QEMU is a large C program with an enormous device model, and you are pointing it at a disk image and a kernel command line derived from attacker-controlled bytes. Treating QEMU as your only isolation boundary means betting the analysis host on a VMM that has historically had escapes in exactly this kind of device code. Firecracker's device surface is deliberately tiny by comparison — a handful of virtio devices and no PCI enumeration circus — which is why it's the outer ring and QEMU is the inner one. /blog/firecracker-vs-qemu works through that difference in detail, and /blog/vm-escape-attacks-explained covers the failure mode itself.

Give it a fake internet and keep the receipts

Firmware phones home. Not occasionally — as a design principle. It checks for updates against a hardcoded hostname, registers with a cloud management service, syncs time against a vendor NTP pool, resolves a domain that expired in 2019 and is now owned by someone enterprising, and in the cases you most want to catch, connects to something that has no business being in a consumer router at all. Every one of those is a finding. The question is whether you find out by observing it in a lab or by having your analysis host make the connection on the vendor's behalf.

Default-deny egress is the only defensible posture, and it has to be enforced outside the emulated guest, because the emulated guest is running code you are specifically trying to characterize. On PandaStack each sandbox gets its own network namespace with its own veth pair and TAP device — each agent pre-allocates 16,384 /30 subnets — so the outer filtering is a property of the host, not a setting the firmware could be politely asked to respect. /blog/controlling-network-egress-untrusted-code covers the enforcement mechanics generally.

Blocking alone, though, throws away the interesting part. A firmware that fails to resolve its update host usually gives up quietly and you learn nothing. So build the fake internet one layer in: a DNS sinkhole and a catch-all recording responder running inside the same sandbox, on a bridge that only the emulated machine can see. The firmware resolves successfully, connects successfully, sends its registration payload — including, in my experience of looking at these, a device serial and a startlingly casual amount of identifying information — and every byte of it lands in a log file instead of on the internet.

#!/usr/bin/env bash
# A complete fake internet, entirely inside one sandbox. The firmware gets
# to succeed at everything it tries, which is the point: a blocked request
# tells you a request happened; a SERVED request tells you what it said.
set -euo pipefail
mkdir -p /work/findings

# A bridge the emulated machine talks to and nothing else routes off of.
ip link add br-lab type bridge
ip addr add 10.77.0.1/24 dev br-lab
ip link set br-lab up
ip tuntap add dev lab0 mode tap
ip link set lab0 master br-lab up

# Every name resolves to us. The query log IS a deliverable: "this device
# contacts three hosts on first boot, one of which is not the vendor's."
cat > /etc/dnsmasq.conf <<'EOF'
interface=br-lab
bind-interfaces
no-resolv
address=/#/10.77.0.1
dhcp-range=10.77.0.50,10.77.0.150,12h
log-queries
log-facility=/work/findings/dns.log
EOF
dnsmasq --conf-file=/etc/dnsmasq.conf

# Catch-all TCP. Whatever port the firmware picked -- 80, 443, 8443, or
# 7547 because it speaks TR-069 -- it lands on the recorder, which peeks
# the first bytes to decide whether to speak TLS or plaintext.
iptables -t nat -A PREROUTING -i br-lab -p tcp -j REDIRECT --to-ports 8080
iptables -A FORWARD -i br-lab -j DROP    # no route out of the lab. ever.

python3 /opt/lab/record_responder.py \
  --listen 10.77.0.1:8080 \
  --tls-cert /opt/lab/lab-ca.pem \
  --log /work/findings/egress.ndjson &

# No -enable-kvm, and not because we forgot: there is no usable /dev/kvm
# inside a Firecracker guest. This is TCG. A MIPS rootfs was going to be
# TCG on an x86 host anyway, so the nesting costs us nothing here.
qemu-system-mips -M malta -m 256 -nographic \
  -kernel /opt/lab/vmlinux-mipseb-malta \
  -drive file=/work/extract.img,format=raw,if=ide \
  -netdev tap,id=n0,ifname=lab0,script=no,downscript=no \
  -device e1000,netdev=n0 \
  -append "root=/dev/sda1 console=ttyS0 rw rdinit=/sbin/preinit" \
  2>&1 | tee /work/findings/console.log

• Record the DNS queries, not just the connections — a name that never resolves still tells you the firmware wanted it, and an expired domain in a shipping product is a supply-chain finding all by itself.
• Record full request bodies — registration payloads are where the device serial, the MAC, the WiFi SSID, and occasionally a support credential go. Grep found the hardcoded string; the lab found what it's actually used for.
• Answer plausibly, not perfectly — a bland 200 with an empty JSON body gets most firmware past its check-in and into the interesting part of boot. You are not building a vendor cloud emulator, you are building enough of one to keep init moving.
• Log the failures too — a firmware that tries TLS with certificate pinning and gives up is telling you something useful about its update path, and that's a finding rather than a broken lab.
• Keep the sinkhole inside the sandbox — running it as shared lab infrastructure means one job's firmware can see another job's traffic, which quietly destroys the attribution that made per-image isolation worth doing.

Snapshot after the boot, fork per test case

This is the part that changes the economics, and it follows directly from the fact that emulated boots are slow. Getting a firmware image from "bytes on disk" to "a running system with a web UI listening on port 80" costs you the unpack, the peripheral shimming, the TCG boot, and usually a few minutes of watching a serial console for the login prompt. Then you want to run four hundred test cases against it: a CVE probe corpus, a fuzz campaign against the CGI handlers, every permutation of a config parameter, a diff between two firmware versions under identical stimulus.

Doing that serially means booting once and hoping nothing you do corrupts the state for the next case, which it will — the third test case wedges a daemon and cases four through four hundred are now testing a different machine than case one did. Doing it in parallel by booting four hundred times means paying the expensive part four hundred times.

The move is to snapshot the outer microVM once the emulated firmware is up, then fork it per test case. This works better than it has any right to, because a Firecracker snapshot captures the entire guest — all of its RAM and device state — and the QEMU process with the emulated machine's memory inside it is just part of that RAM. You snapshot the analysis machine and get the emulated machine for free, mid-boot-completion, with the sinkhole running and the bridge up. A same-host fork lands in 400–750ms; cross-host is 1.2–3.5s because memory has to travel. Against a boot measured in minutes, that is a rounding error. /blog/snapshot-and-fork-explained and /blog/copy-on-write-memory-fork-explained cover the mechanics.

import json
from concurrent.futures import ThreadPoolExecutor
from pandastack import Sandbox

# "firmware-lab" is a template baked on top of PandaStack's base image with
# the emulators, the extractors, the stub kernels, and the NVRAM shim in it.
# The first-party templates (base, code-interpreter, agent, browser,
# postgres-16) are a starting point, not the menu.

def probe_campaign(image_sha: str, image_bytes: bytes, cases: list[dict]) -> list[dict]:
    """Boot ONE emulated firmware, then branch the world per test case.

    The expensive part of this workload is the emulated boot -- minutes of
    TCG, not milliseconds of provisioning. So we pay it once.
    """
    # ---- 1. One guest pays for unpack + peripheral shims + emulated boot ----
    base = Sandbox.create(
        template="firmware-lab",
        ttl_seconds=7200,
        metadata={"image": image_sha, "phase": "warm", "kind": "firmware-emulation"},
    )
    base.filesystem.write("/work/firmware.bin", image_bytes)
    base.exec("bash /opt/lab/unpack.sh", timeout_seconds=1800)
    base.exec("bash /opt/lab/fake-internet.sh", timeout_seconds=120)
    boot = base.exec(
        "bash /opt/lab/boot.sh --wait-for http://10.77.0.50:80/ --deadline 900",
        timeout_seconds=1000,
    )
    if boot.exit_code != 0:
        base.kill()
        raise FirmwareDidNotBoot(image_sha, boot.stdout[-8000:])

    # The snapshot captures the ANALYSIS guest's RAM -- which contains the
    # QEMU process -- which contains the emulated machine's RAM. One capture,
    # both layers, post-boot, with the sinkhole already listening.
    snap = base.snapshot()
    base.kill()

    # ---- 2. One fork per test case. Each starts from the same booted state ----
    def run_case(case: dict) -> dict:
        fork = snap.fork(ttl_seconds=900)
        try:
            fork.filesystem.write("/work/case.json", json.dumps(case, sort_keys=True))
            out = fork.exec(
                "python3 /opt/lab/drive_case.py /work/case.json "
                "--out /work/findings/case.json",
                timeout_seconds=600,
            )
            # A wedged daemon, a guest kernel panic, or a fork that hangs
            # costs exactly one machine. Case 4 does not inherit case 3's
            # damage, which is the entire reason this is not a for-loop.
            if out.exit_code != 0:
                return {"case": case["id"], "status": "error", "log": out.stderr[-4000:]}
            return json.loads(fork.filesystem.read("/work/findings/case.json"))
        finally:
            fork.kill()

    with ThreadPoolExecutor(max_workers=32) as pool:
        return list(pool.map(run_case, cases))

The property that makes this scientifically useful rather than merely fast: every fork started from a byte-identical machine. Not a machine built from the same script — the same machine, same heap layout, same daemon PIDs, same emulated RAM contents. When case 217 finds a crash and case 216 doesn't, the difference is case 217's input and nothing else. That is a much stronger claim than any pipeline that reboots between cases can make.

A finding you can't reproduce is not a finding

If your output is a security report, reproducibility is not a nice-to-have — it is the deliverable. "We observed a stack overflow in the UPnP handler" is a sentence a vendor will push back on, and the pushback is always the same: it doesn't reproduce here. If your answer is a shell script and a hope, you lose that argument regardless of whether you were right.

• Pin the emulator, not just the image — QEMU's TCG output, device model defaults, and machine-type behaviour all change across versions. `-M virt` in one release is not `-M virt` in the next. The emulator version belongs in the finding.
• Pin the stub kernel — you are booting the rootfs against a kernel you supplied, so which kernel, which config, and which modules are part of the result. A crash that only happens under your kernel and not the vendor's is a finding about your lab.
• Pin the shims — the NVRAM values your `LD_PRELOAD` returns are inputs. A firmware that behaves differently when `wan_proto` is `dhcp` versus `pppoe` is behaving correctly, and you need to know which one you gave it.
• Record the fake internet's behaviour — what the sinkhole answered is part of the stimulus. If the recorder returned 200 with an empty body and a later run returns 404, you have changed the experiment.
• Record the snapshot generation — this is the field nobody stores and the only one that makes the rest restorable. "ubuntu-latest with binwalk" is not an environment; a snapshot identifier is.
• Freeze or sync the clock deliberately — never leave it ambient. See the callout below, because this one has a sharp edge specific to snapshots.

Snapshot-restore does most of this work for you as a side effect of how it operates. A PandaStack template's first spawn is a genuine cold boot of around 3 seconds; every create afterward restores that baked snapshot — roughly 49ms for the restore step, 179ms p50 end to end and about 203ms p99. Every job therefore starts from a machine that is identical down to the page: same emulator build, same stub kernels, same extractor versions, same libc. Re-running last quarter's analysis is a restore, not an archaeology project. /blog/reproducible-build-sandboxes covers the build half of that story.

Stage the findings, and give the guest nothing worth stealing

The last structural decision is about what the analysis guest is allowed to hold. The tempting design has the guest write results straight into your findings database, push artifacts to object storage, and post a status to your queue — three credentials, in a machine that just booted an untrusted init as root and connected it to a network stack you built for the occasion.

Invert it. The guest is pure: bytes in, a staged findings artifact out, no credential of any kind. The orchestrator holds the database handle, validates the artifact as a whole, and commits it. This is the same discipline that makes import pipelines safe, and I worked through it at length in /blog/microvm-per-tenant-csv-import-pipeline-isolation. Here it buys you something extra: because the guest cannot write anywhere durable, a partially-completed analysis produces nothing at all rather than half a report with no marker saying so.

from pandastack import Sandbox
import json, hashlib

def analyze_firmware(vendor: str, image_bytes: bytes, shims: dict) -> dict:
    """One image, one machine, one report. The machine holds no secrets.

    Note what does NOT go into this guest: the findings database DSN, the
    artifact bucket credential, the queue token, any other customer's image.
    It gets bytes and a shim config, and it hands back a JSON file.
    """
    image_sha = hashlib.sha256(image_bytes).hexdigest()

    sbx = Sandbox.create(
        template="firmware-lab",
        ttl_seconds=3600,          # a firmware that reboot-loops forever is
                                   # a real outcome; it must not be a resident
        metadata={"vendor": vendor, "image": image_sha[:16], "kind": "firmware-emulation"},
    )
    try:
        sbx.filesystem.write("/work/firmware.bin", image_bytes)
        # Shims are INPUTS, recorded in the report: an image behaves
        # differently with wan_proto=pppoe and that is not a bug.
        sbx.filesystem.write("/work/shims.json", json.dumps(shims, sort_keys=True))

        out = sbx.exec(
            "cd /work && bash /opt/lab/unpack.sh "
            "&& bash /opt/lab/fake-internet.sh "
            "&& bash /opt/lab/boot.sh --shims shims.json --deadline 900 "
            "&& python3 /opt/lab/collect.py --out /work/findings/report.json",
            timeout_seconds=2400,
        )

        # FAIL CLOSED. An extraction bomb hitting ENOSPC, a segfaulting
        # unpacker, a firmware that never reaches init, a TTL expiry -- all
        # land here, and all produce zero rows in the findings table rather
        # than a half-written report that looks like a clean bill of health.
        if out.exit_code != 0:
            raise FirmwareAnalysisFailed(image_sha, out.exit_code, out.stderr[-8000:])

        report = json.loads(sbx.filesystem.read("/work/findings/report.json"))
        report["egress"] = sbx.filesystem.read("/work/findings/egress.ndjson")
        report["dns"] = sbx.filesystem.read("/work/findings/dns.log")
        report["console"] = sbx.filesystem.read("/work/findings/console.log")
        return report
    finally:
        # The image, the extracted rootfs, the symlink farm, the emulated
        # machine, and whatever the firmware decided to install in the first
        # three seconds of its life all cease to exist together.
        sbx.kill()

Shared host vs container vs VM-per-job vs microVM-per-job

• Extraction bombs — Shared analysis host: a nested LZMA bomb fills the disk everyone's jobs write to, and the OOM killer resolves the memory half by heuristic, which may reap a neighbour's run. Container: cgroups bound memory well and a volume quota bounds disk, but pressure still reaches the shared host kernel. VM per job: a real fence, at the cost of a boot you pay for every image. MicroVM per job: fixed vCPU/RAM baked into the guest, a fixed-size extraction filesystem inside it, and a hard TTL, so a bomb exhausts a machine that existed for one image.
• Path traversal and symlink escape — Shared analysis host: an archive member that writes through a symlink lands in a shared filesystem where the next job, or the tooling itself, will read it. Container: contained to the container's filesystem, which is genuinely most of the win, though bind mounts and shared caches are the usual leak. VM per job: contained. MicroVM per job: contained, and the guest is destroyed afterward, so a planted file has nothing to persist into.
• Kernel blast radius — Shared analysis host: mounting a customer-supplied jffs2 or squashfs image feeds attacker-controlled bytes to your host's filesystem driver, with everything else on the box downstream of that. Container: the same kernel, so the same exposure — namespaces do not stand between an image and a filesystem parser. VM per job: own kernel, correct. MicroVM per job: own kernel plus a deliberately minimal virtio device surface, so the outer VMM is a much smaller target than the emulator running inside it.
• Egress control — Shared analysis host: whatever the host firewall says, and the analysis box's rules are always generous because somebody needed to fetch a vendor image in 2023. Container: per-container policy is real and workable, usually at a coarser granularity than one image. VM per job: strong, if you're willing to build the per-VM networking. MicroVM per job: per-sandbox netns with its own veth and TAP, filtered on the host — 16,384 pre-allocated /30 subnets per PandaStack agent — with the sinkhole and recorder living inside, so egress is a logged finding rather than a real connection.
• Reproducibility of a finding — Shared analysis host: the box drifts with every package upgrade, so last quarter's crash is unreproducible for reasons unrelated to the bug. Container: image tags help enormously, but the host kernel underneath is whatever the fleet runs today, and this workload cares about the kernel. VM per job: reproducible if you keep the disk image, which people don't. MicroVM per job: every create restores a pinned snapshot generation, so "the machine that found this" is an identifier you can hand to the vendor.
• Fan-out cost — Shared analysis host: you shard by hand and campaigns end up serial. Container: cheap to start, but every worker repeats the unpack and the emulated boot. VM per job: a full boot per image, which is exactly the cost you were trying to avoid. MicroVM per job: boot one emulated firmware, snapshot it, fork per test case at 400–750ms same-host, so every worker resumes from a running system instead of building one.
• Nested emulation performance — Shared analysis host: no nesting tax, and a same-architecture image can use hardware acceleration. Container: same, since there's no hypervisor in the way. VM per job: TCG unless the platform exposes nested virtualization, which most managed ones do not. MicroVM per job: TCG, full stop — no usable /dev/kvm in the guest. For the cross-architecture firmware that dominates this workload, that costs nothing you weren't already paying; for same-architecture images it is a genuine tax you should measure rather than assume away.

The container and VM columns describe general architectural properties rather than benchmark results, and every project in that space moves quickly — verify current behaviour and limits against their own documentation before making an infrastructure decision on my description of them. The only measured numbers here are PandaStack's. /blog/firecracker-vs-kata-vs-gvisor works through the isolation-model comparison in more depth.

The summary

Firmware analysis has an unusual property: the dangerous part is not where people look. Everyone braces for the emulated device and relaxes during extraction, when in fact the extraction toolchain is decades-old C invoked by heuristic against bytes chosen by whoever built the image, and the shortcut everyone takes — loop-mounting the filesystem — hands those bytes straight to a kernel driver. Do that on a shared analysis host and a hostile image gets a vote on what happens to every other job on the box.

One disposable microVM per image handles the whole family at once. Extraction bombs exhaust a fixed-size filesystem in a machine with a baked memory ceiling and a hard TTL. Traversal and symlink escapes land in a rootfs that ceases to exist minutes later. Kernel filesystem parsing happens in a guest kernel that exists to be destroyed. The firmware's phone-home succeeds against a sinkhole and a recorder that live inside the same sandbox, so the registration payload becomes a line in a findings file instead of a packet on your uplink. And the guest holds no credentials, so a run that dies halfway produces nothing rather than a half-written report shaped exactly like a clean result.

Be honest about the nesting: QEMU inside a Firecracker guest is TCG, because there is no usable /dev/kvm in there. For the ARM and big-endian MIPS images that make up most of this work on x86 hosts, you were in TCG regardless and the microVM is free. For same-architecture images it's a real cost. Either way, the way you make a firmware campaign fast is not by making one emulated boot quicker — it's by booting once, snapshotting, and forking per test case at 400–750ms, then running four hundred images at a time because provisioning is a fifth of a second in front of a job measured in minutes.

For adjacent patterns: the fuzzing side of this workload is in /blog/microvm-fuzzing-harness-isolation, the egress enforcement mechanics in /blog/controlling-network-egress-untrusted-code, the fork primitive in /blog/copy-on-write-memory-fork-explained, and the case for a small VMM under a big emulator in /blog/firecracker-vs-qemu.