Detonating malware in Firecracker microVMs

Almost every workload in this industry involves running code you'd rather trust. Malware detonation is the one where you don't even pretend. You take a file that a stranger emailed to your customer, you place it on a machine, and you deliberately run it — knowing that it was written by someone whose explicit goal is to get out of whatever box you put it in, steal what's nearby, and phone home about it. It is the only category of computing where the input is adversarial by definition rather than by accident.

That framing decides your architecture. If the sample is designed to escape, the boundary isn't a hardening detail you tune later — it is the product. Everything else in a detonation pipeline (behavioural signatures, YARA on the memory dump, pretty process-tree graphs in your SOC console) is downstream of one question: when the sample does something creative, where does it end up?

I'm Ajay — I build PandaStack, which runs Firecracker microVMs as a service, so read this knowing I have a side. I'll be specific about where microVMs are the right answer for this workload, and equally specific about the place they're genuinely weaker than a fat, boring, fingerprint-rich analysis VM: evasion. Skip to that section if you only came for the honest part.

Why a container is the wrong boundary here

Containers are a fantastic packaging and resource-isolation mechanism, and a poor security boundary against a determined attacker. That's not a controversial claim; it follows from the design. A container shares the host kernel. Namespaces and cgroups restrict what a process can see and consume, but every syscall the container makes lands in the same kernel that runs your analysis host, your queue worker, and the other twelve samples you're detonating in parallel.

The Linux kernel syscall surface is enormous, and kernel LPE bugs are a steady, recurring genre — not a freak event. For most workloads that's an acceptable risk: your code isn't hunting for them. In detonation, your input is a program whose job description includes hunting for them. You have inverted the assumption that makes container isolation reasonable.

There's a subtler problem too, and it bites long before anyone escapes anything: cleanliness. "Fresh container from the same image" is not the same as "fresh machine." Kernel state, page cache, shared mounts, entropy, whatever a previous sample poked in a sysctl or a shared tmpfs — the boundary you reset is the process tree, not the machine. When an analyst asks "did the last sample leave something behind that changed this result?", the honest answer on a shared kernel is "probably not, but I can't prove it." That answer is worth nothing in an incident review.

What a hardware-virtualized microVM actually changes

A Firecracker microVM is a real VM: its own guest kernel, its own memory, running under KVM with hardware virtualization extensions. The sample's syscalls go to the guest kernel, and the guest kernel is disposable. To reach your host it needs a VMM or hypervisor escape, not a Linux kernel LPE — a much smaller and much better-audited target.

And Firecracker's device model is deliberately tiny: virtio-net, virtio-block, virtio-vsock, a serial console, an entropy source, and a keyboard controller that exists mostly to accept a reboot. No PCI bus, no BIOS, no bootloader, no emulated graphics, no USB, no sound, no SATA. Historically, an enormous share of hypervisor CVEs live in device emulation — the SVGA adapters, the audio controllers, the floppy controllers nobody has used since the Clinton administration. Firecracker's answer is to not have them. On top of that sits the jailer (chroot, namespaces, cgroups, dropped privileges) and a seccomp filter restricting the VMM process itself.

For detonation specifically, that produces three properties worth naming.

• The boundary is enforced by hardware, not by a filter list. You are not maintaining a seccomp profile that tries to anticipate which of 400 syscalls the malware will pick.
• The escape surface is small enough to reason about. "Which emulated devices can this sample reach?" has a short, enumerable answer.
• Destruction is real. Ending an analysis is unlinking a memory file and a disk image, not attempting to clean a machine that a hostile program had root on.

The point of a disposable VM isn't that malware can't hurt it. It's that you were always going to throw it away, so it doesn't matter if it does.

Snapshot-restore: a known-good machine for every single sample

This is the part that changes the shape of the pipeline, not just its safety story. The classic approach is a golden VM image plus "revert to snapshot" between samples — correct, but slow enough that you batch samples, reuse VMs, or cut corners under queue pressure. Firecracker's snapshot model is different in degree to the point of being different in kind: you boot the analysis machine once, install your tooling, let it settle into a plausible steady state, then freeze it to a memory file plus a state file. Every subsequent analysis starts by restoring that frozen machine.

On PandaStack, creating a sandbox this way is p50 179ms and p99 around 203ms end to end, of which the restore step itself is about 49ms. A first-ever cold boot, before a snapshot exists, is roughly 3 seconds — you pay that once per template, not once per sample. The restored memory is mapped copy-on-write and the rootfs is a reflinked clone, so the golden copy is never written to; the sample scribbles on pages that belong to a machine that will be deleted in thirty seconds.

The operational consequence is that "one pristine machine per sample" stops being a policy you enforce and becomes the cheapest thing to do. Nobody reuses a VM to save time when a fresh one costs a fifth of a second. The question "did the previous sample leave something behind?" stops being a judgement call and becomes structurally unanswerable in the good way: there was no previous sample on this machine, because this machine is 200 milliseconds old.

# ---------------------------------------------------------------
# Bake the analysis machine ONCE. Boot it, install your tooling,
# create the persona (files, history, a user who looks real), let
# it idle until it stops being a machine that just started, then
# freeze it. This snapshot is your known-good state, forever.
# ---------------------------------------------------------------
curl -s --unix-socket "$SOCK" -X PATCH http://localhost/vm \
  -H 'Content-Type: application/json' -d '{"state":"Paused"}'

curl -s --unix-socket "$SOCK" -X PUT http://localhost/snapshot/create \
  -H 'Content-Type: application/json' -d '{
    "snapshot_type": "Full",
    "snapshot_path": "./golden/det.state",
    "mem_file_path": "./golden/det.mem"
  }'

# ---------------------------------------------------------------
# Per sample: clone the disk copy-on-write and restore the frozen
# machine. Firecracker expects its backing files where the snapshot
# left them, so the reflinked clone is placed at that path inside
# this run's jail -- the golden image is never opened for writing.
# ---------------------------------------------------------------
ID=$(uuidgen)
JAIL="/srv/det/runs/$ID"
mkdir -p "$JAIL"
cp --reflink=always ./golden/rootfs.ext4 "$JAIL/rootfs.ext4"   # O(metadata)

curl -s --unix-socket "$JAIL/fc.sock" -X PUT http://localhost/snapshot/load \
  -H 'Content-Type: application/json' -d '{
    "snapshot_path": "/golden/det.state",
    "mem_backend": { "backend_type": "File", "backend_path": "/golden/det.mem" },
    "enable_diff_snapshots": false,
    "resume_vm": true
  }'

# ... thirty seconds of someone else's software happening ...

# Teardown is not "clean up". It is unlink. Nothing survives that a
# root-level process inside the guest could have hidden in.
kill "$FC_PID" 2>/dev/null
rm -rf "$JAIL"

The network: give it an internet, just not ours

Two bad options present themselves and both are popular. Cut the network entirely and most interesting samples do nothing — they resolve a domain, fail, and exit, and you learn that the file is 400KB. Give it real internet and you have volunteered your IP space to participate in a botnet, notified the operator that their sample is being analysed, and possibly fetched a second stage you now have to explain to legal.

The right answer is a convincing fake. The sample gets DNS that resolves, TCP that connects, HTTP that returns 200, SMTP that accepts mail — none of it leaving your rack. This is where per-sandbox network namespaces earn their keep: each detonation lives in its own netns with its own veth pair and tap device, so "no route to the real world" is a property of that namespace rather than a rule you hope is ordered correctly in a shared table. PandaStack pre-allocates 16,384 /30 subnets per agent for exactly this pattern, which is also why parallel fan-out doesn't turn into an IP-management project.

# Everything here runs on the analysis host, inside this sandbox's own
# network namespace. The guest's entire universe is what we serve it.
NS=ns-det-$ID

# 1. Fail closed first, then poke holes. Default-deny in the namespace
#    means a misordered rule is a broken analysis, not a live C2 channel.
ip netns exec "$NS" nft -f - <<'EOF'
table inet det {
  chain input   { type filter hook input   priority 0; policy drop; }
  chain output  { type filter hook output  priority 0; policy drop; }
  chain forward { type filter hook forward priority 0; policy drop; }
}
EOF

# 2. Answer every DNS question with the sinkhole, and LOG the questions.
#    The queried domains are frequently the single best IOC in the run --
#    a sample that never gets past name resolution still tells you who
#    it wanted to call.
ip netns exec "$NS" dnsmasq \
  --address=/#/10.200.7.2 \
  --no-resolv --bind-interfaces --listen-address=10.200.7.2 \
  --log-queries --log-facility=/var/log/det/$ID/dns.log

# 3. Give it something to talk to. A fake-services responder (INetSim,
#    FakeNet-NG, or your own) speaks HTTP/HTTPS/SMTP/IRC at the sinkhole
#    so the sample proceeds past check-in instead of exiting at step one.
ip netns exec "$NS" inetsim --config /etc/inetsim/detonate.conf &

# 4. Capture on the HOST side of the veth. The guest cannot see this
#    process, cannot kill it, and cannot tell it is being recorded.
tcpdump -i "vh-det-$ID" -s0 -U -w "/var/log/det/$ID/capture.pcap" &

Collecting artifacts before you nuke the VM

The whole point of a disposable machine is that you delete it — which means every question you'll want answered later has to be answered before teardown. In practice a useful run produces four things: what changed on disk, what the process tree looked like at peak, what went over the wire, and what was in memory. The first three are cheap; the fourth is where the good indicators hide, because packed samples unpack themselves for you as a professional courtesy.

Here's the loop with the PandaStack Python SDK. Note the ordering: mark time before delivering the sample, start capture before executing it, and extract everything before the context manager destroys the machine.

from concurrent.futures import ThreadPoolExecutor
from pandastack import Sandbox

DETONATION_SECONDS = 30


def detonate(sample_path: str) -> dict:
    """Run one sample on a machine that will not exist in ten minutes."""
    sample = open(sample_path, "rb").read()

    # ttl_seconds is the backstop: if this worker dies mid-analysis, the VM
    # reaps itself instead of quietly staying up with live malware in it.
    with Sandbox.create(template="base", ttl_seconds=600) as sbx:
        # 1. Mark time. Anything newer than this marker is the sample's doing.
        sbx.exec("mkdir -p /det && touch /det/.mark", timeout_seconds=15)

        # 2. Start in-guest capture (convenience; the host-side pcap on the
        #    veth is the evidence you'd actually put in a report).
        sbx.exec(
            "nohup tcpdump -i any -s0 -U -w /det/guest.pcap >/dev/null 2>&1 &",
            timeout_seconds=15,
        )

        # 3. Deliver the sample. It's inert bytes until something runs it,
        #    and the only thing that will run it lives inside this VM.
        sbx.filesystem.write("/det/sample.bin", sample)

        # 4. Detonate. The timeout is not decorative -- a meaningful share of
        #    samples are just an infinite loop wearing a trench coat.
        run = sbx.exec(
            "cd /det && chmod +x sample.bin && ./sample.bin",
            timeout_seconds=DETONATION_SECONDS,
        )

        # 5. Observe: process tree, listeners, and every path touched since
        #    the marker. This is the filesystem diff, done the cheap way.
        procs = sbx.exec(
            "ps -eo pid,ppid,user,etimes,args --forest", timeout_seconds=15
        )
        socks = sbx.exec("ss -tunap || netstat -tunap", timeout_seconds=15)
        diff = sbx.exec(
            "find / -xdev -newer /det/.mark -not -path '/det/*' "
            "-not -path '/proc/*' -not -path '/sys/*' -not -path '/run/*' "
            "2>/dev/null | head -500",
            timeout_seconds=90,
        )

        # 6. Extract evidence BEFORE the machine stops existing.
        sbx.exec("pkill -INT tcpdump || true", timeout_seconds=15)
        guest_pcap = sbx.filesystem.read("/det/guest.pcap")

        return {
            "exit_code": run.exit_code,
            "stdout": run.stdout[-8000:],
            "stderr": run.stderr[-8000:],
            "process_tree": procs.stdout,
            "sockets": socks.stdout,
            "files_touched": diff.stdout.splitlines(),
            "guest_pcap": guest_pcap,
        }
    # Block exit destroys the VM. Not "cleans" it. Destroys it.


def detonate_queue(paths: list[str], workers: int = 16) -> list[dict]:
    """One sample per VM, many VMs at once. They share nothing but a host."""
    with ThreadPoolExecutor(max_workers=workers) as pool:
        return list(pool.map(detonate, paths))


# When an analyst wants to poke at a live machine by hand, skip the context
# manager -- but be explicit about the ending. Someone always forgets.
sbx = Sandbox.create(template="base", ttl_seconds=1800)
try:
    sbx.filesystem.write("/det/sample.bin", open("samples/x.bin", "rb").read())
    print(sbx.exec("file /det/sample.bin", timeout_seconds=15).stdout)
finally:
    sbx.kill()

A `find -newer` sweep is the pragmatic 80% of a filesystem diff and costs nothing. If you want the rigorous version, snapshot the VM mid-run instead: the memory file is a full RAM image you can carry off to Volatility, and because the rootfs clone is a separate file you can mount it read-only from the host afterwards and diff it against the golden image offline — a comparison the sample never had the chance to interfere with, because it happens on a machine it never ran on.

Container vs microVM vs bare-metal reimage

Three architectures people genuinely run for this. The PandaStack timings below are our measured numbers; treat any claim about other tools as "check their docs," because these products all move.

• Isolation boundary — Container: shared host kernel, so a kernel LPE is a host compromise; the syscall surface is exactly the surface the sample is looking for. MicroVM: hardware virtualization plus a minimal virtio-only device model, so escape requires a VMM/hypervisor bug rather than a Linux kernel bug. Bare-metal reimage: no shared kernel at all, but firmware, UEFI, and device firmware are persistent state a sufficiently nasty sample can target.
• Time to a clean machine — Container: milliseconds, but "clean" only covers the process tree, not the kernel. MicroVM: snapshot-restore is p50 179ms / p99 ~203ms on PandaStack (restore step ~49ms), giving a genuinely fresh guest kernel per sample. Bare-metal reimage: minutes to tens of minutes, which is why teams reuse hosts between samples and then argue about whether that mattered.
• Confidence that nothing carried over — Container: low; kernel state, page cache, and shared mounts persist. MicroVM: high; the guest kernel and all guest memory are new, restored from an immutable snapshot with copy-on-write memory and a reflinked disk. Bare-metal reimage: high for disk and RAM, weaker for firmware unless you also flash it, which nobody does per sample.
• Realism to an evasion-aware sample — Container: obviously not a machine; trivially detected. MicroVM: clearly virtual and quite distinctive — few devices, no PCI, no firmware tables to lie in. Bare-metal reimage: maximum realism, which is exactly why it stays in the toolkit for the evasive minority.
• Cost and density per sample — Container: cheapest, and the reason people talk themselves into it. MicroVM: a few MB of VMM overhead per guest with CoW memory sharing across guests, so hundreds in flight on one host is an ordinary Tuesday. Bare-metal reimage: one sample per physical machine per reimage cycle — the most expensive analysis you can buy.
• Operational blast radius — Container: an escape puts the attacker on the box running your queue and your other analyses. MicroVM: an escape puts them in a jailed, seccomp-filtered VMM process on a host you should already be treating as disposable. Bare-metal reimage: an escape owns a machine you were about to wipe anyway — as long as it doesn't reach firmware or your management network.

The honest part: minimal microVMs are extremely fingerprintable

Everything that makes Firecracker a good boundary makes it an obvious one. Malware has checked for virtualization for two decades, and a microVM fails many of those checks loudly. There's no BIOS or DMI/SMBIOS tables full of plausible vendor strings, because there's no firmware. There's no PCI bus to enumerate. There's no GPU, no USB controller, no sound card, no SATA disk — the device list is short and unusual enough to be a signature by itself. The CPUID hypervisor bit is set. The MAC address OUI, the disk size, the hostname, and the machine-id are whatever your bake produced, and if you bake once and restore ten thousand times, they're identical across ten thousand analyses unless you go out of your way to randomize them.

Snapshot-restore adds its own tells. A restored guest has an implausibly low uptime, a suspiciously empty process history, and — a specific gotcha we hit in production — a wall clock frozen at bake time until something re-syncs it. "System time is three weeks stale and uptime is four seconds" is not a subtle signal. There's also the human dimension: no browser history, no documents in Downloads, no recently-used files, no mouse having ever moved. A sample that waits for user activity before doing anything real will simply wait, and you'll record a clean verdict on a file that isn't.

You can push back on most of this, and it's worth doing, but it's an arms race and you should budget for it as one.

1. Bake a persona, not a blank box: a real-looking user account, documents, browser profile and history, cached credentials-shaped junk, installed software with plausible timestamps. All of it lives in the snapshot, so it costs nothing per sample.
2. Randomize the identity on restore: MAC, hostname, machine-id, and volume serial. Identical fingerprints across every analysis are a gift to anyone building a sandbox blocklist.
3. Sync the clock immediately on restore — before the sample runs. This also stops TLS from breaking against certificates issued after your bake date, which is its own separate headache.
4. Simulate activity: synthetic input events, a script that opens files and moves windows, some background network chatter. Cheap tricks that defeat cheap checks.
5. Vary the run: jitter the detonation window, occasionally run much longer, and detonate a subset on a heavier, more realistic full-device VM to catch samples that go dormant on the fast path.

The verdict I'd actually defend: use microVMs for throughput. The overwhelming majority of what lands in a detonation queue is commodity — phishing attachments, macro droppers, recycled loaders, the same packer for the ninth time this week — and for that population, thousands of pristine, isolated, cheap-to-destroy machines per hour is a straightforwardly better pipeline than anything else on the list. Then escalate. Samples that exit suspiciously fast, do nothing at all, or trip your anti-analysis heuristics get promoted to a slower, richer, more convincingly "real" environment. A fast triage tier and a small deep tier beats forcing one environment to be both.

Fan-out: a queue of samples is an embarrassingly parallel problem

Detonation is nearly the ideal parallel workload. Samples are independent, each run is bounded (usually 30 to 300 seconds of mostly waiting), and the correct isolation model — one machine per sample — is also the correct concurrency model. What historically prevented teams from just running all of them at once was that VMs were expensive to create and expensive to keep around.

With snapshot-restore, creation stops being the bottleneck; memory does, which is a much more honest constraint you can buy your way out of. A worker pool that restores a fresh machine per sample, runs it, harvests artifacts, and destroys it will keep a host saturated without any warm pool of idle VMs sitting around costing money between spikes. Pre-allocated per-sandbox networking matters here too — building a netns, veth pair, tap device, and firewall rules from scratch costs on the order of 100ms, which would dominate a 179ms create if you did it cold every time.

Two more primitives are worth knowing about for this workload. Forking a running machine copy-on-write lands in 400 to 750ms on the same host (1.2 to 3.5s across hosts), which lets you branch a detonation at an interesting moment — right before the sample decides whether it's being watched — and explore several environmental variations from the same live state. And if you need the sample's own environment to look substantial rather than empty, a managed Postgres alongside the sandbox takes 30 to 90 seconds to provision; useful when the thing you're detonating is hunting for a database to ransom, and useless if you never give it one.

When this is the wrong architecture

If your corpus is overwhelmingly Windows-targeted — and for most enterprise detonation, it is — Firecracker microVMs are not where you start, because the guest you need is Windows and Firecracker's device model is built around Linux guests with virtio drivers. Verify what any given platform actually supports before you plan around it. If your samples are highly evasive, targeted, or expensive-to-acquire, spend the money on a realistic environment; a fast pipeline that produces confident wrong answers is worse than a slow one. And if you're doing static analysis, unpacking, or signature matching, you don't need to execute anything at all — the safest detonation is the one you didn't perform.

Where microVMs win is the boring middle of the distribution, which is also most of the volume: a queue of suspicious files from users, customers, or an email gateway, each needing a real machine, real behaviour, and a verdict, without any of them touching each other or you. For that job the combination — hardware boundary, minimal device surface, a pristine restored machine per sample in under 200ms, a fake network the sample believes, and artifacts harvested before an unlink — is a genuinely good fit. Just don't let anyone tell you the samples can't tell they're in a VM. They can. Plan for the ones that care.