Run the PoC: Vulnerability Triage in Disposable microVMs

Most "running untrusted code" problems involve a workload you chose to accept. Vulnerability triage is not one of them. Someone you have never met and cannot identify sends you a file and a sentence — "unauthenticated RCE in 3.2.1, run `poc.py` against a default install" — and your job is to find out whether that sentence is true. You can read the script. You should. Reading it still does not answer the question, because the question is whether it works.

I'm Ajay; I build PandaStack, a Firecracker microVM platform, and triage teams keep arriving with the same shape of problem. Bug bounty inboxes, CVE validation, and the quarterly "are we affected?" fire drill all bottom out in one act: execute an anonymous person's code and report what happened. This post is the practical version — why triage is harder than malware detonation, what one-microVM-per-report looks like, and how forking turns a flaky exploit into a number.

The one job where running a stranger's code is the deliverable

Triage has exactly two acceptable outputs: reproduced, or not reproduced, each with the steps that got you there. "Looks plausible" is not an output. A bounty payout, a severity rating, a patch priority, and sometimes an emergency release all hang off which of those two words you write, so the standard is not vibes — it is a run you can replay in front of a skeptical engineer.

That makes triage genuinely unusual. Everywhere else in security, executing an unknown binary is the failure mode you are paid to prevent. Here it is the contractual obligation. A reporter who understands their own bug sends a clean, minimal PoC. A reporter who does not — or one who is not really a reporter — sends something that also drops a miner, phones home the instant it lands, or bundles a second-stage downloader with a perfectly good exploit for the wrong product. You find out by running it.

The second twist: the environment has to be broken on purpose

Malware analysts have it comparatively easy: the sample is hostile, but the environment is a normal machine that happens to be disposable. Triage inverts half of that. To reproduce a report you have to stand up the vulnerable thing — the unpatched service at the reported version, an old kernel for the privesc chain, a known-bad TLS library, debug endpoints on, default credentials left in place. You are deliberately building the host you would fire someone for deploying.

A container escape is a theoretical concern for most workloads; here it is the literal success criterion of the thing you are about to start. If the report claims a sandbox bypass, a successful triage run is a sandbox bypass — inside your triage environment — and the only question left is how far it gets. Your isolation boundary is not braced against a hypothetical adversary; it is being tested by someone who wrote a working exploit and mailed it to you.

• The target usually runs privileged. Services get root, debug agents get root, and the point of many reports is reaching root from somewhere else.
• The kernel may be the vulnerable component. Local privesc and container-escape reports put the bug in exactly the layer a container asks you to share.
• Old and unpatched is the requirement, not an oversight. You cannot triage "affects 3.2.1" on a box your fleet automation dutifully patched to 3.3.0.
• Success means the attacker won. Your oracle for "reproduced" is a shell, a marker file, a crash, or a credential read — states you spend the rest of your career preventing.

This is why a shared-kernel sandbox is the wrong tool here, and not in a hand-wavy way. If the report is a kernel bug, a container runs on the host's kernel — so you either lie about the version and reproduce nothing, or you install the vulnerable kernel under everything else on that machine. A microVM boots its own guest kernel behind hardware virtualization, so "vulnerable kernel" is a property of one VM rather than of your host. Verify against any platform's own docs which kernel images and guest configurations you can supply; that capability decides which classes of report you can handle at all.

The shape: one microVM per report, built from a pinned snapshot

The pattern that holds up is one disposable microVM per report, created from a snapshot of the vulnerable stack at the reported version. You pay the installation cost once per version — install the target, configure it as the report describes, warm it to steady state — and snapshot there. From then on, "give me a fresh vulnerable 3.2.1" is a restore, not a build.

That is what makes this feel different from a lab box. A PandaStack sandbox is created by restoring a pre-baked snapshot rather than cold-booting: p50 179ms, p99 203ms, with the restore step itself around 49ms; only the first-ever boot of a template costs about 3 seconds. When a fresh vulnerable environment is a couple hundred milliseconds away, you stop reusing the dirty one — and reusing the dirty one is where triage results go to die.

# triage.py -- one microVM per inbound report, from a template baked at
# the exact reported version. None of this runs on your laptop.
import hashlib
import json
from pathlib import Path
from pandastack import Sandbox

RUN = """#!/bin/bash
set -uo pipefail
mkdir -p /work/out

# The vulnerable service is already installed in the template. Start it,
# and record what it thinks its own version is -- not what you believe
# you installed three weeks ago.
/opt/target/bin/serve --port 8080 >/work/out/target.log 2>&1 &
echo $! > /work/out/target.pid
/opt/target/bin/serve --version > /work/out/version.txt 2>&1
until curl -sf localhost:8080/healthz >/dev/null; do sleep 0.2; done

# Instrument the attempt: packets and syscalls, both to disk.
tcpdump -i any -s0 -w /work/out/attempt.pcap 'tcp port 8080' &
echo $! > /work/out/tcpdump.pid
strace -f -tt -o /work/out/target.strace -p "$(cat /work/out/target.pid)" &
echo $! > /work/out/strace.pid

# The reporter's code, as an UNPRIVILEGED user, under a hard timeout.
# A PoC that hangs is a PoC that hangs; it is not a reason to wait.
timeout 120 setpriv --reuid=triage --regid=triage --clear-groups \
  python3 /work/poc.py > /work/out/poc.log 2>&1
echo "poc_exit=$?" > /work/out/result.txt

# Cheap oracles, evaluated in-guest while the evidence still exists.
test -f /tmp/pwned && echo "marker=yes" >> /work/out/result.txt
id -u triage >/dev/null && ls -la /root 2>/dev/null | head -5 >> /work/out/result.txt
dmesg | tail -40 > /work/out/dmesg.txt

kill "$(cat /work/out/tcpdump.pid)" "$(cat /work/out/strace.pid)" 2>/dev/null
sleep 1
"""


def triage(report_id: str, poc_source: str, template: str) -> dict:
    """template = a snapshot baked at the exact version the report names."""
    with Sandbox.create(
        template=template,                     # e.g. "target-3-2-1"
        ttl_seconds=3600,                      # vulnerable by design -> expires
        metadata={
            "report": report_id,
            "purpose": "triage",
            "trust": "none",
            "poc_sha256": hashlib.sha256(poc_source.encode()).hexdigest(),
        },
    ) as sbx:
        sbx.filesystem.write("/work/poc.py", poc_source)
        sbx.filesystem.write("/work/run.sh", RUN)

        r = sbx.exec("bash /work/run.sh", timeout_seconds=300)

        out = {
            "report": report_id,
            "exit_code": r.exit_code,
            "duration_ms": r.duration_ms,
            "version": sbx.filesystem.read("/work/out/version.txt").decode().strip(),
            "result": sbx.filesystem.read("/work/out/result.txt").decode(),
            "poc_log": sbx.filesystem.read("/work/out/poc.log").decode()[-8000:],
            "dmesg": sbx.filesystem.read("/work/out/dmesg.txt").decode(),
        }

        # Evidence for the write-up, pulled out as bytes over the
        # filesystem API -- not over the guest's network, and not as a
        # screenshot of somebody's terminal.
        ev = Path(f"evidence/{report_id}")
        ev.mkdir(parents=True, exist_ok=True)
        for name in ("attempt.pcap", "target.strace", "target.log"):
            (ev / name).write_bytes(sbx.filesystem.read(f"/work/out/{name}"))
        (ev / "summary.json").write_text(json.dumps(out, indent=2))
        return out
    # VM destroyed here, along with whatever the PoC left running in it.

Two details matter more than they look. The version string is captured inside the guest, because "which version did we actually test" is what unravels half of all disputed triage results. And the PoC's hash goes into the VM's metadata, so the artifact you ran stays tied to the report even after the reporter edits their gist.

Fork per attempt: turning "it only works sometimes" into a number

A large fraction of interesting reports are probabilistic. Heap sprays land when the allocator cooperates. Races win when the scheduler cooperates. Use-after-free triggers when the object you want gets reclaimed by the object you need. The reporter says it works about one time in twenty; you run it three times, get nothing, and are now one impatient sentence away from closing a real bug as "could not reproduce."

Non-reproduction is a claim about state, and it only means something if you controlled the state. Run five attempts on one long-lived box and each starts somewhere different: warmer page cache, more fragmented heap, a zombie from attempt three still holding a file descriptor. You are not sampling the exploit's success rate. You are sampling your own drift.

Forking fixes the denominator. Snapshot the vulnerable stack once it has reached steady state, then fork that snapshot per attempt: every child starts from identical memory and an identical copy-on-write disk. Same allocator arrangement, same process table, same everything you baked in. What still varies is host-level timing — CPU scheduling, contention, interrupt arrival — which is exactly the variable you want to sample over for a race. Same-host forks land in the 400–750ms range, so fifty attempts is a coffee, not an afternoon.

# Flaky by nature: sprays and races land some fraction of the time.
# "Didn't reproduce" only means something if every attempt started from
# the SAME state -- so snapshot once, fork per attempt.
from concurrent.futures import ThreadPoolExecutor
from pandastack import Sandbox


def prepare(template: str, report_id: str) -> Sandbox:
    """Boot the vulnerable stack, drive it to steady state, snapshot there."""
    sbx = Sandbox.create(
        template=template,
        ttl_seconds=7200,
        metadata={"report": report_id, "role": "fork-parent"},
    )
    sbx.exec("/opt/target/bin/serve --port 8080 >/var/log/target.log 2>&1 &")
    sbx.exec("until curl -sf localhost:8080/healthz; do sleep 0.2; done",
             timeout_seconds=60)
    # Warm it the way the report describes -- if the bug needs a fragmented
    # heap or a populated cache, bake that INTO the snapshot so every
    # attempt inherits it instead of racing to recreate it.
    sbx.exec("/opt/target/bin/warmup --requests 5000", timeout_seconds=120)
    sbx.snapshot()          # <- the known state every attempt starts from
    return sbx


def attempt(parent: Sandbox, poc: str, i: int) -> dict:
    child = parent.fork()   # identical memory + rootfs, copy-on-write
    try:
        child.filesystem.write("/work/poc.py", poc)
        r = child.exec(
            "timeout 60 setpriv --reuid=triage --regid=triage --clear-groups "
            "python3 /work/poc.py",
            timeout_seconds=90,
        )
        marker = child.exec("test -f /tmp/pwned && echo hit").stdout.strip()
        crashed = "segfault" in child.exec("dmesg | tail -20").stdout.lower()
        return {
            "i": i,
            "exit": r.exit_code,
            "won": marker == "hit",
            "crash": crashed,
            "ms": r.duration_ms,
        }
    finally:
        child.kill()        # one attempt, one VM, no residue


parent = prepare("target-3-2-1", "HR-4471")
poc = open("reporter_poc.py").read()
try:
    with ThreadPoolExecutor(max_workers=8) as pool:
        runs = list(pool.map(lambda i: attempt(parent, poc, i), range(50)))
finally:
    parent.kill()

hits = sum(1 for r in runs if r["won"])
crashes = sum(1 for r in runs if r["crash"])
print(f"reproduced {hits}/{len(runs)}  (crashes without control: {crashes})")
# 0/50 from a clean snapshot is a real finding you can defend.
# 3/50 is a real bug with a rate attached -- and 'crashes without
# control' is the column that decides DoS versus RCE.

That loop's output is what belongs in the triage note. "Reproduced 3 of 50 attempts from a clean snapshot of 3.2.1, with 11 further crashes that did not yield control" is a sentence a reporter cannot argue with and an engineer can act on. "Couldn't get it to work" is a shrug with a ticket number.

Which release introduced it, which one fixed it

The follow-up to "is it real" is always the range. Which versions are affected? Did the patch fix it, or only make this particular PoC stop working? Answering that by rebuilding an environment per candidate version is how a day disappears.

Snapshot each candidate version once, then fork to test. The expensive part — install, configure, boot, warm — is paid per version and amortized across every attempt against it. If your platform streams snapshot memory from object storage on demand (PandaStack does, via UFFD, which is why a cross-host fork lands in the 1.2–3.5s range instead of waiting on a full memory image), your shelf of vulnerable versions lives in a bucket rather than on somebody's local disk.

# "Which release introduced it?" as a loop, not a week.
# Each version has a snapshot baked once; attempts are forks of it.
VERSIONS = ["3.0.4", "3.1.0", "3.1.7", "3.2.0", "3.2.1", "3.3.0", "3.3.2"]


def hit_rate(version: str, poc: str, attempts: int = 40) -> float:
    tpl = "target-" + version.replace(".", "-")
    parent = prepare(tpl, f"bisect-{version}")
    try:
        runs = [attempt(parent, poc, i) for i in range(attempts)]
        return sum(1 for r in runs if r["won"]) / attempts
    finally:
        parent.kill()


# Binary search only if you believe the property is monotonic. For a
# flaky exploit it also needs enough attempts per probe that a 0.0 means
# "not observed in 40 tries", not "absent".
lo, hi = 0, len(VERSIONS) - 1
while lo < hi:
    mid = (lo + hi) // 2
    if hit_rate(VERSIONS[mid], poc) > 0:
        hi = mid
    else:
        lo = mid + 1
print("first affected release in this set:", VERSIONS[lo])

# When in doubt -- and you should doubt, because bugs get reintroduced
# and 'fixed' often means 'this PoC stopped working' -- sweep them all
# and publish the curve instead of a single boundary.
for v in VERSIONS:
    print(f"{v}: {hit_rate(v, poc):.0%}")

Instrumentation: the write-up is the product

Nobody remembers the run; they read the ticket. Decide what evidence you need before you press go, capture it inside the guest, and pull it out as files. The filesystem API is the right channel because it does not touch the guest's network — your evidence path and the exploit's egress path are different roads, which lets you keep egress brutally narrow without losing artifacts.

A screenshot of a terminal is not evidence. It is a picture of evidence, cropped by whoever wanted to be believed.

• Exit code, stdout, and stderr with timestamps, plus wall-clock duration — "it took nine minutes" changes the severity of a DoS report.
• A pcap of the attempt. This is what proves the "unauthenticated" adjective: no credential crossed that wire, and the packets say so.
• An strace of the target, dmesg, and the crash artifact — core dump, sanitizer report, minidump. Set `core_pattern` in the template, not in a panic after the interesting run.
• The target's version string read inside the guest at run time, the hash of the PoC as received, and a diff of anything you changed to make it run.
• The egress denial log. "The PoC tried to resolve and POST to an unfamiliar domain mid-exploit" is a high-signal finding you only get if something was there to say no.

One inversion worth adopting: on a confirmed reproduction, do not destroy the VM. It is now the most valuable object in the ticket — sitting in the post-exploitation state with the attacker's processes alive and memory intact. Snapshot or hibernate it and attach the id to the report, so the engineer who owns the fix can wake it and look around instead of asking you to reproduce it live on a call.

Egress: default-deny, and keep the callback inside your own VM

Isolation keeps the PoC off your host. Egress control decides whether a successful exploit is an interesting log line or an incident. Default-deny outbound, log every denial, attach the log to the ticket. The triage VM is not on your corporate network, has no route to internal services, and cannot reach the cloud metadata endpoint — a link-local address that hands credentials to whoever asks is a wonderful thing to find right after you granted someone code execution.

Then the case that makes triage different: a genuine PoC often needs a callback. Reverse shells, blind SSRF, out-of-band XXE, DNS canaries, deserialization gadgets that only prove themselves by reaching a collaborator. The report arrives with `LHOST=collab.reporter.example` baked in and demonstrates nothing until something answers. So allow it — deliberately, narrowly, pointed somewhere you own.

What you must not do is leave the reporter's host in there. Running their PoC unedited against their listener tells an anonymous third party the exact moment you reproduced their bug, hands them a shell into your triage environment, and throws in your egress IP. Point the callback at a listener inside your own VM — the same guest is simplest, since `nc -lvnp 4444` on loopback proves code execution as well as a shell across the internet does. PandaStack pre-allocates 16,384 /30 subnets per agent, so every sandbox already has its own network to be lonely in.

• Default-deny outbound; allowlist per VM, per run, and expire the allowance with the VM.
• Block the instance metadata endpoint explicitly. First thing a real attacker checks, last thing anyone remembers to close.
• Pin DNS to a resolver you control and log the queries. A PoC that resolves an unfamiliar domain mid-exploit told you something the code review missed.
• Host the collaborator yourself: an in-guest listener or a paired VM you own. Never the reporter's box, and never a public interaction service for anything sensitive.
• If the PoC fetches a second stage, allowlist that host for that run and archive what came back — you were just handed a second sample.
• Record every edit you made. "Modified LHOST to 127.0.0.1:4444, otherwise unchanged" is the difference between reproducing their bug and reproducing your version of it.

Four places to detonate a PoC

Same job, four topologies. Characterizations of any specific product's isolation, provisioning, and billing behavior should be verified against that vendor's own documentation and pricing pages — those details differ by configuration and they change.

• Escape blast radius — Bare-metal lab box: the box is the boundary, usually on a VLAN with other lab boxes and a forgotten route to corp. Container on a shared host: namespaces over the kernel you also need to be the vulnerable one, which is a contradiction the moment the report is a kernel bug. Cloud VM per report: a real hardware boundary, same class as a microVM. microVM fork: own guest kernel behind hardware virtualization, one VM per attempt, nothing shared with the next.
• Setup time — Bare-metal lab box: hours to reimage, which is why nobody does. Container on a shared host: fast to start, slow to get right, since "unpatched 3.2.1" is now an image you maintain. Cloud VM per report: minutes per boot plus provisioning — fine for one report, painful for fifty. microVM fork: snapshot restore at p50 179ms / p99 203ms; same-host fork 400–750ms.
• Identical-state repeatability — Bare-metal lab box: none in practice; attempt seven inherits attempts one through six. Container on a shared host: fresh filesystem, but kernel, page cache, and host state are shared and drifting. Cloud VM per report: clean per boot, but you re-warm to steady state each time and hope it lands the same. microVM fork: every attempt is a copy-on-write clone of one snapshot — same memory, same heap layout, same processes.
• Cleanup — Bare-metal lab box: a reimage you schedule and skip. Container on a shared host: `docker rm`, plus whatever the escape did to the host, which by definition you cannot enumerate. Cloud VM per report: terminate it, then remember the disk, the snapshot, the security group, the elastic IP. microVM fork: kill the VM and memory, disk, and every surviving process go with it — TTL as the backstop for the ones you forget.
• Cost — Bare-metal lab box: cheap-looking, expensive in hours nobody bills. Container on a shared host: cheapest to run, most expensive to be wrong about. Cloud VM per report: you pay for boot time and for whatever is still running on Friday. microVM fork: seconds of a small VM per attempt, with the vulnerable environments living as snapshots in object storage instead of machines you keep alive.

Housekeeping, and staying inside the scope you were given

Vulnerable-by-design machines must not outlive the triage that needed them. Every triage VM gets a TTL, with idle reaping as the second net. The failure mode is famous and boring: the lab box from two years ago, running an unpatched build for a report nobody closed, that somebody once gave a public address so a reporter could confirm a fix.

Put the report id in the VM's metadata alongside the version, the CVE, and the PoC hash. Then "which of our machines is running a known-RCE build right now" is a query, not an archaeology project you conduct during an audit. The same metadata makes the fork tree legible later: this VM is attempt 34 of report HR-4471 against 3.2.1, and here is the parent snapshot it came from.

On scope, briefly: reproduce only what you are authorized to reproduce — your own systems, or a report scoped to them. Not the reporter's proof host, not a third-party SaaS the report happens to mention, and not production because it will not reproduce in the lab. If it will not reproduce in the lab, the honest finding is that your lab does not match production, which is a configuration bug worth filing on its own.

None of this is free. You are running a small fleet of deliberately broken machines plus a pipeline that bakes a snapshot per version, and a VM that vanishes is harder to debug than one you can still SSH into — so log aggressively and preserve on confirmed hits. What you get back is that the two hardest words in triage stop being expensive. "Reproduced" comes with a pcap, a rate, and a version range. "Not reproduced" comes with fifty identical attempts from a clean snapshot. And the attachment from a stranger who will not give you their name stops being the scariest tab you have open.