Running Fuzzing Harnesses and Crash Reproduction in MicroVMs

Most workloads execute untrusted input by accident. Fuzzing executes untrusted input on purpose, at a few hundred thousand inputs per second, with a coverage-guided search algorithm whose explicit objective function is "find the input that breaks this." A fuzzer is a professional vandal you hired and gave a corpus to. Its whole job is to steer your parser into the states its author never imagined — off-by-one reads, double frees, unbounded allocations, stack exhaustion, integer wraps that turn a length check into a suggestion. When it succeeds, you have a process that is, by definition, executing in a corrupted memory state on your machine. That is the moment to ask what else is on that machine.

I'm Ajay; I built PandaStack. This post is about running fuzzing harnesses — libFuzzer, AFL++, cargo-fuzz, OSS-Fuzz-shaped pipelines, and the nastier case of syscall fuzzers like syzkaller — inside ephemeral Firecracker microVMs. Along the way: why the shared-kernel assumption breaks specifically for this workload, how to fence a process whose success criteria include exhausting your RAM, how to make a crash from six months ago replay byte-identically, and how snapshot forking turns a serial campaign into a fan-out.

Fuzzing is the purest form of untrusted-input execution

A web service parsing user JSON is executing untrusted input, but the input distribution is overwhelmingly benign and the code path is well-trodden. A fuzzer inverts both properties. Coverage guidance means the inputs that survive selection are precisely the ones reaching code nobody exercises: the error branch in the error branch, the length field nobody validated because "that can't happen," the legacy codec path behind a feature flag that has been on since 2019.

• The mutation engine is adversarial by construction. It doesn't need a threat model or an attacker; the genetic algorithm rediscovers your bugs on its own, and it never gets bored or goes home.
• Sanitizers make corruption loud, not absent. ASan aborts on a heap-buffer-overflow it detects — but a fuzzer also finds corruption ASan doesn't instrument, plus every bug in the sanitizer runtime, plus jitted or hand-written assembly where instrumentation simply isn't present.
• The target is frequently C or C++ someone else wrote. Image codecs, font shapers, protocol parsers, compression libraries, video demuxers — the corpus of code most worth fuzzing overlaps almost perfectly with the corpus of code most likely to have exploitable memory bugs.
• Crashes are the deliverable. In every other pipeline a segfault is an incident; here it's the artifact you're shipping to the triage queue. Any infrastructure that treats a crash as exceptional is fighting the workload.
• The intent boundary is fuzzy in the worst direction. "Fuzz this parser" and "execute a memory-corruption exploit against this parser" describe the same process doing the same thing. Your infrastructure can't tell them apart, so it should be built as though it's always the second one.

Now consider where most fuzzing actually runs: a container on a shared CI host, or worse, a beefy bare-metal box in the corner that also happens to hold the signing keys because it was the machine with the most cores. A container is a polite suggestion to the kernel — namespaces and cgroups are excellent for resource accounting and reasonable for separating cooperating workloads, and they are a single `unshare`-adjacent kernel bug away from being nothing at all. That's an acceptable bet for running your own test suite. It is a strange bet for a process whose full-time occupation is generating novel inputs that violate memory safety assumptions.

Every other workload asks you to imagine an attacker. Fuzzing hires one, gives it your source code, and pays it by the CPU-hour.

Kernel and syscall fuzzing: where shared-kernel isn't a tradeoff, it's a contradiction

Userspace fuzzing is the easy case. The hard case is when the target *is* the kernel. syzkaller and its relatives generate sequences of syscalls with structured, semantically-aware arguments, specifically hunting for kernel panics, use-after-frees in driver code, deadlocks in the VFS layer, and races in the network stack. The success condition is a kernel oops.

If your syscall fuzzer shares a kernel with anything else, you have not built an isolation boundary — you've built a scheduled outage. A found bug doesn't politely terminate the fuzzer process; it takes down the kernel that every co-resident container is also running on. Containers, gVisor, seccomp-filtered processes: none of these help, because all of them are *implemented in terms of* the thing you're attacking. gVisor is genuinely interesting here in an inverted way — its userspace kernel is itself a valuable fuzzing target, and fuzzing it still means the host kernel underneath is exposed to whatever syscalls the Sentry forwards.

This is why kernel fuzzing has always been a VM workload, and why syzkaller's own tooling is built around spinning up disposable VM images and reading the panic off a serial console. Firecracker fits that shape almost too well: a guest kernel per VM, a serial console you can capture, boot times short enough that "reboot after every panic" stops being a scheduling problem, and a snapshot mechanism that lets you restore a known-good machine rather than rebuild one. The panic kills a guest kernel that exists for exactly this purpose, the host never notices, and the campaign continues.

Resource fences: it will exhaust your RAM, and that's not a bug

Here is the operational reality nobody warns you about the first time: a fuzzer will find the input that makes your target allocate four gigabytes from a two-byte length field. It will find the input that recurses until the stack is gone. It will fill your disk with a corpus that grows monotonically because every new edge is worth keeping, and then fill it again with crash artifacts, and then fill it a third time with the coverage data. It will saturate every core you give it, because that's the correct behavior — throughput is the entire point. None of this is malfunction. This is the fuzzer doing its job with enthusiasm.

So the fences have to be structural rather than polite. Inside the guest you use the fuzzer's own knobs and process limits; outside, the microVM's fixed vCPU and RAM allocation is the wall that actually holds. A guest that OOMs kills its own campaign and nothing else, which is exactly the blast radius you want — compare that to a cgroup-limited container on a busy host, where enough memory pressure gets the kernel's OOM killer involved and it reaps whichever process looks tastiest, which may well be a build job belonging to someone else entirely.

#!/usr/bin/env bash
# Runs INSIDE the guest. The fuzzer is a professional vandal --
# fence it before you hire it, not after it eats the host.
set -euo pipefail

# Build the harness with sanitizers so corruption ABORTS loudly instead of
# silently scribbling on the heap and lying to you three million inputs later.
clang -g -O1 -fsanitize=fuzzer,address,undefined -fno-omit-frame-pointer -o /work/parse_fuzzer /work/fuzz/parse_fuzzer.c /work/src/parser.c

# Process-level fences. These are the inner ring; the microVM's baked RAM
# ceiling is the outer ring that actually holds when these are bypassed.
ulimit -v 3145728    # 3 GiB address space -- the target WILL find this wall
ulimit -f 4194304    # 4 GiB max file size -- corpora grow without shame
ulimit -u 512        # cap fork bombs from a target that spawns helpers
ulimit -c 0          # no core dumps; the sanitizer report is more useful

mkdir -p /work/artifacts /work/corpus

# libFuzzer's own limits. -rss_limit_mb catches the allocation blowups,
# -timeout catches the inputs that hang instead of crashing, and
# -max_total_time makes the campaign a bounded job rather than a resident.
export ASAN_OPTIONS=abort_on_error=1:detect_leaks=1:malloc_limit_mb=2048:handle_abort=1
export UBSAN_OPTIONS=print_stacktrace=1:halt_on_error=1

/work/parse_fuzzer /work/corpus -artifact_prefix=/work/artifacts/ -rss_limit_mb=2048 -malloc_limit_mb=2048 -timeout=25 -max_total_time=900 -print_final_stats=1 -jobs=0 || true

# NOTE the '|| true'. libFuzzer exits non-zero when it finds a crash.
# In this pipeline a non-zero exit is SUCCESS, so the shell must not
# treat the good news as a failure and abort before we collect artifacts.
ls -1 /work/artifacts | sed 's/^/artifact: /'

• Cap memory twice — once with the fuzzer's `-rss_limit_mb` and `ulimit -v` so you get a clean, attributable report, and once with the guest's fixed RAM so a bypass of the first ring is still contained.
• Bound the campaign in wall-clock time and set a TTL on the sandbox. A fuzzing job with no end condition is a subscription, and the fuzzer has no opinion about your monthly bill.
• Treat disk as adversarial. Corpus growth, artifact accumulation, and coverage dumps all expand until stopped; give the guest a disk budget and export what matters before you destroy it.
• Don't over-tune CPU. Fuzzing is embarrassingly parallel and throughput-hungry; the right move is more independent guests, not one guest with every core and a scheduler fight.
• Expect non-zero exits and hard aborts as normal operation. Pipelines that interpret SIGABRT as "the infrastructure broke" will page you every time the fuzzer succeeds.

Egress denial: a fuzzed parser that reaches a socket

Fuzzing targets are rarely pure functions. A protocol parser may resolve names. A document parser may fetch remote resources because some standards committee thought external entities were a good idea. A media library may phone home for codec metadata. And once memory corruption is in play, control flow is negotiable — a fuzzer that drives a target into a corrupted state has, in the limit, produced a process that can do anything the process was permitted to do, including opening sockets.

So the network policy for a fuzzing guest should be default-deny with a very short allowlist, and it should be enforced outside the guest where the corrupted process can't reach it. On PandaStack every sandbox gets its own network namespace with its own veth pair and TAP device — each agent pre-allocates 16,384 /30 subnets — so per-sandbox egress rules are a property of the host's filtering, not a setting the guest can politely be asked to respect. If your harness genuinely needs a network dependency, run a stub inside the guest and point the target at localhost. A fuzzer that discovers your staging environment is a bad afternoon; a fuzzer that discovers your metadata service is a much longer one.

There's a second-order benefit: denying egress makes the crash reproducible. A target that reached the network during a campaign produced a crash that depends on what a remote server said at that moment, which means your crash is now a distributed systems problem. Cutting the network converts an unreproducible "it crashed once in the Tuesday run" into a deterministic function of the input bytes. For more on the enforcement mechanics, /blog/controlling-network-egress-untrusted-code covers the general pattern.

Crash reproduction: the entropy you forgot you had

The single most demoralizing experience in fuzzing is a crash artifact that doesn't reproduce. The fuzzer swears it found a heap-buffer-overflow, it wrote the input to disk, and running that input by hand exits zero and looks at you innocently. This happens constantly, and it's almost never a fuzzer bug. It's ambient entropy that the campaign had and your replay doesn't.

• ASLR — the classic. A use-after-free reads freed memory whose contents depend on the heap layout, and the heap layout depends on where the loader put things. Same input, different address randomization, no crash. Run replays under a disabled-ASLR personality (`setarch -R`) and make that part of the recorded environment.
• Environment variables and the argv block — these sit on the stack and shift every subsequent offset. A campaign run from a CI job with forty env vars and a replay from your shell with six are, from the target's perspective, different programs.
• `/dev/urandom` — any hash-table seed, allocator randomization, or nonce pulled at startup changes bucket ordering, which changes allocation sequence, which changes what a stale pointer reads. Pin the seed or stub the device.
• Time — timeouts, TTL checks, expiry logic, and anything reading a monotonic clock introduce input-independent variance. A crash that only fires when a parse takes longer than a threshold is a real bug that reproduces only under matching load.
• Thread scheduling — if the target is multithreaded, the crash may be a race, and the race's reproduction rate is a property of the machine's core count and load, not of the input file.
• Library and allocator versions — glibc's malloc behaves differently across versions, tcmalloc differently again. A crash found against one allocator can be entirely absent under another, which is why "it doesn't repro on my laptop" is such a common and useless data point.

A snapshot-restored microVM removes the last category outright and makes the rest recordable. Every restore of a given snapshot generation starts from a byte-identical machine: same kernel, same glibc, same allocator, same locale data, same everything down to the page. On PandaStack a template's first spawn is a genuine cold boot of around 3 seconds, and every create afterward restores that baked snapshot — roughly 49ms for the restore step, 179ms p50 end to end and about 203ms p99. So "the environment that found this crash" stops being a description and becomes an artifact you can name. Pair that with /blog/reproducible-build-sandboxes for the build half of the story.

Snapshot forking: fan out a campaign without paying startup N times

Fuzzing scales by running more of it. But every worker has to do the same expensive prelude first: compile the harness with instrumentation, load and dedupe the seed corpus, warm the target's initialization path, mmap the dictionary, maybe restore a database or start a stub server the harness talks to. Doing that N times, in N fresh machines, means most of your fan-out budget goes to work you already did.

The microVM answer is to do it once and fork. Bring one guest to a fully post-initialization state, snapshot it — memory and disk — then fork that snapshot N ways. Each fork is an independent machine that resumes exactly where the parent left off, with the harness compiled, the corpus resident, and the target initialized. Memory is copy-on-write until a fork writes, and the rootfs is a reflink clone, so the tenth worker doesn't copy gigabytes. A same-host fork lands in 400–750ms (cross-host 1.2–3.5s, since memory has to travel). See /blog/copy-on-write-memory-fork-explained for the mechanics.

from concurrent.futures import ThreadPoolExecutor
from pandastack import Sandbox

N_WORKERS = 24
CAMPAIGN_SECONDS = 900

# ---- 1. ONE machine pays the prelude: build + corpus load + target init ----
base = Sandbox.create(template="code-interpreter", ttl_seconds=3600)
base.exec(
    "git clone --depth 1 https://github.com/acme/parser /work",
    timeout_seconds=180,
)
base.filesystem.write("/work/build.sh", BUILD_SCRIPT)   # the bash above, minus the run
base.exec("bash /work/build.sh", timeout_seconds=900)

# Minimize the seed corpus once, here, so every worker inherits the small one.
base.exec(
    "/work/parse_fuzzer -merge=1 /work/corpus_min /work/corpus_raw",
    timeout_seconds=600,
)

snap = base.snapshot()   # memory + disk, post-initialization
base.kill()

# ---- 2. Fork N ways. Each worker skips the prelude entirely. ----
def fuzz_shard(worker_id: int) -> dict:
    fork = snap.fork(ttl_seconds=CAMPAIGN_SECONDS + 300)
    try:
        # Different -seed per worker: same machine, divergent search paths.
        # Identical everything else, so a crash's environment is known exactly.
        run = fork.exec(
            "cd /work && ASAN_OPTIONS=abort_on_error=1:malloc_limit_mb=2048 "
            "./parse_fuzzer corpus_min -artifact_prefix=/work/artifacts/ "
            f"-seed={1000 + worker_id} -rss_limit_mb=2048 -timeout=25 "
            f"-max_total_time={CAMPAIGN_SECONDS} || true",
            timeout_seconds=CAMPAIGN_SECONDS + 120,
        )

        # Collect crash artifacts BEFORE the guest goes away. Nothing else
        # survives the kill, which is the point -- a corrupted machine is
        # not something you want to hand to the next job.
        names = fork.exec("ls -1 /work/artifacts 2>/dev/null").stdout.split()
        return {
            "worker": worker_id,
            "stderr_tail": run.stderr[-4000:],
            "crashes": {
                n: fork.filesystem.read(f"/work/artifacts/{n}") for n in names
            },
        }
    finally:
        fork.kill()   # the vandal, the corpus, and the corrupted heap all vanish

with ThreadPoolExecutor(max_workers=N_WORKERS) as pool:
    results = list(pool.map(fuzz_shard, range(N_WORKERS)))

found = sum(len(r["crashes"]) for r in results)
print(f"{N_WORKERS} workers, {found} crash artifacts collected")

This shape has a property worth stating explicitly: every worker started from the same snapshot, so when one of them crashes, you know the machine. Not approximately — exactly. The crash's environment is a snapshot generation, and the only intentional divergence is the fuzzer's own seed. That turns a crash from "something happened on worker 17" into a reproducible tuple you can hand to triage.

Corpus minimization and triage as their own jobs

A campaign produces two things that need follow-up work, and both are natural microVM jobs of their own. The first is a corpus that has grown fat — thousands of inputs, many of which cover the same edges. Minimization (`-merge=1` in libFuzzer, `afl-cmin` and `afl-tmin` in AFL++) re-executes every input to compute coverage, which means it is itself a bulk execution of adversarial inputs against an instrumented target, with all the same containment requirements as the campaign that produced them. Run it in a guest, keep the small corpus, throw away the machine.

The second is a pile of crash artifacts that need to be deduplicated and classified. Triage means running each crashing input again, capturing the sanitizer report, computing a stack hash so N crashes collapse into M unique bugs, and often bisecting to find the first commit where the input crashes. Every one of those steps is "execute a known-malicious input against a known-vulnerable binary," repeatedly. Doing that on a shared runner because it's "just triage" is how a crash-reproduction step becomes an intrusion.

1. Deduplicate by stack hash, not by input bytes. A hundred distinct inputs frequently reduce to two bugs, and a triage queue that doesn't collapse them will bury the interesting one.
2. Re-run each artifact under a pinned, ASLR-disabled environment and record whether it reproduces. Non-reproducing artifacts are not garbage — they're evidence of a bug with an environmental dependency you haven't identified yet.
3. Minimize the input itself. A 40 KB crashing blob and a 12-byte one describe the same bug, but only one of them gets fixed this quarter.
4. Bisect inside disposable guests, one commit per fork. Each candidate build is an independent machine, so a bisect step that panics or hangs costs you one guest instead of stalling the whole run.
5. Fail closed on inconclusive results. A triage job that crashes for infrastructural reasons should report "unknown," never "not reproducible" — silently downgrading a real bug to noise is the one outcome worse than a slow queue.

The crash bundle: making a six-month-old crash replay byte-identically

The artifact people usually keep is the crashing input. That's the *least* portable part of the record, because the input alone means nothing without the machine that made it crash. What you actually want to store is a bundle: the input, the exact snapshot generation of the environment, the target commit and build flags, the runtime knobs that were pinned, and the observed failure. Then "reproduce CVE-candidate-2026-0043" means restoring a machine, not reconstructing an archaeology project.

// A crash bundle is the unit that survives the campaign. The input bytes
// alone are useless six months later -- what makes a crash replayable is
// the environment it crashed in, recorded as an identifier you can restore.
export type CrashBundle = {
  id: string;                  // stable, derived from stackHash + inputSha256
  foundAt: string;             // ISO-8601, from the ORCHESTRATOR's clock

  target: {
    repo: string;
    commit: string;            // exact SHA, not a branch name
    harness: string;           // fuzz/parse_fuzzer.c
    buildFlags: string;        // -O1 -fsanitize=fuzzer,address,undefined
    sanitizers: string[];      // ["address", "undefined"]
  };

  // The single most important field, and the one nobody records: WHICH
  // machine. A snapshot generation is restorable; "ubuntu-latest" is not.
  environment: {
    template: string;          // e.g. "fuzz-parser"
    snapshotGeneration: string;
    guestKernel: string;
    libc: string;
  };

  // Pinned entropy. Every one of these, left unpinned, is a reason a real
  // crash "doesn't reproduce" and gets closed as flaky.
  runtime: {
    aslr: "disabled";          // replay under setarch -R
    fuzzerSeed: number;
    urandomSeed: string;       // stubbed device, fixed stream
    clock: string;             // frozen guest clock, recorded explicitly
    env: Record<string, string>;  // exact env block: it lives on the stack
    argv: string[];
    networkEgress: "denied";
  };

  input: { sha256: string; bytes: number; storedAt: string };

  observed: {
    sanitizer: string;         // "heap-buffer-overflow READ of size 4"
    signal: string;            // "SIGABRT"
    stackHash: string;         // triage dedup key
    topFrames: string[];
    reproducedOutOf: [number, number];  // e.g. [10, 10] -- deterministic
  };
};

// Replay, months later: restore `environment.snapshotGeneration`, write
// `input` into the guest, apply `runtime`, run once. If `observed.stackHash`
// matches, the bug is confirmed against the ORIGINAL machine -- and any
// difference is a real finding rather than unexplainable environment drift.

The discipline this enforces is the same one that makes billing runs auditable and CI results trustworthy: the environment is an input, so record it like one. A team that stores only crash inputs will, within a year, have a directory of blobs that nobody can reproduce and therefore nobody triages. A team that stores bundles can answer "is this still a bug?" by restoring a machine — and can distinguish "we fixed it" from "the environment moved and the bug is hiding."

Bare metal vs containers vs gVisor vs microVMs for fuzzing

• Memory-corruption blast radius — Bare metal: a successful exploit chain owns the host, and the fuzzing box is usually the one with the most cores and the loosest access rules. Container: namespaces plus seccomp bound a *cooperating* process, but a corrupted one is separated from the host by kernel code paths it is professionally employed to break. gVisor: a userspace kernel absorbs most syscalls and genuinely narrows the surface, at the cost of syscall-heavy throughput — which fuzzing has in abundance. MicroVM: hardware-virtualized guest with its own kernel, so a fully corrupted target owns a machine you were about to delete anyway.
• Kernel / syscall fuzzing — Bare metal: a found bug panics the machine you were using. Container: incoherent — the target is the kernel you're sharing, so a success is a host outage. gVisor: you can fuzz the Sentry, but the host kernel is still downstream of forwarded syscalls. MicroVM: the intended shape; the panic kills a disposable guest kernel and the host doesn't notice.
• Resource exhaustion — Bare metal: the fuzzer discovers your RAM ceiling on your behalf and takes the box with it. Container: cgroups bound memory well, but sustained pressure invites the host OOM killer, which reaps by heuristic and may pick a neighbor. gVisor: similar cgroup story with extra memory overhead per sandbox. MicroVM: fixed vCPU/RAM baked into the guest plus a hard TTL, so an oversized campaign exhausts only itself.
• Crash reproducibility — Bare metal: the machine drifts with every apt upgrade, so last quarter's crash is unreproducible for reasons unrelated to the bug. Container: image tags help a lot, but base-image and layer-cache drift still bite, and the host kernel underneath is whatever the fleet is running today. gVisor: the intercepted syscall surface is stable, which is genuinely nice, but the underlying image drifts the same way. MicroVM: every create restores a pinned snapshot generation — same kernel, same libc, same allocator — so a replay is a genuine replay.
• Parallel fan-out cost — Bare metal: you scale by buying machines and manually sharding, which is why campaigns end up serial. Container: cheap to start, but every worker repeats the build and corpus-load prelude. gVisor: comparable start cost with a syscall-throughput tax on the fuzzing loop itself. MicroVM: fork one post-initialization snapshot N ways at 400–750ms same-host, so workers resume warm instead of starting over.
• Egress control — Bare metal: whatever the host firewall says, and the fuzzing box's rules are usually generous because someone needed to `curl` a dependency in 2023. Container: per-container network policy is real and workable, though often applied at a coarser granularity than one job. gVisor: netstack gives strong in-sandbox control. MicroVM: per-sandbox network namespace with its own veth and TAP, filtered on the host — 16,384 pre-allocated /30 subnets per PandaStack agent — so egress rules aren't something the guest can renegotiate.

One honest caveat on that table: the container and gVisor columns describe general architectural properties, not benchmark results, and both projects move quickly. Verify current behaviour and limits against their own documentation before making an infrastructure decision on my say-so. The only measured numbers here are PandaStack's, and /blog/firecracker-vs-kata-vs-gvisor works through the isolation-model comparison in more depth.

The summary

Fuzzing is the workload where the untrusted-code question stops being hypothetical. You are running a search algorithm whose success condition is a memory-corrupted process, at maximum throughput, for hours, on purpose. Running that on a shared kernel is a bet that the kernel is perfect against inputs you are specifically paying to make imperfect — and if the target is the kernel itself, it isn't even a bet, it's a contradiction.

A microVM per campaign gets you five things at once. A real guest kernel, so a panic or a corrupted process destroys a machine that existed for this and nothing else. Hard resource fences, because a fuzzer will exhaust RAM, fill disk, and pin every core — that's it working correctly, not failing. Host-enforced egress denial, so a target that reaches a socket in a corrupted state finds nothing and your crash stays a deterministic function of input bytes. A pinned, snapshot-backed environment, so the crash bundle you file today still replays byte-identically next year. And snapshot forking, so a campaign fans out from one post-initialization machine at 400–750ms per worker instead of repeating the build prelude N times.

You hired a vandal to break your code. Give it a room with no windows, a hard budget, and a door you can lock behind it — then keep the receipts, because the whole point was to find something.