Mutation Testing at Scale: Thousands of Broken Builds on Purpose

A test suite is a claim. It claims that if the code underneath it were wrong, something in here would fail. Almost nobody checks that claim. We check a much weaker one — did the line execute at least once — and then we quietly treat the weak claim as if it were the strong one. That is how a repository arrives at 94% coverage and ships a sign error.

Mutation testing checks the strong claim, by the only method available: break the code on purpose and see whether the suite notices. Change a `>` to a `>=`. Delete a `return`. Flip a boolean. Replace `+` with `-`. Then run the tests. If they go red, the mutant is killed and that line was genuinely defended. If they stay green, the mutant survived — the line ran, nothing asserted anything about what it did, and your coverage number for it was a participation trophy.

Cosmic Ray, one of the Python implementations, draws the line cleanly: coverage tells you a line executed, while mutation testing tells you whether your tests "actually check the behavior of your code." Those are very different statements and only one of them is about correctness.

I'm Ajay; I built PandaStack, which runs code in Firecracker microVMs. This post is about the part of mutation testing that nobody warns you about before you try it: it is not really a testing problem. It is a scheduling problem, and the things you are scheduling are thousands of programs you have specifically arranged to be wrong.

The tools, and what each one actually mutates

This is a mature field with real implementations per ecosystem, and they differ in ways that matter for how you schedule them.

• PIT (`pitest`) — JVM. It mutates the bytecode the compiler produced rather than the source files, which it describes as a speed and build-integration win with one honest cost: a bytecode mutation can be hard to describe as an equivalent change to a Java source file. Its default operators are the classic catalogue — conditionals boundary, negate conditionals, math, increments, invert negatives, void method calls — with returns mutators, constructor calls and a long optional/experimental list behind flags.
• Stryker — a family, not one tool: StrykerJS for JavaScript and TypeScript, Stryker.NET for C#, Stryker4s for Scala. Shared vocabulary, deliberately similar reports, and genuinely different option sets, which is where people get burned copying flags between them.
• `mutmut` — Python, with an explicit focus on ease of use: an interactive terminal UI, knowledge of which tests to execute, and between-run incrementality at function granularity. Note that `mutmut` 3 has a different execution model from `mutmut` 2, so advice you find online may be describing the other one.
• `cosmic-ray` — also Python, with distributed and concurrent execution as a first-class concern rather than an afterthought. If your inclination is to run mutants on a fleet, this is the Python tool whose architecture already assumes you will.
• `cargo-mutants` — Rust. It works without coverage instrumentation, which is unusual and convenient, and it has the cleanest diff-scoping story of the bunch.
• Go — `go-mutesting` mutates the AST and reports a mutation score as passed mutations over total; it is the long-standing option. `gremlins` is the newer one, still in `0.x` with explicit backward-compatibility caveats, and it advertises exactly the two optimisations that matter: it only tests mutants covered by tests, and it can test only the mutants in a PR's changes.

The cost model is a multiplication, which is the whole problem

Cost is roughly the number of mutants times the time to run the tests that cover each one. Both terms are larger than people expect. A mid-size repository generates thousands of mutants, because the operator set is applied at every eligible site — every comparison, every arithmetic operator, every return, every conditional. And the covering-test time is not two seconds for anything that touches a database, a browser or a compiler.

Multiply those out and you get the reason mutation testing has a reputation as a thing teams try once. Somebody runs it on a Friday, watches it still going on Monday, writes a thoughtful internal doc about how interesting the results were, and nobody ever runs it again. The failure is not intellectual. It is that the job was sized for a laptop and the arithmetic said fleet.

The arithmetic is also the good news, because every term in that product is independent. Mutant 1,847 does not care what happened to mutant 3. There is no shared state, no ordering constraint, no coordination, nothing to merge at the end but a list of verdicts. This is as embarrassingly parallel as a workload gets. The only things between you and a forty-minute mutation run are how many isolated machines you can get, how fast you can get them, and how quickly each one can be made ready to run your suite.

The four optimisations that actually move the number

1. Run only the tests that cover the mutated line. This is the single biggest lever and most tools have it. StrykerJS calls it `coverageAnalysis: "perTest"`: it works out which tests cover each mutant during the initial run and then executes only those. It has a precondition worth reading twice — the tests must be able to run independently and in random order — and static mutants fall back to running everything. `gremlins` only tests mutants that are covered at all. And a mutant with no covering test is not a survivor; it reports as `NoCoverage`, which is a different and much cheaper finding.
2. Kill a mutant on the first failing test, not after the whole suite. Once one test has gone red, the verdict is decided and every remaining second is waste. Over thousands of mutants this is the difference between a mean and a maximum.
3. Throw away mutants that cannot teach you anything. Unviable mutants that do not compile, mutants that are semantically equivalent to the original program, and whatever your config has explicitly ignored. Equivalent mutants are the interesting case: they are undetectable by construction, because no test can distinguish two programs that compute the same thing. They are also the reason a 100% mutation score is not a target you can reach.
4. Scope the run to the diff. In practice this is the one that turns mutation testing from a nightly marathon into a per-PR check, and it is where the tools differ most.

On that last point, be precise about which tool offers what, because the flags are not portable. `cargo-mutants` has `--in-diff`, which takes a unified diff and only generates mutants inside the changed regions. Stryker.NET has `since`, which uses git information to test only changes since a given target and reports only on mutants within the changed code. StrykerJS does not have a `since` option — its levers are incremental mode, which stores results in a file to speed up the next `--incremental` run, plus a `mutate` glob you build from the diff yourself. PIT's version is incremental analysis through a history file, skipping re-analysis of mutations killed by unchanged tests in unchanged classes, previously detected infinite loops, and survivors with no new covering tests. `mutmut` re-tests only mutants in functions whose source changed.

And the tools are commendably honest about what diff-scoping costs you. PIT marks incremental analysis experimental and notes that its central assumption — that tracking a class plus its superclasses and outer classes is enough, when behaviour also depends on dependencies — is currently unproven. `cargo-mutants` says plainly that an edit in one region can cause code in a different region or file to no longer be well tested, so incremental runs give faster feedback but are not a substitute for a full run. That is not a reason to skip diff-scoping. It is the shape of a sane policy: diff-scoped on every PR as a gate, full-corpus on a schedule where nobody is waiting.

#!/usr/bin/env bash
# Per-PR mutation run: mutants inside the diff only. The full-corpus run is a
# SEPARATE, slower job. Diff-scoped is faster feedback, not a substitute --
# cargo-mutants' own docs are blunt that an edit in one region can leave code
# in a different region no longer well tested, and PIT labels its incremental
# analysis experimental on an assumption it calls unproven.
set -euo pipefail

BASE="${GITHUB_BASE_REF:-main}"
git fetch --no-tags --depth=200 origin "$BASE"

# --- 1. Baseline. Every timeout threshold below is a multiple of this number,
#        so measure it in the SAME kind of guest the mutants will run in. A
#        baseline measured on a noisy shared runner makes every threshold you
#        derive from it noise as well.
#        And it must be GREEN. A red suite kills every mutant for free.
start=$(date +%s)
cargo test --no-fail-fast
BASELINE=$(( $(date +%s) - start ))
echo "baseline suite: ${BASELINE}s"

# --- 2. Rust. --in-diff takes a unified diff and only generates mutants in the
#        changed regions. Note it matches against code under test, not test
#        code: a commit that only edits tests produces no mutants at all.
#        Default timeout is 5x baseline with a 20s floor. --build-timeout is
#        separate, because a mutated const expression can hang rustc rather
#        than the tests.
#        --jobs 1: concurrency belongs at the FLEET level, one guest per
#        batch. Stacking jobs inside one guest makes the baseline multiple lie.
git diff "origin/${BASE}...HEAD" > /tmp/pr.diff
cargo mutants \
  --in-diff /tmp/pr.diff \
  --timeout-multiplier 5 \
  --minimum-test-timeout 20 \
  --build-timeout 300 \
  --jobs 1 \
  --output target/mutants-pr

# --- 3. JVM. PIT's diff story is incremental analysis through a history file;
#        use --withHistory for the easy path, or historyInputFile /
#        historyOutputFile when you want to control where it lands. Scope the
#        run with targetClasses too, built from the diff.
#        timeoutFactor x normal-time + timeoutConstant is the whole timeout
#        rule; raise the constant when JVM class loading causes false
#        TIMED_OUT verdicts, which the PIT FAQ warns about explicitly.
CHANGED=$(git diff --name-only "origin/${BASE}...HEAD" -- '*.java' \
  | sed 's|^src/main/java/||; s|\.java$||; s|/|.|g' | paste -sd, -)
if [ -n "$CHANGED" ]; then
  mvn -q org.pitest:pitest-maven:mutationCoverage \
    -DtargetClasses="$CHANGED" \
    -DwithHistory=true \
    -DtimeoutFactor=1.5 \
    -DtimeoutConstant=8000
fi

# --- 4. JS/TS. StrykerJS has NO --since option; `since` is Stryker.NET's.
#        Do not copy the flag across runtimes -- for StrykerJS the levers are
#        incremental mode (persist reports/stryker-incremental.json between CI
#        runs) plus a --mutate glob you build from the diff yourself.
#        coverageAnalysis: "perTest" is the big one: only the tests covering
#        each mutant run. It needs tests that work independently and in random
#        order; static mutants fall back to the whole suite.
MUTATE=$(git diff --name-only "origin/${BASE}...HEAD" -- 'src/*' \
  | grep -E '\.(ts|js)$' | grep -v -E '\.(spec|test)\.' | paste -sd, -)
if [ -n "$MUTATE" ]; then
  npx stryker run --incremental --mutate "$MUTATE" --concurrency 4
fi

Every mutant run is a deliberately broken build

Here is the sentence that should change how you provision this. You are executing thousands of programs that you have personally arranged to be incorrect, at maximum concurrency, and then you need every single one of them to die cleanly with a verdict attached.

Broken code does things healthy code does not. Not as an edge case you could grep for — as the mutation operators working exactly as designed.

• Infinite loops. The canonical mutant. Delete the decrement from a loop body, or mutate a termination predicate so it never fires, and you have one. The `cargo-mutants` docs use precisely this example: flip a `should_stop()` to always return false and execution hangs forever.
• Memory exhaustion. Mutate a size check, a chunk constant, a growth guard, and the allocation that was bounded is not. PIT reports a `MEMORY_ERROR` status for a reason, and it is not a theoretical one.
• Filesystem damage. The operator that removes a void method call is perfectly capable of removing the call that was keeping something destructive from happening, and the operator that removes a conditional is perfectly capable of removing `if (dryRun)`. A mutated path constant and a mutated cleanup guard are the same class of event. Nobody writes a test asserting that `rm -rf` was not called on the wrong directory.
• Unexpected network traffic. Mutate a retry limit or a backoff constant and a polite client becomes a load generator aimed at whatever the test configuration points at — which, in a depressing number of repositories, is not as fake as everyone assumes.
• Collisions with the neighbours. Ports, temp directories, lock files, a database the suite truncates on setup. One run of a suite is fine with these. Thousands of runs at maximum concurrency on a shared host are not, and the failures look like flakiness rather than like what they are.

The right shape for this is a guest per mutant batch with a hard RAM ceiling and a kill that does not negotiate. On our stack the ceiling is structural rather than advisory: Firecracker cannot change guest RAM at snapshot restore, so the number is chosen once at template-build time with `--memory-mb` and the guest simply cannot exceed it. A mutant that allocates without bound hits a wall that is not a cgroup limit someone can be talked into raising mid-incident. Each sandbox also gets its own network namespace out of a pool of 16,384 pre-allocated /30 subnets, so a mutated client hammering port 8080 is hammering its own port 8080, and a mutant that writes outside its temp directory is writing inside a filesystem that evaporates.

The alternative is a shared runner, and the shared-runner version of this exercise is how you discover that your CI fleet has no timeout. You find out at 03:00, from a graph, about a job that has been burning a core since Tuesday because somebody mutated a `while`.

A timeout is a verdict, not an ergonomics setting

This is the part that makes isolation a correctness requirement rather than a hygiene preference. If you cannot distinguish a mutant that hung from a mutant that survived, the infinite-loop mutant gets scored as survived, and you spend an afternoon hunting a test gap that does not exist. The suite was fine. The mutant simply never finished telling you so.

Every serious tool therefore treats timeout as its own status alongside killed and survived, and derives the threshold from a measured baseline of the unmutated suite rather than from a number somebody liked:

Notice the common structure: a multiple of a measured baseline, plus an absolute buffer for fixed overhead. Which quietly makes your baseline measurement load-bearing. Measure the baseline on a noisy shared runner alongside four other jobs and every threshold you derive from it is noise with a decimal point. Measure it inside a guest that is identical to the guests the mutants will run in, and the multiple means something — which is a second, less obvious argument for the per-mutant-guest shape.

Fuzzing mutates inputs; this mutates the program

The closest cousin to this workload is fuzzing, and the distinction is worth one crisp paragraph because the two get conflated constantly. A fuzzer holds the program fixed and searches the input space for an input that makes correct-looking code misbehave; its oracle is external — a crash, a sanitiser report, an assertion. Mutation testing does the opposite: it holds the inputs fixed, because your test suite IS the input, and searches the space of slightly-wrong programs for one that your inputs fail to reject. Its oracle is your own assertions, which is why it is the only technique on this list that is testing the tests rather than the code. Fuzzing audits the program. Mutation testing audits the thing you were going to trust instead of reading the program. They share exactly one engineering property, and it is the subject of this post: both run large numbers of executables you expect to behave badly, and both need an environment where badly-behaved is boring.

Four techniques, four different questions

These are not ranked and they do not substitute for each other. Coverage tells you where to look. Mutation testing tells you whether what you found is actually defended. Property-based testing is frequently how you write the assertion that kills the mutant. Fuzzing is how you find out your parser holds a different opinion from your spec. The one relationship worth internalising: a surviving mutant is a prompt for a property. If you cannot write an assertion that fails when `>` becomes `>=`, there is a reasonable chance you do not yet know what that function is for.

The shape on PandaStack

Two artefacts and one fan-out. Bake a template whose image already contains the repository's resolved dependencies and a clean build. Create one guest from it, run the unmutated suite once to get a warm process and a baseline number, and snapshot that. Then restore one guest per mutant batch from the snapshot.

# One bake, paid once, for every guest in every mutation run after it. A run
# that resolves dependencies per mutant is paying for the network thousands of
# times for an answer that does not change.
cat > Dockerfile <<'DOCKERFILE'
FROM pandastack/base:latest

COPY Cargo.toml Cargo.lock /work/
WORKDIR /work
RUN cargo fetch

# A clean baseline build, at bake time. cargo-mutants needs one to diff
# against, and every restored child inherits it.
COPY . /work
RUN cargo build --tests --locked

# Docker ENV does NOT reach a detached process in the Firecracker guest --
# only /etc/environment (PAM) does. Put anything the mutant runner needs there.
RUN printf '%s\n' \
      'CARGO_TERM_COLOR=never' \
      'CARGO_NET_OFFLINE=true' \
      'RUST_BACKTRACE=1' >> /etc/environment
DOCKERFILE

# --memory-mb is the ONLY place guest RAM is chosen. Firecracker cannot change
#   vCPU or RAM at snapshot restore, so cpu= / memory_mb= on a create are
#   silently corrected to the baked values. This number is also your OOM fence
#   for a mutant that allocates without bound, so pick it on purpose.
# --size-mb (rootfs) DEFAULTS TO 1024 MB. No real toolchain fits in that.
# --cpu is deprecated and ignored; every template gets 8 burstable vCPUs.
pandastack template build \
  --name mutants-rs \
  -f Dockerfile \
  --context . \
  --size-mb 16384 \
  --memory-mb 4096

Once the template's snapshot exists, a create is a snapshot restore rather than a boot: p50 179 ms, p99 203 ms, of which the `/snapshot/load` step itself is around 49 ms. The very first create of a template before its snapshot exists is a real cold boot at roughly 3 seconds, after which the snapshot is captured automatically and every create afterwards takes the fast path. For a workload whose entire character is "start thousands of short-lived machines," the restore cost is the thing you were going to be bottlenecked on, and it is the term this architecture exists to shrink.

from pandastack import Sandbox

sbx = Sandbox.create(template="mutants-rs", ttl_seconds=1800)

# Run the suite ONCE, unmutated, in a guest from this template. Two artefacts
# come out of it: the baseline number every timeout threshold is a multiple
# of, and a warm process + page cache that every restored child inherits.
rc = sbx.exec_stream(
    "cd /work && /usr/bin/time -f 'BASELINE %e' cargo test --no-fail-fast",
    on_stdout=lambda c: print(c, end=""),
    timeout_seconds=1800,
)
assert rc == 0, "the baseline suite must be GREEN before you mutate anything"

# snapshot() captures memory AND disk, and returns an ID *string*.
WARM_SNAPSHOT = sbx.snapshot()
print("warm snapshot:", WARM_SNAPSHOT)

# DO NOT kill() this sandbox. An explicit delete cascade-deletes that
# sandbox's snapshots -- including the one we just took, which the entire rest
# of this pipeline restores from. The idle reaper deliberately does NOT
# cascade, so let the ttl_seconds idle timeout take the guest instead.
# sbx.kill()   # <- this line would delete WARM_SNAPSHOT. Leave it commented.

"""Fan a mutation run out across restored microVM guests.

One baked template, one warm snapshot, N restores. Each guest gets a BATCH of
mutants rather than a single mutant -- a restore is p50 179 ms, which is cheap
but not free, and a batch amortises even that. The PER-MUTANT bound is a shell
`timeout` inside the runner, because a hanging mutant has to become a VERDICT
rather than a wedged guest.
"""
import json
import math
from concurrent.futures import ThreadPoolExecutor, as_completed

from pandastack import Sandbox

WARM_SNAPSHOT = "snap_..."    # from the warm-up above
BASELINE_SECONDS = 42         # measured in a guest from the SAME template
TIMEOUT_MULTIPLIER = 5        # cargo-mutants' default; choose yours on purpose
MINIMUM_TIMEOUT = 20          # floor, for suites fast enough that 5x is noise
BATCH = 25
FLEET = 32                    # concurrent guests

PER_MUTANT_TIMEOUT = max(
    MINIMUM_TIMEOUT, math.ceil(BASELINE_SECONDS * TIMEOUT_MULTIPLIER)
)


def run_batch(index: int, mutants: list[dict]) -> dict:
    # from_snapshot restores memory AND disk: deps present, baseline build
    # done, suite already warm. ttl_seconds is an IDLE timeout (the reaper
    # measures time since last activity), not a walltime budget.
    sbx = Sandbox.create(
        from_snapshot=WARM_SNAPSHOT,
        metadata={"job": "mutants", "batch": str(index)},
        ttl_seconds=900,
    )
    sbx.filesystem.write("/work/batch.json", json.dumps(mutants))

    chunks: list[str] = []
    # timeout_seconds is CLIENT-side only -- the agent decodes it on both exec
    # endpoints and applies it on neither. On exec_stream it raises the HTTP
    # timeout; one-shot exec() aborts at 30s. The ONLY real deadline is the
    # one in the shell, which is why every mutant is wrapped below.
    # run-mutant-batch.sh wraps each individual mutant in:
    #     timeout --kill-after=10s "$PER_MUTANT_TIMEOUT" <test command>
    # and emits one JSON line per mutant with status in
    # {killed, survived, timeout, unviable, no_coverage}.
    sbx.exec_stream(
        f"cd /work && PER_MUTANT_TIMEOUT={PER_MUTANT_TIMEOUT} "
        f"./run-mutant-batch.sh batch.json",
        on_stdout=chunks.append,
        timeout_seconds=PER_MUTANT_TIMEOUT * len(mutants) + 120,
    )

    verdicts = [
        json.loads(line)
        for line in "".join(chunks).splitlines()
        if line.startswith("{")
    ]
    survivors = [v for v in verdicts if v["status"] == "survived"]

    evidence = None
    if survivors:
        # Hand a survivor back as something a human can restore and poke at
        # six weeks from now: the exact guest, with the exact broken code.
        evidence = sbx.snapshot()
        # DO NOT kill() here. An explicit kill() cascade-deletes that
        # sandbox's snapshots -- a tidy try/finally would destroy the evidence
        # we just captured. Let the idle reaper take the guest; it does not
        # cascade.
    else:
        sbx.kill()

    return {"batch": index, "verdicts": verdicts, "evidence": evidence}


def main(mutants: list[dict]) -> None:
    groups = [
        (i // BATCH, mutants[i:i + BATCH]) for i in range(0, len(mutants), BATCH)
    ]

    # NOT fork(): fork() is DISK-ONLY, so children cold-boot and lose the warm
    # suite. NOT fork_tree() either -- it does carry memory, but it caps at 16
    # children and pins them to the parent's host, which is the opposite of
    # what a thousand-mutant fan-out wants. snapshot() + create(from_snapshot=)
    # goes through the scheduler and spreads across hosts.
    results = []
    with ThreadPoolExecutor(max_workers=FLEET) as pool:
        futures = [pool.submit(run_batch, i, g) for i, g in groups]
        for f in as_completed(futures):
            results.append(f.result())

    flat = [v for r in results for v in r["verdicts"]]
    tally: dict[str, int] = {}
    for v in flat:
        tally[v["status"]] = tally.get(v["status"], 0) + 1
    print(tally)

    # The score is the boring output. The survivor list is the product.
    for r in results:
        for v in r["verdicts"]:
            if v["status"] == "survived":
                print(f"SURVIVED {v['file']}:{v['line']}  {v['operator']}"
                      f"  restore={r['evidence']}")

The per-mutant guest buys you one more thing that is easy to undervalue until the first time you need it: reproducibility. A surviving mutant is a finding somebody will have to act on, possibly weeks later, possibly not the person who ran the job. "Line 412, the comparison-boundary operator, survived" is a decent bug report. A snapshot ID that restores the exact guest with the exact broken code and the exact suite that failed to notice is a better one, and it costs you a `snapshot()` call on the batches that found something.

What this does not fix

Everything above is the good news, and a post that stopped there would be an advert. Several of these are structural.

• A red or flaky baseline invalidates the entire run. If the suite is already failing, every mutant is killed for free and your score is a fiction. Worse, if a test is nondeterministic at all, it kills mutants at random, and you get a mutation report whose verdicts are partly a coin flip — with no indication of which ones. Quarantine your flaky tests before you mutate anything; it is a prerequisite, not an adjacent nice-to-have.
• Guest RAM is a template decision, not a per-run one. Firecracker cannot change vCPU or RAM at snapshot restore, so a create's `cpu` and `memory_mb` are silently corrected to the baked values. If your suite needs 16 GiB, that is a template baked with `--memory-mb 16384`, not an argument you pass at submit time. This is the same mechanism that makes the OOM fence trustworthy, so it is a trade rather than a straight loss — but it does mean maintaining a small family of templates instead of treating memory as a parameter.
• Parallelism changes the wall clock, not the bill. Four thousand mutants is four thousand mutants' worth of CPU whether you run them over a weekend or over forty minutes. Our rate card is $0.054 per active vCPU-hour and $0.0162 per working-set GiB-hour, metered per second, so a mutation run is a visible line item and diff-scoping is a cost control as much as a latency one.
• The guest clock is frozen at bake time and re-synced best-effort on restore, resume and wake. If your suite talks TLS, the stale-clock failure mode is "certificate not yet valid" rather than "expired" — a test failure that looks like a mutant kill and is not. Worth knowing before you debug it from first principles at scale.
• Mutation testing cannot find a bug no operator generates. It will not tell you the requirement was wrong, the feature is missing, the concurrency is broken, or the design is unserviceable. It audits the suite against a catalogue of small local wrongness, and that catalogue is finite and published.

Which brings us to the number itself. A mutation score is not a metric to put on a dashboard and chase to 100%. It cannot reach 100%, for two unavoidable reasons: equivalent mutants are undetectable by construction, and plenty of code is untested on purpose — logging, generated code, debug helpers, the branch you left in for the next migration. A team incentivised to push the number up will write tests that assert implementation details, because those are the tests that kill mutants cheapest, and implementation-detail tests are how a codebase becomes expensive to change. If you must gate on it, gate on the diff's own score and on specific surviving mutants in code you care about, never on the corpus-wide percentage.

Read the survivor list, not the score. Each entry is a place where you were trusting something that nothing was checking, which is a far more actionable sentence than any percentage.

Mutation testing does not audit your code. It audits the thing you were planning to trust instead of reading your code.

Run it on the diff, in guests you can kill without asking permission, with a timeout derived from a baseline you measured somewhere quiet. Then go and read what survived.