Testing Against Ten Toolchains Without Ten Broken Runners

If you ship a library, you don't have a test suite. You have a matrix. Python 3.9 through 3.13. Node 18, 20, 22. Go 1.21 through 1.25. glibc and musl. GCC and Clang. arm64 and x86-64. Nobody wants this — it's the tax you pay for other people depending on you, and the number of cells multiplies faster than your patience does. The uncomfortable part is that most matrices are less trustworthy than they look. A green grid on a shared CI runner is often measuring one environment wearing several costumes.

I'm Ajay, I build PandaStack. This post is about why matrix legs contaminate each other on shared runners, why containers fix most of that but structurally cannot fix the part you're testing when your matrix includes kernels and libcs, and what a microVM per leg actually buys you. It is also about the limit: no amount of virtualization turns an x86-64 host into an arm64 one, and I'll be specific about where the honest boundary sits.

What a shared runner does to a matrix

The default shape is one runner image with every toolchain preinstalled and a version manager to switch between them. It works right up until the point where you're relying on it. Here's what leaks, roughly in order of how often it has cost me an afternoon.

• Global package caches. pip's wheel cache, npm's ~/.npm, Go's module and build cache, Cargo's registry. Leg one populates them; leg two resolves against a cache built by a different interpreter or a different compiler flag set. The classic outcome: a wheel built against 3.11 headers gets reused by the 3.12 leg because the cache key wasn't as specific as you assumed, and your 3.12 leg is now testing a binary that will never exist on a user's machine.
• PATH shadowing. Two version managers, or a version manager plus the distro's packages, plus whatever a previous job did with 'npm i -g'. Whichever shim wins the PATH race defines your test run. You find out because 'python --version' in your logs disagrees with the interpreter that actually imported your module.
• Mutating installs. 'pip install -e .' and 'npm link' change the machine, not just the job. If your legs run sequentially on the same runner, leg N+1 inherits leg N's editable install and cheerfully imports the wrong source tree.
• Orphaned daemons. A leg starts Postgres, a mock server, a headless browser, and exits without reaping it. The next leg binds the same port, fails with EADDRINUSE, or — the worse case — connects successfully to the previous leg's process and passes against stale data.
• One leg's OOM killing a sibling. The kernel OOM killer selects a victim by score, not by fairness. A memory-hungry compile in the Clang leg can get your Node leg SIGKILLed, and the log you get is a bare exit code 137 with no explanation attached to the leg that actually caused it.
• Timing contamination. Ten legs on one box means ten legs sharing page cache, disk queue and CPU. Any test with a timeout, a retry budget, or a benchmark assertion is now measuring your CI scheduler.

Containers get you most of the way, and then stop

One container image per leg fixes a genuine majority of that list. Separate filesystems mean separate caches, separate PATHs, separate site-packages. Separate PID namespaces mean an orphaned daemon dies with its leg. If your matrix is 'five Python versions, pure Python code', containers are the correct answer and this post is over — go use them, they're cheaper and simpler and you already know how.

The problem is the part of the matrix that isn't about userspace. Containers share the host kernel. That's the whole design. So the moment a matrix dimension is a kernel property, containers cannot represent it honestly:

• Kernel-version-dependent behaviour. If your library has a fast path gated on a syscall or a flag that landed in a particular kernel release, every container leg on that host reports the host's kernel. You can label a leg 'kernel 5.10' in your matrix config; the code will still see whatever the runner is running.
• io_uring, seccomp, eBPF, and anything else that negotiates capabilities with the kernel. These are exactly the features where behaviour varies across kernel versions and where the runner's seccomp profile may block the syscall you're trying to test. A container leg tells you what the CI host's kernel allows, not what your users' kernels allow.
• Anything that reads or writes /proc, /sys, module state, or sysctls. A container gets a partly-masked view of the host's, and writing one changes it for every sibling leg on the box. That is both an isolation failure and a correctness failure in the same move.
• cgroup-visible resource limits. Code that sizes a thread pool or an arena from what it believes is 'available memory' reads something different inside a container than on a real machine, so your memory-pressure tests measure the container runtime's accounting rather than your allocator.
• The boundary itself. A container is a polite suggestion to the kernel: a set of namespaces and cgroups the kernel agrees to honour. It's a strong suggestion, and for first-party code it's usually enough — but the failure mode is a shared kernel, and a shared kernel is one bug away from not being a boundary.

A Firecracker microVM doesn't share the leaking thing. Each PandaStack sandbox is a real VM with its own guest kernel, its own /proc, its own sysctls, its own full port space, and its own filesystem behind a hardware virtualization boundary. If the glibc leg tunes a sysctl or the io_uring leg wedges its ring, it does so inside a VM that exists for ninety seconds and then doesn't. The related mechanics — why a shared kernel leaks under parallelism — are covered in the parallel-test-isolation piece linked below; here the point is narrower: a matrix dimension is only real if the environment can actually differ along it.

The fan-out: one sandbox per leg, in parallel

The control loop is boring, which is the compliment. Define the matrix as data, spawn one sandbox per leg, install the pinned toolchain inside it, run the suite, read the exit code, kill the VM. Nothing is shared, so nothing needs coordinating, so the whole thing is a thread pool and a list comprehension. Set PANDASTACK_API_KEY in your environment first.

from concurrent.futures import ThreadPoolExecutor
from pandastack import Sandbox

REPO = "https://github.com/acme/widget.git"
SHA  = "a1b2c3d4"   # pin the commit, not the branch

# The matrix as data. Each leg names the exact toolchain it wants; nothing
# is inherited from a runner image that someone else maintains.
MATRIX = [
    {"id": "py39",  "tools": "python@3.9.19",  "test": "pytest -q"},
    {"id": "py310", "tools": "python@3.10.14", "test": "pytest -q"},
    {"id": "py311", "tools": "python@3.11.9",  "test": "pytest -q"},
    {"id": "py312", "tools": "python@3.12.4",  "test": "pytest -q"},
    {"id": "py313", "tools": "python@3.13.0",  "test": "pytest -q"},
    {"id": "node18", "tools": "node@18.20.4",  "test": "npm test"},
    {"id": "node20", "tools": "node@20.16.0",  "test": "npm test"},
    {"id": "node22", "tools": "node@22.6.0",   "test": "npm test"},
    {"id": "go121",  "tools": "go@1.21.13",    "test": "go test ./..."},
    {"id": "go125",  "tools": "go@1.25.0",     "test": "go test ./..."},
]

def run_leg(leg: dict) -> dict:
    # A fresh microVM per leg: own guest kernel, own /proc, own port space,
    # own package caches. Created from a baked snapshot, ~179ms at p50.
    sbx = Sandbox.create(template="base", ttl_seconds=1800)
    try:
        sbx.exec(
            f"git clone --depth 1 {REPO} /work && cd /work "
            f"&& git fetch --depth 1 origin {SHA} && git checkout {SHA}"
        )
        # Toolchain is installed INSIDE the leg. No shim races, because
        # there is exactly one toolchain on this machine.
        setup = sbx.exec(
            f"cd /work && mise use --global {leg['tools']} && mise install && mise reshim",
            timeout_seconds=600,
        )
        if setup.exit_code != 0:
            return {"leg": leg["id"], "ok": False, "stage": "setup", "log": setup.stderr}

        run = sbx.exec(f"cd /work && {leg['test']}", timeout_seconds=1800)
        return {
            "leg": leg["id"],
            "ok": run.exit_code == 0,
            "stage": "test",
            "log": run.stdout + run.stderr,
        }
    finally:
        sbx.kill()   # caches, daemons, editable installs — all gone

with ThreadPoolExecutor(max_workers=len(MATRIX)) as pool:
    results = list(pool.map(run_leg, MATRIX))

for r in results:
    print(("PASS" if r["ok"] else "FAIL"), r["leg"])
raise SystemExit(0 if all(r["ok"] for r in results) else 1)

Two things are load-bearing here and neither is the parallelism. First, the toolchain install happens inside the leg, so there is exactly one Python on the machine and the shim-ordering question doesn't exist. Second, teardown is 'kill the VM'. There is no cleanup step to forget, because the thing you would have cleaned up ceases to exist. A leg that leaves a daemon running, fills its disk, or corrupts its own package cache has done all of that to a machine with a ninety-second lifespan.

Width is bounded by host memory and CPU rather than by isolation. Each agent pre-allocates 16,384 /30 subnets, so networking isn't the ceiling; the baked 'base' template is 4 GiB of RAM and 8 vCPU, so ten legs in flight is a memory question you can answer with arithmetic. That's a much better conversation than 'how many legs can I run before they start lying to each other'.

Fork from a warmed toolchain instead of re-downloading it

The obvious objection to the loop above is that every leg redownloads a toolchain, and 'mise install' for a Python build is not fast. Correct, and the fix is snapshot-and-fork. Do the expensive install once per toolchain family, freeze it, and have each leg fork the frozen state. A same-host fork is roughly 400-750ms and shares memory and rootfs copy-on-write, so the fork doesn't recopy the interpreter — it shares pages with the snapshot until it writes to them.

from pandastack import Sandbox

# ---- Once per toolchain (cache-key it on your .tool-versions hash). ----
def bake(tools: str, deps: str):
    base = Sandbox.create(template="base", ttl_seconds=1800)
    base.exec(f"mise use --global {tools} && mise install && mise reshim",
              timeout_seconds=900)
    base.exec("git clone --depth 1 https://github.com/acme/widget.git /work")
    base.exec(f"cd /work && {deps}", timeout_seconds=900)   # the slow part
    snap = base.snapshot()      # interpreter + deps, frozen
    base.kill()
    return snap

SNAPS = {
    "py312": bake("python@3.12.4", "pip install -e '.[test]'"),
    "node20": bake("node@20.16.0", "npm ci"),
}

# ---- Per leg: fork, sync the source, run. ----
def run_leg(snap_key: str, sha: str, cmd: str) -> bool:
    sbx = SNAPS[snap_key].fork()       # ~400-750ms, copy-on-write
    try:
        sbx.exec(f"cd /work && git fetch --depth 1 origin {sha} "
                 f"&& git checkout {sha}")
        return sbx.exec(f"cd /work && {cmd}", timeout_seconds=1800).exit_code == 0
    finally:
        sbx.kill()

The important property is not the speed, it's that every leg in a family starts from byte-identical state. When the 3.12 leg fails and the 3.11 leg passes, the difference is the interpreter, because you removed every other variable by construction. That's what makes a matrix diagnostic rather than merely decorative.

Make 'green' falsifiable: hash the resolved versions

A matrix result is a claim: 'this commit works on Python 3.12'. Claims should be checkable. The cheap way to make it checkable is to have every leg emit what it actually resolved — not what the config asked for — and hash it into the result. The two numbers disagreeing is exactly the bug class this whole exercise exists to catch.

#!/bin/sh
# provenance.sh -- runs INSIDE the leg's microVM, after install,
# before the test command. Records what the environment resolved to,
# not what the matrix config asked for.
set -eu

mise install >/dev/null
mise reshim

{
  echo "leg=$LEG_ID"
  echo "commit=$COMMIT_SHA"
  mise ls --current 2>/dev/null || true
  command -v python3 >/dev/null && python3 -VV
  command -v node    >/dev/null && node --version
  command -v go      >/dev/null && go version
  cc --version 2>/dev/null | head -1 || true
  # libc identity: glibc prints a version banner, musl prints to stderr.
  ldd --version 2>&1 | head -1
  uname -srm
} > /work/toolchain.txt

sha256sum /work/toolchain.txt | cut -d' ' -f1 > /work/toolchain.sha
cat /work/toolchain.txt

Read /work/toolchain.txt and /work/toolchain.sha back out through the SDK and attach them to the leg's result. Now a green matrix carries evidence. If someone edits the matrix config to add 'python 3.13' and the shim quietly serves 3.12, the hash for that leg is identical to a sibling's and you can assert on it in CI. The version-manager and PATH failure modes from the first section stop being invisible: they become a mismatched hash.

The same file is useful when a user reports a bug. 'It passed our matrix' is worth very little; 'it passed on this exact interpreter build, this libc, this compiler, this kernel' is a starting point for a bisect.

The caveat: architecture is not a software setting

Everything above is honest for toolchain versions, libcs, compilers and kernel-visible behaviour. It is not honest for architecture, and I'd rather say so plainly than let a matrix imply something it can't deliver.

A Firecracker microVM runs guest code natively on the host CPU. An arm64 guest needs an arm64 host; an x86-64 guest needs an x86-64 host. If your matrix includes both, you need agent hosts of both architectures and you schedule legs onto the right ones. There is no configuration flag that makes this go away, and any platform that implies otherwise is emulating.

The practical version: run the full matrix on your primary architecture on every commit, and run a reduced but native matrix on the secondary architecture on a slower cadence — nightly, or on release branches. That's a real tradeoff with a real cost, and it's still better than a grid of emulated cells that tells you what you want to hear.

Shared runner vs. container per leg vs. microVM per leg

• Package caches — Shared runner: global and cross-contaminating. Container per leg: private per image. MicroVM per leg: private per VM, and destroyed with it.
• PATH and version managers — Shared runner: shims race, wrong interpreter wins silently. Container per leg: one toolchain per image, mostly solved. MicroVM per leg: one toolchain per machine, plus a resolved-version hash you can assert on.
• Orphaned daemons and ports — Shared runner: leak into the next leg. Container per leg: die with the PID namespace, but host-port mappings still collide. MicroVM per leg: full private port space, every leg can bind :8080.
• Kernel-dependent behaviour — Shared runner: one kernel for everything. Container per leg: still one kernel, so the leg label is aspirational. MicroVM per leg: real per-leg guest kernel, so io_uring, seccomp, eBPF and /proc tests mean something.
• OOM blast radius — Shared runner: the kernel picks a victim that may be a sibling leg. Container per leg: cgroup limits help, host pressure still crosses over. MicroVM per leg: memory is allocated to the VM, so a leg OOMs itself.
• Isolation boundary — Shared runner: none. Container per leg: namespaces and cgroups, one kernel bug from gone. MicroVM per leg: hardware virtualization, the model Lambda and Fargate are built on.
• Cost of a clean start — Shared runner: a cleanup script that's as correct as its last edit. Container per leg: image pull plus install. MicroVM per leg: ~179ms create from a baked snapshot, or ~400-750ms to fork a warmed toolchain.
• Architecture — Shared runner: whatever the runner is. Container per leg: whatever the host is, unless you emulate. MicroVM per leg: whatever the host is, no emulation available — you need real hosts per architecture.

When this is the wrong answer

I'd rather you didn't adopt this by default. The honest scope:

• Pure-Python or pure-JS libraries with no native extensions. Your matrix dimension is the interpreter and nothing else. Containers are cheaper, simpler, and already correct for this. Use them.
• A matrix that's already green and already trustworthy. If your legs are independent, your caches are keyed properly, and nobody has ever debugged a cross-leg contamination bug on your team, you've solved it another way. Don't rebuild it.
• Very short legs. If a leg is a four-second unit test, provisioning dominates and you're measuring your platform. Group the fast legs into one VM per toolchain and keep a VM per leg for the slow, stateful, port-binding ones.
• Matrices whose real variable is an external service. If leg one and leg two both hit the same staging API, isolating your side changes nothing — the shared thing is on the other end of the socket.
• Anything where the constraint is genuinely CPU architecture and you don't have hosts of that architecture. A microVM won't invent an arm64 core for you.

The case for a microVM per leg is strongest when your matrix crosses libcs, compilers, kernel-visible syscall behaviour, or native extensions built against version-specific headers — the cells where 'the environment leaked' and 'the code is broken' produce indistinguishable failures. Give each leg its own kernel and its own disk, fork it from a warmed toolchain so the clean start is cheap, hash what it actually resolved so the green cell is falsifiable, and be honest in your README about which architectures you tested natively. That's a matrix that means something. The alternative is ten costumes and one actor, and your users will notice before you do.