Cross-Compilation Build Farms: One MicroVM Per Target

There is a moment in a project's life where the build stops being a command and becomes a matrix. Someone opens an issue asking for an arm64 build. Someone on Alpine reports that your binary exits with "not found" — which is the dynamic loader's charmingly unhelpful way of saying the interpreter path baked into the ELF header does not exist on a musl system, and not, as everyone first assumes, that the file is missing. Someone on a four-year-old LTS distro reports "GLIBC_2.38 not found". A Windows user asks politely. And the one artifact you had is now five, or fifteen, and each cell of that grid is a slightly different engineering problem wearing the same name.

The two answers everybody reaches for both hurt, in opposite directions. The first is emulation: register QEMU's user-mode binaries with the kernel's binfmt_misc handler and run a foreign-architecture build as if it were local, which is what a docker buildx build with a foreign --platform is doing under the hood on a single-architecture machine. It is beautifully convenient, it is slow in a way that reshapes your CI budget, and in a small number of important cases it quietly produces a green result that a real machine would have failed. The second is native runners per architecture: fast, honest, and now you are maintaining two or four fleets whose toolchains drift apart until the arm64 image has a compiler nobody has upgraded since the migration.

This post is about the third option, and about the distinction most matrix discussions skip entirely: cross-compiling and emulating are not two implementations of the same idea. They fail differently, they lie differently, and the pattern that works — a farm of native-architecture microVMs, one snapshot-baked template per target triple, a fresh guest per build job — is built out of knowing which cells are which.

The matrix nobody asked for

Start by writing down what actually varies, because "we support Linux and Mac" is not a specification and the gap between that sentence and a shipping artifact is where all the pain lives. The axes are not what most teams think they are — architecture is only the first one, and it is not the one that generates the most support tickets.

• CPU architecture. x86_64 and aarch64 for almost everyone; add riscv64, armv7 or ppc64le if your users are somewhere interesting. This is the axis that cannot be faked without either a foreign toolchain or a translator, and it is the axis everyone plans for.
• Operating system. Linux, macOS, Windows. Different executable formats (ELF, Mach-O, PE), different linkers, different system libraries, and in two of those three cases a licence agreement that constrains where you are allowed to build.
• The C library. glibc versus musl on Linux is a genuine ABI boundary, not a packaging preference. A glibc binary asks for /lib/ld-linux-*.so and a musl binary asks for /lib/ld-musl-*.so, and neither one is going to find the other's loader.
• The glibc VERSION floor — the axis that surprises people. glibc uses symbol versioning, so a binary linked on a host with glibc 2.39 records that it needs GLIBC_2.39 symbols and will refuse to start on a system with 2.28. Nothing warns you at build time. You find out from a user.
• Linkage. Static versus dynamic changes the artifact's portability, its size, its licence obligations, and whether NSS-based hostname resolution behaves the way you expect on a fully static glibc build (it does not, reliably).
• The CPU feature baseline. Compiling with -march=native on a build host that happens to have AVX-512 produces a binary that dies with an illegal instruction on a customer's older machine. On arm the equivalent is quietly assuming ARMv8.2 atomics or SVE.
• Per-language ABI axes stacked on top of all of the above. Python native extensions carry an interpreter ABI tag and a manylinux or musllinux platform tag; Node native addons carry a module ABI unless you stick to N-API; a Go binary with cgo enabled is a different build problem from the same source with CGO_ENABLED=0.

The multiplication is not really the problem. Fifteen cells is fine; computers are good at fifteen of things. The problem is that the cells are not the same kind of object. Some of them are a pure compile you can do anywhere. Some of them require code to execute during the build, which means they require a machine that can execute that code. And at least one of them requires a machine you are not allowed to rent from a Linux provider. A matrix that pretends all its cells are interchangeable is a matrix that will eventually ship someone a broken binary.

Cross-compiling is not emulation

These get used interchangeably in conversation and they are not remotely the same mechanism. Getting the distinction crisp is what lets you decide, per cell, which tool is honest.

What cross-compiling actually is

A cross-compiler is a program that runs on one machine and emits code for another. The GNU build system has three names for the machines involved and they are worth internalising: the build machine is where the compiler is being compiled, the host machine is where the compiler will run, and the target machine is what the compiler emits code for. For everyday purposes you care about the last two — your toolchain runs on x86_64 and produces aarch64. Nothing is translated at runtime, because there is no runtime involved: the compiler is just a program writing bytes into a file, and it does not need to be able to execute those bytes any more than a printing press needs to be able to read.

Some ecosystems make this nearly free. Go's toolchain cross-compiles by setting GOOS and GOARCH, and because the compiler, the runtime and the linker are all written in Go with no dependency on a system C toolchain, a pure-Go program with CGO_ENABLED=0 cross-compiles to a dozen platforms from one laptop with no extra installation whatsoever. That is genuinely remarkable engineering and it is the single best reason to keep cgo out of a program you intend to ship widely.

Rust is close but has one more step that trips everyone. Running rustup target add aarch64-unknown-linux-gnu downloads a precompiled standard library for that target — and stops there. It does not give you a linker. Rust compiles your crate to object files and then hands them to a C toolchain's linker to produce the final binary, so you must also install a cross linker and tell Cargo about it in .cargo/config.toml. "I added the target and it still fails at the link step" is the most-asked cross-compilation question in the Rust world, and the answer is always the same: the target is the library, not the toolchain.

For C and C++ the interesting modern option is Zig. zig cc is a Clang frontend that ships the headers and enough of libc for a long list of targets in the distribution itself, so zig cc -target aarch64-linux-gnu.2.28 gives you a working cross C compiler with a chosen glibc symbol floor from a single tarball. That last part deserves emphasis: it lets you pick your glibc floor as a compile-time argument rather than by hunting for an old enough build container. Check Zig's current target support table before you bet a release pipeline on a specific triple, because the list moves.

What emulation actually is

Emulation takes the opposite approach: leave the build alone, and make the machine lie. On Linux this is binfmt_misc, a kernel facility that lets you register an interpreter for a binary format identified by its magic bytes. Install qemu-user-static, register it for aarch64 ELF headers, and now when the kernel is asked to execute an arm64 binary on an x86 host it silently hands it to QEMU instead of refusing. QEMU translates the guest's instructions to host instructions as it goes, and maps the guest's system calls onto the host kernel's system calls. From inside, everything looks arm64. Even uname reports the target machine, because QEMU fakes it.

This is what makes docker buildx build --platform linux/arm64 work on an x86 laptop, and it is why that command is so seductive: one Dockerfile, one machine, N architectures, no toolchain work at all. It is also why the resulting builds are slow enough that people schedule them nightly and stop noticing when they break. Note that Docker's own tooling has moved toward cross-compilation where it can, precisely because emulating a whole compiler is such an expensive way to produce a file.

Where cross builds actually break

If cross-compiling were simply the correct answer, this post would be four paragraphs long. It breaks, reliably, in five places, and every one of them is about the environment surrounding the compiler rather than the compiler itself.

The sysroot

A cross-compiler on its own has no idea what the target system looks like. It needs the target's headers and the target's libraries — a directory tree that mirrors the target's /usr, called a sysroot, passed with --sysroot or assembled by Debian's multiarch layout under /usr/aarch64-linux-gnu. Without one you can compile a program that uses nothing but the language, and nothing else. The moment your build links against OpenSSL or zlib or libpq, you need those libraries built for the target, present in the sysroot, and findable.

Findable is where it goes wrong, and pkg-config is the usual culprit. Run it in a cross build without setting PKG_CONFIG_SYSROOT_DIR and PKG_CONFIG_LIBDIR and it will cheerfully answer with the host's include and library paths, your build will pass those to the cross-compiler, and the linker will report that the file format is not recognised. That error message is the sound of a host library being fed to a foreign linker, and it costs everybody an afternoon exactly once.

The linker and the build-system plumbing

Every build system has its own dialect for "you are not building for this machine", and each one has to be taught separately. CMake wants a toolchain file setting CMAKE_SYSTEM_NAME, CMAKE_SYSTEM_PROCESSOR, the compiler paths and the CMAKE_FIND_ROOT_PATH_MODE variables that stop find_library from wandering into the host's /usr/lib. Meson wants a --cross-file with explicit binaries and host_machine sections. Autotools wants --host set to the target triple. Cargo wants the linker key in .cargo/config.toml, and the cc crate wants CC_<target-with-underscores> so that any C shim in your dependency tree compiles with the right compiler rather than the default one.

None of this is difficult. All of it is fiddly, environment-specific, easy to get subtly wrong, and completely invisible in your source tree — which is precisely why it belongs in a baked image with a version number rather than in a CI configuration file that four people edit.

The glibc version floor

This one deserves its own section because it is the failure that reaches actual users. glibc versions its symbols. When you link against a system's glibc, the linker records the versioned symbols you used, and the loader on the running system refuses to start a binary that asks for a version it does not have. Build on Ubuntu 24.04, ship to a RHEL box from a few years earlier, and the user gets a version-not-found error for a program that works perfectly on your machine and on every machine in your CI fleet.

There is no compiler flag that lowers the floor. You either build against an older sysroot, use a toolchain like Zig's that lets you name the glibc version as part of the target, statically link a libc that does not do symbol versioning at all (which is a large part of musl's appeal for shipped binaries), or — in the Python world — adopt the manylinux specification, whose whole purpose is to define a per-year glibc baseline and whose auditwheel tool computes the tag from the maximum symbol version your extension actually requires. The Python ecosystem got here first and built real infrastructure for it; everyone else is still checking objdump output by hand.

Code that has to run at build time

This is the structural limit, and it is the one that decides which cells can be cross-compiled at all. Cross-compiling is only sound when nothing produced for the target needs to execute during the build. That condition is violated constantly and often invisibly.

• Autotools configure scripts detect features by compiling and RUNNING tiny test programs. In a cross build they cannot run anything, so those checks are skipped or guessed, and you are expected to supply cache variables telling configure the answers by hand. A configure script that silently guesses wrong produces a build that compiles and misbehaves.
• Rust build.rs scripts and procedural macros are compiled for the HOST and run on the host during the build — while the crate itself is compiled for the target. That split is correct and deliberate, and it means a build script that probes the machine it is running on learns about the wrong machine.
• Python native extensions want the target interpreter's headers and ABI. setup.py is Python that executes at build time, against the host interpreter, and the cross story is famously rough. cibuildwheel's documented path to Linux aarch64 wheels on an x86 runner is emulation, precisely because cross-compiling this reliably is so hard.
• Ahead-of-time compilers for managed languages frequently cannot cross-compile at all. GraalVM's native-image has historically required a build machine of the same architecture as the target; check the current documentation before assuming otherwise. Bytecode was architecture-neutral right up until you decided to AOT it, and then you rejoined the matrix.
• Code generation steps that run a tool you just built. Any build that compiles a generator and then executes it needs two toolchains — one for the host tool, one for the target output — and most build systems make that awkward.

Where emulation actually breaks

So you reach for QEMU, because it runs everything and asks no questions. It genuinely does work, and for a compile-and-package job with no interesting runtime behaviour it is a perfectly reasonable tool. But understand what you are buying.

The speed cost is the obvious one and the least interesting: dynamic binary translation is a large multiple slower than native, badly enough that teams routinely move emulated legs to a nightly schedule and then stop reading the results. Measure your own workload rather than trusting a ratio from a blog post, including this one. The important costs are the correctness ones.

• The kernel is not emulated. qemu-user translates instructions and forwards system calls to the HOST kernel. So your "arm64" build is running against an x86 host kernel with x86 kernel behaviour, x86 page-size assumptions and x86 answers to anything kernel-shaped. Anything testing io_uring, seccomp filters, eBPF or unusual syscalls is testing the wrong kernel.
• Memory ordering is emulated in the SAFE direction, which is the dangerous direction. aarch64 has a weaker memory model than x86-64. Emulating arm on x86 usually gives your code STRONGER ordering guarantees than real hardware would. A data race that segfaults a real Graviton box can run green under emulation forever. The emulator is not lying maliciously; it is just not obligated to be as hostile as the real machine.
• JIT-compiled runtimes are pathological. A JIT emits target machine code at runtime, which the emulator must then translate, continuously, as it changes. JVM, V8, LuaJIT and .NET workloads under user-mode emulation range from extremely slow to genuinely broken.
• Runtime CPU feature detection gets confusing answers. Libraries that pick a SIMD path by probing CPU features at startup may select a different path under emulation than on the real chip, so the code you shipped is not the code your tests exercised.
• Syscall coverage is good, not complete. Unimplemented calls surface as ENOSYS in odd places, and the classic "qemu: uncaught target signal 11" is what a deep incompatibility looks like from the outside: a crash with no useful stack.
• Timing is meaningless. Any benchmark, timeout-sensitive test, or race reproduction you run under emulation is measuring the translator.

The pattern: a template per target triple

Here is the arrangement that has held up for me. Stop treating the matrix as a list of outputs and start treating it as a list of (target triple, build mode, host architecture, template) tuples. Each triple is bound to a specific baked machine image, and each image is bound to a specific host architecture. The mode column — native, cross, or off-box — is written down explicitly, so nobody can accidentally believe a cross cell tested anything.

The unit of execution is a fresh Firecracker microVM per build job, restored from the template for that cell. A microVM runs guest code natively on the host CPU under KVM, so an aarch64 guest requires an aarch64 host — there is no configuration flag that changes this, and any platform implying otherwise is emulating underneath. That constraint is a feature here: it is what makes a native cell honestly native. Your fleet needs hosts of each architecture you claim to support natively, and the scheduler places each job on a matching one.

What makes this affordable rather than theoretical is that creating the guest is a snapshot restore, not a boot. On PandaStack a create runs around 179ms at p50 and 203ms at p99, because the template's frozen state is restored and resumed rather than started from a bootloader. The genuine cold boot — about three seconds — happens once, when the template is baked, and is amortised over every job that ever restores it. Memory is copy-on-write and the rootfs is a reflink clone, so the fiftieth parallel leg of your matrix does not copy a multi-gigabyte toolchain to exist. Each guest gets its own network namespace from a pool of 16,384 pre-allocated /30 subnets, which means an egress policy per build rather than a shared bridge, and a TTL means a job your orchestrator forgot about deletes itself.

#!/usr/bin/env bash
# build-matrix.sh -- the target matrix, as data.
#
# The load-bearing column is MODE. It says HOW a cell is produced, not just
# what it produces, because "aarch64-unknown-linux-gnu" is three completely
# different engineering problems depending on whether you cross-compile it,
# emulate it, or build it on an actual arm64 machine.
#
#   native  -> built on a host of that architecture. No translation anywhere.
#   cross   -> host toolchain emits foreign code. Fast, and only safe for
#              targets where nothing has to RUN during the build.
#   offbox  -> we cannot legally or practically produce this here. Say so.
set -euo pipefail

REPO_SHA="${1:?usage: build-matrix.sh <commit-sha>}"

#        triple                       mode    hostarch  template
TARGETS=(
  "x86_64-unknown-linux-gnu           native  amd64     build-gnu-x86_64"
  "aarch64-unknown-linux-gnu          native  arm64     build-gnu-aarch64"
  "x86_64-unknown-linux-musl          cross   amd64     build-musl"
  "aarch64-unknown-linux-musl         cross   amd64     build-musl"
  "x86_64-pc-windows-gnu              cross   amd64     build-mingw"
  "aarch64-apple-darwin               offbox  -         -"
)

dispatch() {
  local triple="$1" mode="$2" hostarch="$3" template="$4"

  case "$mode" in
    native)
      # A guest of this architecture, on a host of this architecture. The
      # compiler, the test binary and the linker all run natively; nothing is
      # being translated and nothing is being guessed.
      pandastack sandbox run \
        --template "$template" \
        --arch "$hostarch" \
        --ttl 1800 \
        -- "/opt/build/native.sh $REPO_SHA $triple"
      ;;

    cross)
      # One host toolchain, foreign output. Legitimate when the build is a
      # pure compile: no configure test programs, no build.rs that has to
      # execute target code, no native extension probing a target interpreter.
      pandastack sandbox run \
        --template "$template" \
        --arch "$hostarch" \
        --ttl 1800 \
        -- "/opt/build/cross.sh $REPO_SHA $triple"
      ;;

    offbox)
      # Apple targets need Apple hardware and an Xcode licence. A Linux
      # microVM does not fix that, and pretending otherwise is how you end up
      # with an unsigned .app nobody can open. Route it, do not fake it.
      echo "SKIP $triple -- built on the macOS fleet, not here" >&2
      return 0
      ;;
  esac
}

pids=()
for row in "${TARGETS[@]}"; do
  # shellcheck disable=SC2086
  set -- $row
  dispatch "$1" "$2" "$3" "$4" &
  pids+=("$!")
done

fail=0
for p in "${pids[@]}"; do wait "$p" || fail=1; done
exit "$fail"

Two details in that script are doing real work. The first is that mode is data, not a comment — the dispatch function branches on it, so a cell cannot quietly change category without someone editing the table. The second is that the Apple row refuses rather than degrades. A pipeline that silently substitutes a worse method for a cell it cannot handle is worse than one that fails, because the failure is at least visible.

Toolchain pinning is a template artifact, not a CI step

The most valuable property of this arrangement has nothing to do with speed. It is that the toolchain stops being something a build job resolves and becomes something a build job inherits. Every rustup install, every apt-get of a cross linker, every .cargo/config.toml with a linker path in it, every sysroot — all of it happens at bake time, and the result is frozen into a snapshot with a version. The compiler version is not a line in a YAML file that might resolve differently on a Tuesday when an upstream package moves. It is a property of the machine, captured.

"""Bake one build template per target triple.

The point of this script is that it runs RARELY -- when a toolchain version
moves, not when a commit lands. Everything slow and everything version-
resolving happens here, once, and is then frozen into a snapshot. A build job
never resolves a toolchain; it inherits one.
"""
import json

from pandastack import Sandbox

TRIPLE = "aarch64-unknown-linux-gnu"

# Pin everything that can drift. These strings ARE the template's identity --
# if one of them changes, you get a new template, not a mutated one.
TOOLCHAIN = {
    "triple": TRIPLE,
    "rust": "1.83.0",
    "cc": "gcc-12-aarch64-linux-gnu",
    "glibc_floor": "2.28",   # oldest runtime we promise to support
    "sysroot": "/usr/aarch64-linux-gnu",
}

sbx = Sandbox.create(template="base", ttl_seconds=1800)

# 1. The toolchain, pinned by exact version. No "latest", no unversioned
#    curl | sh, nothing that resolves differently next Tuesday.
sbx.exec(f"rustup toolchain install {TOOLCHAIN['rust']} --profile minimal")
sbx.exec(f"rustup target add {TRIPLE} --toolchain {TOOLCHAIN['rust']}")

# 2. `rustup target add` ships the precompiled std for the target. It does
#    NOT ship a linker. This is the single most common cross-build failure and
#    it is fixed here, at bake time, not in a CI YAML file six months later.
sbx.filesystem.write(
    "/root/.cargo/config.toml",
    f'[target.{TRIPLE}]\n'
    f'linker = "aarch64-linux-gnu-gcc"\n'
    f'\n[env]\n'
    f'CC_{TRIPLE.replace("-", "_")} = "aarch64-linux-gnu-gcc"\n'
    f'PKG_CONFIG_SYSROOT_DIR = "{TOOLCHAIN["sysroot"]}"\n'
    f'PKG_CONFIG_LIBDIR = "{TOOLCHAIN["sysroot"]}/lib/pkgconfig"\n',
)

# 3. Warm the dependency cache INTO the image. A build job then reads a
#    read-mostly copy-on-write view of it and can never write back into the
#    layer the next job restores from -- cache speed, no cross-job write path.
sbx.exec("git clone --depth 1 https://github.com/acme/widget.git /src")
sbx.exec(f"cd /src && cargo fetch --locked --target {TRIPLE}")

# 4. Leave a machine-readable record of what is frozen in here. Every build
#    emits this into its provenance, so an artifact can name its toolchain.
sbx.filesystem.write("/etc/build-toolchain.json", json.dumps(TOOLCHAIN, indent=2))

# 5. Freeze it. The ~3s cold boot is paid once, here. Every build that
#    restores this snapshot gets it back in ~179ms p50.
snap = sbx.snapshot()
print(f"baked {TRIPLE} -> {snap.id}")
sbx.kill()

Note the /etc/build-toolchain.json file. It costs nothing and it turns "which compiler built this artifact?" from an archaeology exercise into a read. Every build emits it alongside the binary, so a release six months from now can be traced back to the exact frozen machine that produced it. If you are doing provenance attestation properly, this is the natural place for the toolchain half of the statement to come from.

Why a baked template is a reproducibility asset

Think about where non-determinism enters a normal CI build. Not from your source — you pinned that with a lockfile. It enters from everything the build resolves while running: the package index it hit, the version a floating constraint picked today, the base image tag that moved under you, the transitive apt dependency that got a security update between Monday and Thursday. Every one of those is an undeclared input, and they are all resolved on the hot path of a build that is supposed to be a pure function.

A baked snapshot moves every one of those resolutions to a moment you control, and then freezes the result. Two builds of the same commit, run two months apart on different physical hosts, restore byte-identical baked state — not "cleaned back to clean", but never dirty in the first place. The template identity becomes your toolchain version, which means the way you upgrade a compiler is to bake a new template and change one string in the matrix, and the way you roll back is to change it back. That is a much better upgrade story than editing an install command and hoping.

A build is only as reproducible as its least-pinned input, and on most CI systems the least-pinned input is the machine. Baking the machine is the cheapest pin you will ever add.

Cache strategy: bake it, do not mount it

The obvious objection to a fresh machine per job is that you throw away the crate registry, the module cache and the object cache that make builds tolerable, and re-downloading a dependency tree per cell is not a trade anyone accepts. The answer is to move the cache from the machine into the image: run the fetch during the template bake, then snapshot. Because restore is copy-on-write, every job inherits a read-mostly view of the warmed cache and only dirties the pages it touches, which vanish with the guest.

The alternative — a shared writable cache volume mounted into every build job — is the pattern I would push back on hardest, especially in a matrix, and especially if any of your cells build code you did not write. A writable shared cache is a channel from one build into every subsequent build. A build that can write to it can change what a later, unrelated build resolves and executes, and "my dependency cache got poisoned" is not a fun sentence to say to a security team. It is also a correctness hazard even with no adversary: a matrix where legs share mutable state is a matrix whose cells are not independent, which defeats the entire purpose of running them separately.

If one cell genuinely needs cross-job persistence — a large incremental compile cache, say — attach it deliberately, scope it to that one cell, mount it read-only for anything untrusted, and give it exactly one trusted writer. The thing you are avoiding was never persistence. It was persistence that everybody shares because it happened to live on the same box.

Fanning out: one job per target

With the templates baked, the orchestration collapses into something dull, which is the correct shape for a component whose job is to not surprise you. Create a guest from the cell's template, get the exact commit in, run the build, verify the artifact, read it out, destroy the machine.

import concurrent.futures as cf
from dataclasses import dataclass

from pandastack import Sandbox

REPO = "https://github.com/acme/widget.git"
SHA = "a1b2c3d4e5f6"  # immutable commit, never a branch name


@dataclass(frozen=True)
class Target:
    triple: str
    template: str   # the baked, version-pinned toolchain for this cell
    mode: str       # "native" or "cross" -- an honest label, not a hope


MATRIX = [
    Target("x86_64-unknown-linux-gnu",  "build-gnu-x86_64",  "native"),
    Target("aarch64-unknown-linux-gnu", "build-gnu-aarch64", "native"),
    Target("x86_64-unknown-linux-musl", "build-musl",        "cross"),
    Target("aarch64-unknown-linux-musl","build-musl",        "cross"),
    Target("x86_64-pc-windows-gnu",     "build-mingw",       "cross"),
]


def build(t: Target) -> tuple[str, bytes | None, str]:
    """One target, one microVM. Nothing is shared between legs except the
    commit SHA, which is the only input that is supposed to be shared."""
    with Sandbox.create(
        template=t.template,
        ttl_seconds=1800,               # backstop: a wedged build reaps itself
        metadata={"triple": t.triple, "sha": SHA, "mode": t.mode},
    ) as sbx:
        # The toolchain is already inside. We are not installing rustc here;
        # we are restoring a machine that has had rustc since bake time.
        src = sbx.exec(
            f"git -C /src fetch --depth 1 origin {SHA} && "
            f"git -C /src checkout --detach {SHA}"
        )
        if src.exit_code != 0:
            return (t.triple, None, src.stderr)

        out = sbx.exec(
            f"cd /src && cargo build --release --locked --target {t.triple}",
            timeout_seconds=1500,
        )
        if out.exit_code != 0:
            return (t.triple, None, out.stderr[-4000:])

        # "cargo exited 0" is the weakest claim in this whole pipeline.
        # Verify the ELF/PE we actually produced before believing it.
        check = sbx.exec(f"/opt/build/verify.sh /src/target/{t.triple}/release/widget")
        if check.exit_code != 0:
            return (t.triple, None, check.stdout + check.stderr)

        blob = sbx.filesystem.read(f"/src/target/{t.triple}/release/widget")
        prov = sbx.filesystem.read("/etc/build-toolchain.json")
        return (t.triple, blob, prov.decode())
    # VM destroyed here: build tree, caches, any leftover daemon, all of it.


# Fan out. Each leg is a separate machine with its own kernel and its own
# /30 network namespace, so nothing one leg does can reach another. There is
# no warm pool to size, because creates are snapshot restores.
with cf.ThreadPoolExecutor(max_workers=len(MATRIX)) as pool:
    for triple, artifact, note in pool.map(build, MATRIX):
        status = f"{len(artifact)} bytes" if artifact else "FAILED"
        print(f"{triple:32} {status}")

There is no warm pool in that code and there does not need to be one, which is the quiet consequence of a sub-second create. Every team that adopts per-job VMs on conventional instances eventually finds provisioning slow enough to hurt and starts reusing machines to hide the latency — at which point they have paid for isolation and given it back. And note the two independent stop conditions: the exec timeout catches a build that hangs, and the create-time TTL catches an orchestrator that crashes between creating a guest and reaping it. Neither depends on the other being correct.

Verify the artifact, not the exit code

Every cross-build disaster I have watched had a green build step. The compiler exited zero. The tarball was produced. The pipeline went green and stayed green for weeks. The lie was in the ELF header, and nobody read it, because reading it is not part of anybody's default pipeline.

Add a verification step to every cell that asserts the artifact is what the cell claims. It is thirty lines of shell, it runs in milliseconds, and it catches the three failures that actually reach users: a cross-build that silently fell back to the host compiler and produced a perfectly valid binary for the wrong architecture, a glibc floor that crept upward when a base image moved, and a libc mismatch that will manifest as an incomprehensible "not found" on a user's Alpine container.

#!/usr/bin/env bash
# /opt/build/verify.sh -- assert the artifact is what the cell claims it is.
#
# Every cross-build disaster I have watched had a green build step. The lie is
# never in the exit code, it is in the ELF header nobody read.
set -euo pipefail
BIN="${1:?usage: verify.sh <path>}"

# 1. Is it even the right machine? A cross-build that silently fell back to
#    the host compiler produces a perfectly valid binary for the WRONG arch.
file -b "$BIN"
#    e.g. "ELF 64-bit LSB pie executable, ARM aarch64, ... dynamically linked"

# 2. Which dynamic loader does it demand? This is the difference between a
#    glibc binary and a musl binary, and it is why an Alpine user gets the
#    legendary "not found" when they run a perfectly present executable.
readelf -l "$BIN" | grep -i 'interpreter' || echo "  (static -- no interpreter)"
#    glibc: /lib/ld-linux-aarch64.so.1     musl: /lib/ld-musl-aarch64.so.1

# 3. The glibc symbol floor. Linking against a newer glibc silently raises the
#    oldest system your binary can run on. Nothing warns you. This does.
FLOOR="$(objdump -T "$BIN" 2>/dev/null \
  | grep -o 'GLIBC_[0-9]\+\.[0-9]\+' | sort -V -u | tail -1 || true)"
echo "requires at most: ${FLOOR:-none (static or non-glibc)}"

MAX_ALLOWED="GLIBC_2.28"   # our published support floor
if [ -n "$FLOOR" ] && [ "$(printf '%s\n%s\n' "$FLOOR" "$MAX_ALLOWED" \
      | sort -V | tail -1)" != "$MAX_ALLOWED" ]; then
  echo "FAIL: needs $FLOOR, we promise $MAX_ALLOWED" >&2
  exit 1
fi

# 4. On a NATIVE cell only: actually run the thing. A cross cell cannot do
#    this and should not pretend to -- that is precisely the guarantee you
#    gave up when you chose to cross-compile.
if [ "${BUILD_MODE:-cross}" = "native" ]; then
  "$BIN" --version
fi

The last block is the interesting one. On a native cell you can execute the binary you just built, so you do — it is the cheapest smoke test in existence and it proves the loader, the interpreter path and the symbol floor all agree. On a cross cell you cannot, and the script does not pretend to. That asymmetry is not a gap in the tooling; it is the exact guarantee you traded away when you chose to cross-compile, and writing it down in the script keeps everyone honest about which cells were tested and which were merely produced.

The limits I cannot engineer around

A Linux microVM farm is a good answer to a specific part of this problem and not to all of it. Here is the part where I would rather be blunt than sell you something.

• Apple targets need Apple hardware. Building, signing and notarizing macOS or iOS artifacts requires Xcode and the Apple toolchain, whose licence terms restrict use to Apple-branded hardware. Unofficial cross-toolchains exist and require you to supply an SDK, which brings its own licensing question. No amount of Firecracker changes this. Route macOS cells to a macOS fleet — a cloud Mac provider or hosted macOS runners — and check the current terms yourself rather than taking a blog post's word for it.
• Windows splits into two very different problems. The MinGW-w64 target cross-compiles from Linux perfectly well for a great deal of C, C++, Go and Rust, and that cell fits the farm neatly. The MSVC target needs Microsoft's toolchain and its own licensing, and tools that download the MSVC CRT and Windows SDK for cross use are a licence question you should route through someone who enjoys that kind of question. Authenticode signing needs your certificate and its own key handling regardless.
• Architecture is a hard partition for microVMs. Guest code runs natively on the host CPU, so an arm64 template requires an arm64 host and a snapshot taken on one architecture cannot be restored on the other — not partially, not in a degraded mode. Put the architecture in the artifact's storage path rather than in a metadata field somebody might forget to check, and make the scheduler fail closed when no matching host exists.
• Some targets have no native host you can rent. If you ship for riscv64 or an embedded triple, there is no fleet to build a native cell on, and cross-compiling plus emulated smoke tests plus real hardware in a lab is the actual answer. Be explicit in your support matrix about which platforms are tested on hardware and which are merely built.
• A microVM does not make an emulated cell honest. If you run QEMU inside the guest — and sometimes that is the right call for a target with no native host — you have contained the emulator, which is worth something, but the emulated result carries exactly the same caveats it did before.

Four strategies, side by side

With the usual caveat: everything about third-party tooling here is qualitative and moves between releases, so verify specifics against their current documentation. Only the PandaStack numbers are measured.

• QEMU user-mode emulation (buildx --platform on a single-arch host) — Setup cost: near zero; register binfmt handlers and go. Fidelity: runs anything, gets the kernel, memory ordering, JIT behaviour and timing wrong, and errs toward passing. Speed: a large multiple slower than native; measure yours. Fleet: one architecture. Best for: producing an artifact for a foreign arch when the build has no interesting runtime behaviour, and for smoke-testing that a thing compiles and starts.
• Native runners, one fleet per architecture — Setup cost: high and ongoing; you now operate N fleets. Fidelity: perfect, by definition. Speed: native. Fleet: multiplied, plus the toolchain drift that comes from the arm image nobody has upgraded since the migration. Best for: teams already running heterogeneous infrastructure who can afford to keep every fleet's provisioning identical.
• Cross-compile everything on one host — Setup cost: concentrated in the toolchain (sysroots, linkers, pkg-config, per-build-system cross files) and then near zero per build. Fidelity: exact for the compile, and structurally incapable of running anything for the target, so configure-time execution, build.rs probes, native extensions and AOT compilers are where it stops. Speed: the fastest option by a wide margin. Fleet: one. Best for: pure-Go and static-Rust binaries, MinGW targets, and anything where the build is genuinely just a compile.
• MicroVM template per target triple — Setup cost: bake one template per cell, then it is data in a table. Fidelity: native cells are honestly native (own guest kernel, own memory, real hardware ordering); cross cells are labelled as such rather than dressed up. Speed: create is a snapshot restore at about 179ms p50 and 203ms p99, with the roughly 3s cold boot paid once at bake time; cache is baked in and copy-on-write. Fleet: hosts per architecture you claim natively, but no warm pool and idle costs nothing. Best for: matrices with more than a couple of cells, builds that execute untrusted code, and anyone who needs the toolchain frozen rather than resolved.

What I would actually build on Monday

Do not migrate the matrix. Take the two cells that hurt — usually the emulated arm64 leg that takes forty minutes and the one whose failures nobody trusts — and give them baked native templates. Leave everything else exactly where it is. The end state is deliberately modest: same repository, same lockfiles, same release process, same people. The only thing that changed is that each cell of the matrix now names its method out loud and runs on a machine that was created a fraction of a second ago from an image you can read.

Then write the support matrix in your README as the thing it really is: a list of platforms with a column saying whether each one is tested on hardware, tested under emulation, or merely built. That column will be uncomfortable the first time you fill it in. It is also the most useful sentence about compatibility you will ever publish, and it is the sentence a per-triple build farm exists to let you write truthfully.