Bazel Remote Execution Workers on Firecracker microVMs

Remote build execution arrives in a monorepo as a performance project. The build takes 40 minutes on a laptop and four on a fleet, someone stands up a scheduler and a few hundred workers, and the graph starts fanning out across machines that are not yours. It is one of the genuinely great wins in build engineering — and it quietly relocates a security boundary that nobody wrote a ticket for. What you just built is a service that accepts arbitrary commands from anyone who can run a build, executes them, and writes the results into a cache every engineer trusts implicitly.

That is fine while every action comes from a trusted first-party branch. It stops being fine the moment the inputs stop being trusted: an open-source monorepo taking forked pull requests, contractors with commit rights to a directory, an AI agent that has been given the ability to edit BUILD files and rerun the build until it goes green. Those are all now normal, and none of them were normal when the standard worker designs were written.

What the Remote Execution API actually asks a worker to do

Strip away the protobufs and the worker's job is three steps. Fetch an input tree from the content-addressable store by digest, so the worker materialises exactly the files the action declared and nothing else. Run one command in a hermetic working directory, with the environment variables and platform properties the action specified. Upload the declared output files back to the CAS by digest, and report the exit code, stdout and stderr as an ActionResult keyed by the action's own digest.

Bazel, Buck2 and Pants all speak roughly this protocol, which is why an executor like Buildbarn, Buildfarm, BuildBuddy, EngFlow or NativeLink can serve more than one build tool. The details differ and move between releases — check the current Remote Execution API and your executor's docs rather than this post for field names and flags. But the correctness contract underneath is one sentence, and it is worth writing on a wall: the worker must add nothing the action did not declare.

Everything a build system promises rests on that sentence. Bazel computes an action digest from the command line, the environment, the platform properties and the digests of every input, and treats it as a complete description of the computation: same digest, interchangeable outputs. That is the whole trick, and it is also an assumption. The worker is the only component in a position to violate it.

Hermeticity is a security property, not just a caching one

Most teams meet hermeticity as a flakiness story. An action reads something that wasn't in its inputs — a header in /usr/include, a Python package installed on the worker image, a file another action left in /tmp — and the build passes on the machine where that thing exists and fails everywhere else. Annoying, well understood, usually fixed by tightening the sandbox until the action's declared inputs are all it can see.

The security version of the same bug is considerably less funny. If action A can influence what action B sees, then B's action digest — which was computed only from B's declared inputs — no longer describes B's computation. B runs, produces a compromised output, and the executor writes that output into the CAS under the digest that every honest B in the fleet will look up. Every developer, every CI job and every release build now gets the poisoned artifact as a cache hit. Nobody rebuilds it, because rebuilding is precisely what a cache exists to avoid.

A remote cache poisoned by a leaky worker is a supply-chain compromise with excellent uptime. The digests all check out. That is the problem: they are checking out against a computation that didn't happen the way the digest says it did.

And there is no shortage of arbitrary code with which to do it. A genrule is a shell script by definition. A repository rule runs at loading time, before most of your policy has an opinion, and typically fetches something off the network. Custom Starlark rules invoke whatever tool the author points them at. Test targets run the code under test. "Untrusted input" in a build system is not an exotic scenario you have to construct — it is the primary feature. Bazel's own security model has long been explicit that BUILD files and the build tool are not a security boundary, which puts the boundary squarely where the action executes.

Where container-based workers leak

The common designs run on a spectrum. At the strict end, a fresh container per action from a pinned image, torn down afterwards. At the loose end, a long-lived worker process that runs actions one after another in a working directory it scrubs between runs. The second is much faster and much more popular than anyone admits in a design doc.

The word "hermetic" survives contact with /usr/bin about as well as any other word does. Here is where it stops being true:

• Shared kernel. Every container on the node is one kernel bug from every other one, and your actions are compilers and linkers: large, old, memory-unsafe C++ programs parsing attacker-supplied files. That is the classic path from a parsing bug to a kernel bug to somebody else's action.
• Shared page cache and filesystem layers. Two actions on the same node see the same image layers and host caches — mostly benign, and occasionally the exact channel by which one action leaves something for another.
• /tmp and $HOME residue. Toolchains write outside the action's working directory constantly: ccache, the Go build cache, ~/.cargo, ~/.m2, ~/.npm, __pycache__, JVM temp files. A persistent worker's cleanup script covers the paths its author thought of, on the day they wrote it.
• Network access during actions. If the action can reach the internet, the input tree is not the input set. A rule that curls a URL is undeclared input at best and exfiltration at worst — and the node's metadata endpoint is one HTTP request from the container's network namespace.
• Persistent worker state. Bazel's persistent workers keep a JVM (or an equivalent) alive across actions for exactly the right reason: JVM startup is brutal and warm workers are much faster. They also keep in-memory caches, loaded classes, static fields and open file descriptors across actions from different sources. A persistent worker is a build machine with a memory and a grudge.
• Helpful toolchains. Compilers search default include paths. Linkers search default library paths. Interpreters read site-packages. Half of a toolchain's ergonomics consist of finding things you did not tell it about, and every one of those searches is a hole in the sandbox that the toolchain considers a feature.

Bazel's linux-sandbox mitigates a real chunk of this with mount and PID namespaces, and it is worth using. But it is a sandbox built to catch honest mistakes — undeclared inputs — not to withstand an action that is actively trying to get out. Those are different engineering problems with different failure modes, and it is worth being clear about which one you have bought.

The microVM shape: one action, one kernel, one lifetime

The alternative is structurally simple. Restore a snapshot of a machine with the toolchain already baked in, materialise the action's input tree into the guest, run the one command, read the declared outputs back out, destroy the machine. Then do it again for the next action, from the same snapshot, on a machine that has never seen a previous action.

What that buys is a shift in where hermeticity lives. In a container worker, hermeticity is a property of your cleanup logic — a promise, maintained by a script, that can silently fail. In a per-action microVM it is a property of the machine: there is a fresh guest kernel under KVM, no shared page cache, no residue in /tmp because nothing has ever written to /tmp, and no persistent worker process to carry a grudge. Destroying a VM is a fact, not a promise.

# The leaf of a Remote Execution worker: one action, one microVM, no residue.
# The scheduler, CAS client and ActionResult plumbing live above this; this is
# the part that decides what the action is allowed to see.
import shlex

from pandastack import Sandbox


def execute_action(action, cas):
    # Snapshot restore, not a boot: the toolchain is already resident in the
    # baked image. ~179ms p50 to get a machine, ~203ms p99.
    sbx = Sandbox.create(
        template="bazel-worker",
        ttl_seconds=action.timeout_seconds + 60,   # backstop; the guest reaps itself
        metadata={"action_digest": action.digest, "instance": action.instance_name},
    )
    try:
        # Materialise ONLY the declared input tree. Every file here came from
        # the CAS by digest; nothing else exists in the working directory,
        # because the working directory did not exist thirty seconds ago.
        for path, blob in cas.walk_input_root(action.input_root_digest):
            sbx.filesystem.write(f"/work/{path}", blob)

        for d in action.output_directories:
            sbx.exec(f"mkdir -p /work/{d}", timeout_seconds=10)

        # env -i, then exactly the environment_variables the action declared.
        # An inherited variable is an input you did not write down, which makes
        # it the input that breaks the build in six months for nobody's reason.
        env = " ".join(f"{k}={shlex.quote(v)}" for k, v in action.environment)
        run = sbx.exec(
            f"cd /work && env -i {env} PATH=/opt/toolchain/bin:/usr/bin:/bin "
            f"sh -c {shlex.quote(action.command)}",
            timeout_seconds=action.timeout_seconds,
        )

        # Upload only declared outputs, by path. "Tar up whatever ended up in
        # /work" is how an action smuggles a file into somebody else's cache.
        outputs = {}
        for p in action.output_paths:
            try:
                outputs[p] = cas.put(sbx.filesystem.read(f"/work/{p}"))
            except FileNotFoundError:
                pass   # a missing declared output is the action's bug, not ours

        return ActionResult(
            exit_code=run.exit_code,
            stdout=cas.put(run.stdout.encode()),
            stderr=cas.put(run.stderr.encode()),
            output_files=outputs,
        )
    finally:
        sbx.kill()   # the only cleanup step, and it cannot half-succeed

Note what is not in that function: no scrub step, no allowlist of directories to wipe, no check that the previous action's compiler daemon has exited. The isolation is not code you maintain. The same holds on the network side — the guest gets its own namespace, so "deny egress except the CAS" is a rule on a namespace that exists for one action rather than a policy you hope applies to the right container. On PandaStack those namespaces are pre-allocated, 16,384 /30 subnets per agent, so setup is not on the hot path either.

Routing actions: two exec platforms, one build

You do not have to choose one worker shape for the whole graph, and you should not. Bazel already has the mechanism: exec platforms carry exec_properties, the executor reads them, and individual targets can demand a platform. Route untrusted and network-touching actions to microVMs; leave your own first-party C++ compiles on the pooled workers.

# .bazelrc
#
# Property NAMES below are executor-specific -- BuildBuddy, EngFlow, Buildbarn
# and NativeLink each define their own vocabulary, and they change between
# releases. Verify every key against your executor's current docs.

build:rbe --remote_executor=grpcs://rbe.internal:8980
build:rbe --remote_cache=grpcs://cas.internal:8980
build:rbe --remote_instance_name=projects/acme/instances/default
build:rbe --remote_timeout=3600
build:rbe --jobs=200

# Two exec platforms in one build. Toolchain resolution picks the pooled one
# by default; targets that need the VM ask for it explicitly.
build:rbe --extra_execution_platforms=//platforms:linux_pooled,//platforms:linux_microvm
build:rbe --host_platform=//platforms:linux_pooled

# Don't drag every intermediate output back to the developer's machine; let
# the CAS keep them and fetch only what the build actually needs locally.
build:rbe --remote_download_toplevel
build:rbe --experimental_remote_downloader=grpcs://cas.internal:8980

# Forked-PR config: everything goes to the VM lane, and nothing this build
# produces is allowed to become a cache entry that a release build might hit.
build:untrusted --config=rbe
build:untrusted --extra_execution_platforms=//platforms:linux_microvm
build:untrusted --host_platform=//platforms:linux_microvm
build:untrusted --remote_upload_local_results=false
build:untrusted --noremote_accept_cached

The last two lines are the ones people skip and then regret. A build triggered by a fork should be able to read from the cache — that is most of the speedup — but the decision about whether it may write into the cache is separate, and for an untrusted trigger the answer is no. Read-only for untrusted, read-write for trusted, and a separate instance name if your executor lets you split the CAS namespace outright.

# platforms/BUILD -- the two lanes, and how a target opts into the strict one.

platform(
    name = "linux_pooled",
    constraint_values = [
        "@platforms//os:linux",
        "@platforms//cpu:x86_64",
    ],
    exec_properties = {
        "OSFamily": "Linux",
        "container-image": "docker://registry.internal/rbe-toolchain@sha256:9f3c...",
        "Pool": "trusted-warm",
    },
)

platform(
    name = "linux_microvm",
    constraint_values = [
        "@platforms//os:linux",
        "@platforms//cpu:x86_64",
    ],
    exec_properties = {
        "OSFamily": "Linux",
        "workload-isolation-type": "firecracker",
        "vm-template": "bazel-worker",     # snapshot with the toolchain baked in
        "dockerNetwork": "off",            # no egress; CAS reached out-of-band
        "Pool": "untrusted",
    },
)

# A genrule that runs a vendor's binary blob. It is a shell script executing
# code nobody on this team has read, which is a fine description of most
# genrules, and this one is honest about it.
genrule(
    name = "vendor_codegen",
    srcs = ["//third_party/vendor:sdk", "schema.json"],
    outs = ["generated.cc"],
    cmd = "$(location //third_party/vendor:codegen) --in $(location schema.json) --out $@",
    tools = ["//third_party/vendor:codegen"],
    exec_compatible_with = ["//platforms:requires_vm"],
)

Keeping the VM off the critical path of a 10,000-action build

Here is the reality check that kills naive versions of this design. A large monorepo build is not a hundred big actions; it is thousands of tiny ones. Compile one .cc file. Run one 200ms unit test. Copy a file. If the median action runs for 40 milliseconds, per-action setup is not overhead — it is the build.

Snapshot restore is what makes the numbers survivable rather than absurd. On PandaStack a create is a restore of a baked snapshot, not a boot: roughly 179ms p50 and 203ms p99, with the restore step itself near 49ms. The first cold boot, before a snapshot exists for that template, is about 3 seconds — paid once per template generation, not per action. Compare that with waiting for a fresh container to pull, start and warm its caches and it is competitive. Compare it with 40 milliseconds of actual work and it is still four times the action.

So be honest about the arithmetic and design around it:

1. Sort your actions by duration before you architect anything — Bazel's execution log and your executor's metrics both give you this. The distribution is usually bimodal: a long tail of sub-100ms actions that dominate the count, and a few link steps, integration tests and codegen runs that dominate the wall clock.
2. Give the long tail to pooled workers. A per-action VM for a 40ms compile is a rounding error you have chosen to multiply by ten thousand. These are also, overwhelmingly, your own first-party source files compiled by your own toolchain — the actions you actually trust.
3. Give the expensive and the untrusted actions VMs. A three-minute integration test, a vendor codegen blob, a forked-PR test target: 179ms of setup against 180 seconds of work is 0.1% overhead, and the isolation is worth considerably more than 0.1%.
4. Amortise where the work is genuinely serial. If several actions run in sequence against the same state — build, then test, then package over one tree — run them in a single VM lifetime. You lose isolation between steps that already share a trust domain, and save two restores.
5. Fork instead of restoring when you need warm state. A same-host copy-on-write fork is 400-750ms, cross-host 1.2-3.5s — slower than a plain restore, so use it only where the warm state earns the difference: a populated dependency cache, a started database for integration tests.
6. Put the CAS near the workers. Input-tree materialisation is real work: for a fat action the transfer can cost more than the command itself. Same-zone CAS, local blob caching on the host, output-path allowlists rather than whole-directory uploads.

The input tree is the part people underestimate. A microVM restores fast; writing a toolchain sysroot into it does not. That is the argument for baking: the toolchain, the SDK, the standard library — anything identical across actions — belongs in the snapshot, so it is resident at restore rather than streamed per action. What crosses the boundary per action should be source files and nothing else.

The four shapes, honestly compared

• Persistent pooled worker — Hermeticity: aspirational; it rests on a scrub step, and daemon state survives by design. Isolation: none between actions on the same worker; one compromised action owns every later one it serves. Latency: the best available — no setup, warm caches, warm JVM. Use it for: first-party actions from trusted branches where throughput is all that matters.
• Container per action — Hermeticity: good for honest mistakes; the working directory really is fresh. Isolation: namespaces on a shared kernel, shared page cache, and usually a shared host cache volume mounted in for speed. Latency: low, especially with a warm image on the node. Use it for: the default lane in most fleets, and a reasonable place to stay if every action is first-party.
• MicroVM per action — Hermeticity: a property of the machine, not of your cleanup code; there is no previous action to leak from. Isolation: a guest kernel under KVM, own network namespace, no host cache mounted in. Latency: ~179ms p50 to restore a baked snapshot, plus input-tree materialisation. Use it for: untrusted actions, network-touching rules, and anything long enough that 179ms disappears.
• MicroVM per logical unit (several actions, one lifetime) — Hermeticity: strong against other tenants, deliberately weak between the steps you grouped. Isolation: same hypervisor boundary, coarser granularity. Latency: one restore amortised across a sequence. Use it for: build-then-test-then-package chains over the same tree, where the steps already share a trust domain.

Where to draw the line

The honest recommendation is a split fleet, and it is not a compromise — it is the correct answer. Run trusted first-party actions on pooled workers: they are the overwhelming majority of your action count, they are fast, and the marginal security value of isolating your own compiler from your own compiler is close to zero. Route the rest to microVMs: forked pull requests, contractor-owned directories, vendored codegen, anything that touches the network, anything an AI agent authored.

Then enforce the split at the cache boundary as well as the execution boundary, because the execution boundary alone does not save you. An isolated action that still writes its result into the shared CAS has moved the compromise from "the worker" to "the cache," which is worse, because the cache is the thing everyone trusts. Untrusted lanes read from the cache and do not write to it, or write into a separate instance that release builds never consult.

None of this is exotic. Buck2 and Pants sit on the same protocol and take the same split cleanly, and every serious executor — Buildbarn, Buildfarm, BuildBuddy, EngFlow, NativeLink — has some notion of routing by platform property, though the vocabulary differs enough that you should read their current docs rather than assume. The part that is yours to decide is which actions you are willing to run on a machine that remembers the last one.

You can have a worker that is hermetic because a script cleaned it, or a worker that is hermetic because it was created eleven seconds ago and will not exist in another ninety. Only one of those survives an action that is trying.