Hermetic Builds and SLSA Provenance on MicroVMs

"We sign our releases" is one of those sentences that sounds like a control and is usually a decoration. The interesting question is never whether a signature exists — it's whether the thing being signed could have influenced the signing. If the key lives on the same machine that just ran a build script from a repo with four hundred transitive dependencies, then the signature attests to something narrower than everyone assumes: that at some point, a process on that runner had the key. Which is true, and which is also true for whatever the build decided to do with it.

I'm Ajay; I build PandaStack, which runs every sandbox as a Firecracker microVM. This post is about the build-step half of supply-chain security: what SLSA's build levels are actually asking you to prove, why the hard part is unforgeability rather than signing, and how a one-microVM-per-build architecture — with hermetic inputs inside the guest and the signing key strictly outside it — gets you a provenance statement that means something. Also, honestly, what it still doesn't mean.

What the SLSA build levels are actually asking for

SLSA (Supply-chain Levels for Software Artifacts) describes a ladder of build integrity, and the useful way to read it is not as a checklist but as a sequence of questions a skeptical consumer of your artifact would ask. Paraphrasing the intent rather than the spec text, and in rough order of increasing difficulty:

• Is there provenance at all? Can you produce a machine-readable document saying what was built, from what source, by what process — even if that document is generated by a script you wrote and trust on faith?
• Was it built by a build service rather than a laptop? Someone's workstation producing release binaries is the classic gap: nobody can distinguish "the official build" from "a build," because the process lives in one person's shell history.
• Is the provenance generated by the build platform rather than by the build itself, and is it protected against forgery by the very build steps it describes? This is the step where most homegrown pipelines quietly fail, and it's the one this post is mostly about.
• Are the builds isolated from each other, so one build cannot influence another's inputs, outputs, or attestations — including the cached state a later build will read?
• Beyond that, the higher rungs get into stronger claims about the build process itself: hermeticity, dependency completeness, and independent verification of the result.

Notice what's missing from that ladder: nothing on it says "use a fancier signing algorithm." The cryptography was never the weak part. The weak part is architectural — who holds the key, what else can reach it, and whether the code being attested to shares an address space, a filesystem, or a user account with the thing doing the attesting.

The hard part: a build must not be able to write its own alibi

Strip the framework away and the core requirement is one sentence: the build must not be able to influence its own provenance, or any other build's. Everything else is implementation detail. That sentence is deceptively hard because a build step is, by construction, arbitrary code execution that you invited. It compiles things, runs codegen, executes test suites, shells out. Any capability sitting in its reachable environment is a capability it has.

So the failure modes are less about exotic exploits and more about ambient access:

• The key is reachable — a signing key on disk, in an environment variable, in an agent socket, or behind a metadata endpoint the build process can also curl. If the build can reach the signer, the provenance is a statement the build can author about itself.
• The digest is self-reported — the pipeline trusts a hash the build printed to stdout rather than hashing the artifact bytes itself, on the other side of the boundary. A compromised build prints a very convincing hash of a very different file.
• The inputs aren't pinned — provenance that lists "npm registry" as a dependency source is describing a moving target. The materials section is only meaningful if the things it names are content-addressed.
• The builds share state — a cache, a toolchain directory, a shim on PATH, a workspace that isn't wiped. Build A doesn't have to attack the signer if it can quietly change what build B compiles, and build B will happily sign the result for it.
• The builder identity is inside the guest — if the credential that proves "this was built by our platform" is handed into the same environment as the untrusted build, then "our platform built it" degrades to "something with our platform's credential built it."

The standard failure: one runner, one user, everybody's postinstall scripts

Here's the setup I've seen more than any other, usually in an organization that is genuinely trying. A self-hosted CI runner — big machine, fast disk, warm caches, because the managed runners were too slow or too small. Jobs land on it as containers or bare processes under one runner user. The release job needs to sign artifacts, so the signing key is on the box: a file, a keychain entry, an env var, or a cloud KMS credential that any process running as that user can present. Caches are shared between jobs because that's the entire reason the runner is self-hosted.

Now run a build in that environment. A dependency install executes lifecycle scripts — npm postinstall, a setup.py, a Gradle plugin, a build.rs — as the runner user, with the runner user's filesystem access, the runner user's network egress, and the runner user's credentials. Not one of those scripts was reviewed by anyone on your team. Most of them are fine. The security property you're relying on is that all of them are fine, forever, across every transitive update. That's covered in more depth at /blog/sandbox-untrusted-npm-install, and the short version is that your supply-chain security is one `curl | bash` in a transitive postinstall away from being someone else's.

A signing key on a shared build runner isn't a signing key. It's a shared signing key, and the sharing terms are set by your dependency tree.

The cache-poisoning variant is worse because it's quieter. A hostile build doesn't need to touch the key at all. It writes a backdoored artifact into the shared layer cache, the module cache, or a ccache directory, and then waits for the release job — the trusted one, the one with the key, the one that will produce a perfectly valid signature — to consume it. The signature is real. The provenance is well-formed. The artifact is compromised. Nothing in the attestation is a lie; the attestation just never claimed the inputs were clean, because nobody made it prove that.

The microVM shape: one ephemeral guest per build, and no identity inside it

The architecture that makes the requirement structurally true rather than aspirationally true has four properties, and they're all about what is absent from the guest rather than what's present.

• One microVM per build, created fresh and destroyed on completion. Not a pooled worker that gets cleaned between jobs — cleanup scripts are a promise, and a promise is a thing that can fail to run. A destroyed VM is a fact.
• No persistent runner identity inside the guest. The guest has no long-lived credential, no CI agent token, no cloud instance role that outlives the build. If it gets compromised, the attacker's prize is a machine that is about to stop existing and has nothing to steal but the source it was already given.
• Hermetic, content-addressed inputs. The source arrives as bytes with a digest we verified before extraction; dependencies come from an internal mirror at pinned versions; the toolchain comes from a snapshot we can name by digest.
• The signer lives outside the guest, on the control plane, and never crosses the boundary. The guest emits an artifact; the control plane hashes those bytes itself and signs a statement about them. There is no code path by which the build touches the key, because the key is on the other side of a hypervisor.

A Firecracker microVM is the right boundary for this because it isn't a namespaced view of a shared kernel — it's a guest kernel under hardware virtualization with a small emulated device surface. Between two builds there is no shared kernel to exploit, no shared page cache, no shared PATH, no shared /tmp. Related reading on the isolation argument itself: /blog/microvm-ci-cd-pipeline-isolation.

# hermetic_build.py -- one build, one guest, one digest that WE computed.
# Nothing secret goes in. The signing key never comes near this function.
import hashlib
from pandastack import Sandbox

BUILD_SH = '''#!/usr/bin/env bash
set -euo pipefail
cd /work/src

# Pinned lockfile, no lifecycle scripts, packages from the internal mirror.
npm ci --ignore-scripts
npm run build

# Deterministic tar: sorted names, fixed owner and mtime, so the digest
# describes the build output rather than whatever order readdir felt like.
tar --sort=name --owner=0 --group=0 --numeric-owner \\
    --mtime='UTC 1980-01-01' -cf /work/out/artifact.tar dist
'''


def hermetic_build(source_tar: bytes, source_digest: str, commit: str) -> dict:
    sbx = Sandbox.create(
        template="base",
        ttl_seconds=1800,               # backstop: the guest reaps itself
        metadata={"role": "hermetic-build", "commit": commit},
    )
    try:
        sbx.filesystem.write("/work/src.tar", source_tar)
        sbx.filesystem.write("/work/build.sh", BUILD_SH)

        # Confirm the source is the source we meant BEFORE any of it runs.
        # A build cannot vouch for its own inputs; that is the caller's job.
        check = sbx.exec(
            f"sha256sum /work/src.tar | grep -q ^{source_digest}",
            timeout_seconds=60,
        )
        if check.exit_code != 0:
            raise SourceDigestMismatch(commit)

        sbx.exec(
            "mkdir -p /work/src /work/out && "
            "tar -xf /work/src.tar -C /work/src --strip-components=1",
            timeout_seconds=120,
        )

        # enter.sh applies default-deny egress and strips the ambient
        # environment before handing control to the untrusted build script.
        run = sbx.exec("bash /opt/hermetic/enter.sh /work/build.sh",
                       timeout_seconds=1500)
        if run.exit_code != 0:
            raise BuildFailed(commit, run.stderr)

        artifact = sbx.filesystem.read("/work/out/artifact.tar")

        # Hash on THIS side of the boundary. A digest the guest printed is a
        # log line, not evidence -- a compromised build prints what it likes.
        return {
            "commit": commit,
            "source_digest": source_digest,
            "artifact": artifact,
            "artifact_digest": hashlib.sha256(artifact).hexdigest(),
            "build_log": run.stdout,
        }
    finally:
        sbx.kill()   # no runner identity survives, because there is no runner

Hermetic inputs: default-deny egress and a mirror with no fallback

Isolation stops one build from harming another. Hermeticity is a different claim: that the build's inputs are fully enumerated and fixed, so the same declared inputs produce a build you can reason about. The enforcement mechanism is boring and effective — take away the network, then hand back exactly one route.

Because each guest has its own network namespace, the rules below are the guest's entire world for the length of the build, not a shared host firewall that six other teams also depend on. There's a longer treatment of egress control at /blog/controlling-network-egress-untrusted-code; here the goal is narrower — anything that isn't the internal mirror should fail immediately and loudly, so a build that silently depends on reaching the open internet breaks at build time instead of becoming an unrecorded input.

#!/usr/bin/env bash
# /opt/hermetic/enter.sh -- runs INSIDE the guest, before the build command.
# Baked into the build template; the untrusted build never edits it, because
# the untrusted build has not started yet.
set -euo pipefail

MIRROR_IP="10.10.0.20"        # internal read-only package mirror, pinned versions

# Default deny in both directions. Loopback stays up so local toolchains that
# talk to themselves (language servers, test harnesses) keep working.
iptables -P OUTPUT DROP
iptables -P INPUT  DROP
iptables -A OUTPUT -o lo -j ACCEPT
iptables -A INPUT  -i lo -j ACCEPT
iptables -A OUTPUT -d "$MIRROR_IP" -p tcp --dport 443 -j ACCEPT
iptables -A INPUT  -s "$MIRROR_IP" -p tcp --sport 443 \
  -m conntrack --ctstate ESTABLISHED -j ACCEPT

# Resolve the mirror name locally. No resolver means no DNS-shaped side channel,
# and no accidental fallback to a public registry that happens to be reachable.
echo "$MIRROR_IP mirror.internal" >> /etc/hosts
: > /etc/resolv.conf

# Strip the ambient environment. Inherited variables are inputs you did not
# write down, which makes them exactly the inputs that break a build six months
# from now for reasons nobody can reconstruct.
exec env -i \
  PATH=/usr/local/bin:/usr/bin:/bin \
  HOME=/work \
  SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:-0}" \
  NPM_CONFIG_REGISTRY="https://mirror.internal/npm/" \
  PIP_INDEX_URL="https://mirror.internal/pypi/simple" \
  PIP_REQUIRE_HASHES=1 \
  GOPROXY="https://mirror.internal/goproxy" \
  GOFLAGS="-mod=readonly" \
  bash "$1"

What counts as an input (more than you'd like)

The uncomfortable exercise is listing everything that can change a build's output. Source and dependencies are the obvious two. The toolchain — compiler, linker, interpreter, and their own shared libraries — is the one people forget until a patch release changes codegen. Then there's the environment: variables, locale, timezone, hostname, user and group IDs, the build path itself, the guest's clock, and any file the build reads outside the workspace. A hermetic build isn't one that has none of those; it's one where all of them are fixed and named, so the provenance's materials list is complete rather than optimistic.

Pre-warmed toolchain snapshots, so hermetic doesn't mean slow

The reason teams tolerate shared, mutable runners is speed: nobody wants a ten-minute dependency install in front of a ninety-second compile, and "hermetic" tends to arrive sounding like "we deleted the cache." The snapshot model resolves that tension without reintroducing shared mutable state. You build the environment once — toolchain installed, dependencies fetched from the mirror at pinned versions, everything laid out — and snapshot it. Every subsequent build restores from that snapshot instead of reconstructing it.

The economics matter here. On PandaStack a create restores a baked Firecracker snapshot rather than cold-booting: roughly 179ms p50 and ~203ms p99, with the restore step itself near 49ms. The first-ever boot of a new template is around 3s, after which every create takes the fast path. If a build matrix shares an expensive setup phase, do it once, snapshot the configured guest, and fork per matrix cell — a same-host fork lands in roughly 400–750ms, cross-host 1.2–3.5s, and each fork is still a fully isolated VM rather than a shared workspace with optimistic locking.

The crucial difference from a mutable cache is directionality. A snapshot is read-only input: the build restores from it, mutates its own private copy-on-write view, and that view is destroyed with the guest. No build can write into the snapshot that a later build restores from. Updating the snapshot is a deliberate, reviewed, separately-attested operation — which means the toolchain snapshot's own digest becomes a legitimate entry in the provenance materials list, and "what compiler built this" stops being a question you answer by SSHing into a runner and hoping it hasn't been reimaged.

Generating and signing the provenance where the build can't reach

Now the payoff. The guest is gone. What crossed the boundary was: artifact bytes, a build log, and an exit code. What never crossed it in the other direction: the signing key, the builder identity, and any credential that could impersonate the platform. The control plane hashes the artifact bytes it received, assembles an in-toto style statement from facts it knows independently — the source digest it verified, the toolchain snapshot digest it chose, the command it issued, the invocation id it generated — and signs. Every field in the statement is asserted by the party that observed it, and none by the party being described.

Note especially what the builder identity is: the control plane, not the guest. "Built by our platform" is a claim about the orchestrator that provisioned the VM, issued the command, and collected the output — which is the only component in this system that can honestly make it.

#!/usr/bin/env bash
# Runs on the CONTROL PLANE, after the build guest has already been destroyed.
# The key lives here and has never existed inside a build VM.
set -euo pipefail

ARTIFACT_DIGEST="$1"    # WE hashed the bytes; not a number the build printed
SOURCE_DIGEST="$2"      # verified before extraction, inside the guest
COMMIT="$3"

cat > provenance.json <<EOF
{
  "_type": "https://in-toto.io/Statement/v1",
  "subject": [
    { "name": "artifact.tar", "digest": { "sha256": "${ARTIFACT_DIGEST}" } }
  ],
  "predicateType": "https://slsa.dev/provenance/v1",
  "predicate": {
    "buildDefinition": {
      "buildType": "https://example.com/buildtypes/hermetic-microvm/v1",
      "externalParameters": {
        "repository": "https://github.com/acme/widget",
        "ref": "${COMMIT}",
        "command": "npm ci --ignore-scripts && npm run build"
      },
      "internalParameters": {
        "networkPolicy": "default-deny, internal mirror only",
        "lifecycleScripts": "disabled"
      },
      "resolvedDependencies": [
        { "name": "source.tar",  "digest": { "sha256": "${SOURCE_DIGEST}" } },
        { "name": "toolchain-snapshot", "digest": { "sha256": "${TOOLCHAIN_DIGEST}" } }
      ]
    },
    "runDetails": {
      "builder": { "id": "https://example.com/builders/hermetic-microvm" },
      "metadata": { "invocationId": "${BUILD_ID}" }
    }
  }
}
EOF

# Sign here, outside the guest, over bytes the guest cannot alter any more.
# Check predicate field names against the CURRENT SLSA provenance spec and the
# flags against your cosign version -- both have changed between releases.
cosign attest-blob \
  --predicate provenance.json \
  --type slsaprovenance1 \
  --output-attestation artifact.att.jsonl \
  artifact.tar

Four build architectures, compared honestly

• Shared CI runner — Build isolation: namespaces on one shared kernel, shared caches, shared runner user. Key exposure: any lifecycle script in any dependency runs as the identity that can reach the signing key. Input control: whatever the network allows. Provenance value: attests that something on the runner produced this; forgeable by the build itself. Cost: cheapest, and priced accordingly.
• Container per job on a shared host — Build isolation: better process separation, still one kernel and usually one cache volume. Key exposure: reduced if the release job is segregated, but cache poisoning still lets an untrusted job influence what the trusted job signs. Input control: possible but rarely enforced. Provenance value: honest about the command, silent about the inputs. Cost: low.
• MicroVM per build, signing inside the guest — Build isolation: real hypervisor boundary, no shared kernel or state between builds. Key exposure: total within that build — you moved the key into the same box as the untrusted code, which undoes most of what the boundary bought you. Input control: good. Provenance value: unforgeable by other builds, fully forgeable by this one. Cost: the isolation without the payoff.
• MicroVM per build, signing on the control plane — Build isolation: hypervisor boundary plus an ephemeral guest with no persistent identity to steal. Key exposure: none — the key is never on the guest's side of the boundary and the artifact digest is computed after the guest is destroyed. Input control: default-deny egress, mirror-only dependencies, toolchain pinned by snapshot digest. Provenance value: every field asserted by the party that observed it. Cost: a snapshot-restore per build, roughly 179ms p50 — the boring answer, which is the point.

What this does not give you

Three limits worth stating plainly, because provenance work attracts overclaiming the way monitoring dashboards attract green.

First: reproducible and hermetic are not the same property, and neither implies the other. Hermetic means the inputs are fixed and enumerated. Reproducible means the same inputs yield bit-identical output, which additionally requires the build to be free of embedded timestamps, absolute paths, nondeterministic iteration order, parallelism-dependent output, and embedded build ids. You can absolutely have a perfectly hermetic build that produces a different tarball every time. Hermeticity makes reproducibility achievable; it doesn't hand it to you. More on that distinction at /blog/reproducible-build-sandboxes.

Second, and most important: a signed provenance for a malicious build is a signed malicious build. Nothing in this architecture inspects what the source code does. If a contributor lands a backdoor and it passes review, the pipeline described here will hermetically compile it in a pristine guest and sign a beautifully-formed attestation saying exactly which commit produced exactly which artifact. Provenance answers "where did this come from" with high confidence. It does not answer "is this safe," and conflating the two is how you end up with a compromised release that has a perfect audit trail. Dependency review, SBOM generation and scanning (see /blog/microvm-sbom-vulnerability-scanning-isolation), and code review remain entirely your problem.

Third: an attestation nobody verifies is a very expensive log line. The provenance only becomes a control when something at deploy time refuses to proceed without it — an admission controller, a registry policy, a release gate that checks the signature chains to your builder identity, that the subject digest matches the artifact actually being deployed, and that the source repository and ref are ones you expected. Until that gate exists and fails closed, you have built an elaborate machine for producing JSON. Secrets handling for the deploy side of that gate is its own topic: /blog/secure-ci-secrets-microvm.

The summary

SLSA's build requirements come down to one property: the build must not be able to influence its own provenance, or anyone else's. A shared runner violates that by construction — the key, the cache, and the next job's workspace are all reachable by every postinstall script in the dependency tree. The microVM shape fixes it structurally rather than procedurally: one ephemeral guest per build with no persistent identity in it, hermetic inputs enforced by default-deny egress and a mirror-only dependency path, a pre-warmed toolchain snapshot so hermeticity costs a snapshot-restore rather than a ten-minute install, and a signer that lives on the control plane and hashes the artifact bytes itself after the guest no longer exists. Then go build the deploy-time gate that actually checks the thing — and verify every schema detail against the current SLSA spec, which will have moved again by the time you ship.