Isolating CI Build Caches Per-Job with MicroVMs

A build cache is one of the most under-scrutinized pieces of trust in a CI pipeline. It's a directory — `~/.npm`, `~/.m2`, `~/.gradle/caches`, `~/.cache/pip`, a ccache store, a Bazel disk cache — that every job on a shared runner reads from and writes to, and whose contents flow straight into the artifacts you ship. Put bluntly: your build cache is a shared mutable global variable that ships to production. On a shared runner it's mutated by every job that ran before yours, including jobs building untrusted forks, third-party PRs, or a dependency that got compromised last Tuesday.

The tension is real, because caches exist for a good reason: cold `npm ci` or a from-scratch Gradle build is painfully slow, and a warm cache is the difference between a 30-second job and a 6-minute one. So teams reach for a shared, persistent cache on a long-lived runner — and quietly hand every future job a writable channel into every future build. This post is about resolving that tension: keeping the cache-hit-rate that makes CI bearable while removing the cross-job trust bleed that makes shared caches dangerous. The tool that resolves it cleanly is a per-job Firecracker microVM restored from a snapshot with the cache already warm, and thrown away when the job ends. I'm Ajay — I built PandaStack on exactly this pattern.

How a shared build cache actually gets poisoned

The threat isn't hypothetical, and it isn't only about "malicious PRs." The cache is writable by the build, and modern builds run arbitrary code from the dependency graph: npm lifecycle scripts (`postinstall`), pip's `setup.py`, Gradle build scripts, Maven plugins. Any of them can write into the cache directory. If a job's dependency tree contains one compromised package, that package runs with the job's permissions and can tamper with the shared cache before it exits.

Cross-job cache poisoning

The concrete attack: job A builds a branch containing a compromised transitive dependency. Its `postinstall` script overwrites a cached package tarball, a resolved wheel, or a ccache object with a trojaned version that keeps the same name and — if the cache trusts filenames over content hashes — the same apparent identity. Job B, an entirely different repo or a trusted mainline build, runs next on the same runner, pulls that poisoned entry out of the warm cache, and links it into a release artifact. Nothing in job B's own source or dependency list looks wrong. The poison came sideways, through the shared filesystem, from a job B never saw.

Secret and state leakage

The same shared surface leaks in the other direction. A cache directory on a shared runner sits next to whatever the previous job left behind: a `.netrc`, a leaked token in `~/.npmrc`, a `.git-credentials`, build outputs, temp files. A malicious build doesn't even need to break out of a container to read those — they're right there on the shared kernel's filesystem, in another user's home or a world-readable tmp. Persistent shared runners accumulate this residue over hundreds of jobs.

Why per-job containers don't fully close the gap

The obvious first fix is a fresh container per job — and it genuinely helps: a new container starts from a clean image layer, so it doesn't inherit the previous job's home directory. But two things pull you back toward shared state. First, to keep cache-hit-rate you almost always mount a shared cache volume into the container (`-v cache:/root/.npm`), which re-introduces the exact writable shared directory you were trying to escape. Second, and more fundamentally, containers on a runner share one host kernel. A container escape — a real, recurring class of bug — puts a malicious build back onto the host filesystem where every other job's cache and secrets live.

So the container story is: clean-per-job is good, but the cache mount and the shared kernel are the two seams where trust still bleeds. To keep the cache fast you re-open the poisoning channel; to keep it isolated you'd have to give up the shared cache. A microVM lets you keep both because the isolation boundary is a hardware-virtualized guest kernel, not a namespace on the host's.

The per-job microVM pattern: warm cache, zero bleed

Here is the shape of the pattern. You bake a runner template once with the toolchain installed and the cache pre-warmed — do a representative build inside it so `~/.npm`, `~/.m2`, `~/.gradle`, or your ccache store fills up, then snapshot the VM. Every CI job then restores a fresh microVM from that snapshot, runs its build against the already-warm cache, has its artifacts extracted, and is destroyed. The next job restores its own fresh copy of the same snapshot. No two jobs ever touch the same live cache.

1. Snapshot-restore a runner VM from a template whose cache is already warm (baked at snapshot time).
2. Optionally seed a job-scoped cache on top — e.g. write a per-tenant lockfile or a scoped registry config into the guest.
3. Run the build (install, compile, test) inside the guest. Every cache write it makes lands in the guest's private, copy-on-write memory and disk.
4. Extract the artifacts you want — a built bundle, a wheel, a container image tarball — via the filesystem API.
5. Destroy the VM. All of the job's cache mutations vanish with it; the baked warm cache underneath was never modified.

The key is that step 3's writes never persist. A poisoned `postinstall` can trojan a cached tarball all it wants — that tarball lives in one throwaway guest and is deleted 40 seconds later. The next job restores a pristine copy of the baked cache, byte-for-byte, with none of the previous job's writes. The poisoning channel is closed not by scanning the cache but by making writes non-durable.

How copy-on-write resolves the speed-vs-isolation tension

The reason this doesn't cost you cache-hit-rate is copy-on-write. The obvious naive implementation — give each job its own full copy of a multi-gigabyte cache — would be slow and enormous. CoW makes the warm cache a shared read-only baseline: every job's VM maps the same underlying cache pages and disk blocks, and only the pages and blocks a job actually writes get privately duplicated. Reads are shared (fast, no copy); writes are private (isolated, invisible to neighbors).

This is what lets you have both properties at once. High hit-rate comes from every job seeing the same fully-warm cache. Full isolation comes from the fact that a job can only ever mutate its own CoW copies of the pages it touches — it cannot reach the shared baseline or any other job's private pages. On PandaStack the memory side is `MAP_PRIVATE` on the restored snapshot (the kernel copies a page only on first write), and the disk side is filesystem reflink / dm-snapshot CoW on the rootfs. Restoring the snapshot is the fast path: the create step's snapshot-restore is around 49ms, with sandbox creation landing at roughly 179ms p50 (about 203ms p99). Only the very first cold boot of a template — before there's a snapshot — costs a few seconds; after that every job takes the restore path.

Shared-runner cache vs per-job container vs per-job microVM

• Cache-hit-rate — Shared runner: high (one persistent warm cache). Per-job container: high if you mount a shared cache volume, low if you don't. Per-job microVM: high (CoW-shared warm baseline, no volume mount needed).
• Cross-job write isolation — Shared runner: none (all jobs write the same cache). Per-job container: partial (clean home, but the mounted cache volume is still shared-writable). Per-job microVM: full (writes are private CoW overlays, discarded on destroy).
• Isolation boundary — Shared runner: a user/dir on one host kernel. Per-job container: namespaces + cgroups on one shared host kernel. Per-job microVM: a hardware-virtualized guest kernel per job.
• Blast radius of a compromised dependency — Shared runner: every later job on the runner. Per-job container: the container, plus the shared cache volume and (via escape) the host. Per-job microVM: one disposable VM.
• Secret/residue leakage across jobs — Shared runner: high (accumulated home-dir residue). Per-job container: low for home dir, but shared kernel + mounts remain. Per-job microVM: none across VMs (fresh guest each time).
• Startup cost per job — Shared runner: ~0 (already up). Per-job container: image/layer + runtime init. Per-job microVM: snapshot-restore fast path (~49ms restore step; ~179ms p50 create on PandaStack). Verify container and other-platform numbers against their own docs.

A concrete walkthrough

Below is the runner-side shape in shell, then the same thing driven programmatically through the PandaStack Python SDK. The shell version is what you'd conceptually run inside one job's guest; the SDK version is the orchestrator that spins the guest up, runs the job, pulls the artifact, and throws the VM away.

The build step, inside one throwaway guest

#!/usr/bin/env bash
set -euo pipefail

# Point the toolchain at the caches baked into the snapshot.
# These dirs are already warm from the template bake, so 'npm ci'
# and the pip install are near-instant cache hits.
export npm_config_cache=/opt/cache/npm
export PIP_CACHE_DIR=/opt/cache/pip
export GRADLE_USER_HOME=/opt/cache/gradle

cd /workspace/repo

# Install + build. Any postinstall / setup.py that runs here can write
# into /opt/cache freely -- those writes are this guest's PRIVATE
# copy-on-write pages. They never touch the shared baked cache, and
# they die when this VM is destroyed after the job.
npm ci --no-audit
npm run build

# Emit the artifact to a known path the orchestrator will read back.
tar -czf /workspace/artifact.tgz -C /workspace/repo/dist .
sha256sum /workspace/artifact.tgz > /workspace/artifact.sha256
echo "build ok"

Notice there is no cache-scrubbing step and no allowlist scan. We don't need to prove the cache is clean because a dirty cache from this job can't reach the next one — non-durable writes make the poisoning channel structurally absent rather than something we police.

The orchestrator, with the Python SDK

from pandastack import Sandbox

# The build script we want to run inside the isolated guest.
BUILD = open("build_step.sh").read()

def run_ci_job(repo_tarball: bytes) -> bytes:
    """Run one CI build in a fresh, warm-cache microVM and return the artifact."""
    # ttl_seconds is a backstop: even if we crash before kill(), the VM reaps itself.
    with Sandbox.create(template="ci-runner-warm", ttl_seconds=900) as sbx:
        # Seed this job's source and the build step into the private guest.
        sbx.filesystem.write("/workspace/repo.tgz", repo_tarball)
        sbx.filesystem.write("/workspace/build_step.sh", BUILD)

        # Unpack source, then run the build. All cache writes it makes are
        # this guest's copy-on-write overlay -- invisible to every other job.
        unpack = sbx.exec(
            "mkdir -p /workspace/repo && "
            "tar -xzf /workspace/repo.tgz -C /workspace/repo",
            timeout_seconds=60,
        )
        assert unpack.exit_code == 0, unpack.stderr

        build = sbx.exec("bash /workspace/build_step.sh", timeout_seconds=600)
        if build.exit_code != 0:
            raise RuntimeError(f"build failed:\n{build.stderr}")

        # Extract the artifact bytes before the VM is destroyed on block exit.
        artifact = sbx.filesystem.read("/workspace/artifact.tgz")
        checksum = sbx.filesystem.read("/workspace/artifact.sha256")
        print("artifact:", len(artifact), "bytes;", checksum.decode().strip())
        return artifact
    # VM destroyed here -- every cache mutation from this job is gone.

# Two different tenants' jobs never share a live cache: each gets its own
# fresh restore of the same warm snapshot.
# artifact_a = run_ci_job(tenant_a_repo)
# artifact_b = run_ci_job(tenant_b_repo)

Two jobs from two tenants call `run_ci_job` independently. Each gets its own microVM restored from the same warm `ci-runner-warm` snapshot, so both start with a fully-warm cache — but tenant A's `postinstall` writes and tenant B's `postinstall` writes land in two separate copy-on-write overlays that can't see each other and both evaporate on `kill()`. Same speed as a shared cache, none of the shared-cache trust.

When this is worth it — and when it isn't

Reach for per-job microVM cache isolation when the code you build is untrusted or multi-tenant: a SaaS CI product building customer repos, a platform that runs third-party PRs or forks, a monorepo where one team's compromised dependency shouldn't be able to poison another team's release. The higher the mix of trust domains sharing a runner, the more the shared cache is a liability and the more this pattern earns its keep.

It's overkill when every job is first-party and fully trusted — your own team building your own code on runners only you control. There, a shared persistent cache is a reasonable trade: the poisoning vector requires an attacker already inside your dependency graph, and you may accept that risk for simplicity. The honest framing is the same as with any isolation boundary: the microVM protects you from the code it runs, not from a threat model where you already trust that code. Match the boundary to the trust, and don't pay for VM-grade isolation to run builds you wrote yourself.

One caveat worth stating plainly: the warm baked cache is only as trustworthy as the build that warmed it. Bake the template from a known-good, audited dependency set, and re-bake it on a schedule so it doesn't drift stale — a poisoned baseline gets faithfully forked to every job. Isolation stops jobs from poisoning each other; it doesn't bless the thing you started from.