Running Nix Builds Inside a Disposable VM

There is a specific moment where running Nix in CI stops being a purity exercise and becomes an infrastructure decision. It is the moment the flake you are building is not yours. A contributor's pull request, a vendored third-party flake, an internal team you trust but whose derivations you have not read, an AI agent that has been handed a repository and told to make the build pass. At that point the question changes from "is this build reproducible" to "what happens to this machine if the build is hostile," and Nix has a careful, honest, frequently misread answer to that second question.

The answer is: not much, and that is by design. I have argued the conceptual version of this elsewhere — the Nix sandbox and a hypervisor are answering different questions, and it is a category error to treat them as competitors. This post is the operational half. Assume you have accepted the argument and want to actually run `nix build` inside a disposable virtual machine. What does that cost, where does it hurt, and what do you have to build so that it is not slower than the thing you replaced?

What Nix's sandbox covers, in one paragraph

When Nix realises a derivation with sandboxing enabled — the default on Linux, not on macOS — it does not run the build script in your shell. It builds a restricted environment out of Linux namespaces: a fresh mount namespace containing only the store paths the derivation declared, a private network namespace with no route anywhere, private PID and IPC namespaces, and a chroot-style root. The build sees its declared inputs and essentially nothing else. No `$HOME`, no ambient `/usr/local`, no `~/.cargo`, no live network, no environment variable you forgot you had exported. That is what makes the output a function of the inputs rather than a function of the machine.

The network exception is the tell. Because the sandbox has no network, anything that must fetch has to be a fixed-output derivation: you declare the hash of the result up front, and Nix opens a controlled hole precisely because the output is pinned. If the bytes do not match, the build fails. The honesty is coming from the hash, not from the confinement.

None of this is a criticism of Nix. It is doing the job it advertised, unusually well. It is just a job that stops one layer short of the one you need when the derivation is untrusted, and the fix is not to make Nix's sandbox stronger — it is to put a machine boundary underneath it. Nix keeps the build honest; the VM keeps a dishonest build off your host.

The cold store tax, stated plainly

Here is where the naive version of this design falls over. You spin up a fresh VM per build. The VM is clean, which is the point. It is also empty, which is the problem. A Nix build on a machine with an empty `/nix/store` has to pay three separate costs before it does one second of your work:

1. Installing Nix itself. Downloading and unpacking the installer, creating the store, setting up the daemon and build users. This is on the order of a minute, and it is pure overhead — the same minute, on every job, forever.
2. Evaluating the flake. Nix has to fetch the flake inputs, which means git clones and tarball fetches of nixpkgs and every other input, and then evaluate the Nix expression. On a cold machine with no evaluation cache, nixpkgs evaluation is not free.
3. Realising the closure. Every dependency of what you are building has to arrive. Even with a perfect binary cache this is a large download — a toolchain closure is comfortably gigabytes — and without one, it is a full build of your dependency tree from source, which is the difference between three minutes and three hours.

The third one dominates and it is worth being precise about why. `/nix/store` is not a cache in the sense of an optional accelerator you can lose cheaply. It is the materialised result of every build that has ever run on that machine, keyed so that Nix can prove it does not need to run them again. Losing it does not slow the build down; it changes the build into a different, much larger build. Every disposable-VM design for Nix is, underneath, a design for how the store survives the VM.

The disposability you want is of the machine. The disposability you emphatically do not want is of the store. Every good version of this architecture is a way of keeping those two separable.

Bake the install, don't pay for it

The first cost is the easy one, and the fix is the same one that works for every other slow-provisioning problem: do it once, at image build time, and never at job time. Nix installs into a directory. Directories go into images. A custom template built from a Dockerfile gets you a machine that already has Nix, already has the daemon configured, and already has whatever store paths you chose to pre-populate.

On PandaStack the template is the unit here: you hand it a Dockerfile, it becomes a rootfs, and the first spawn of that template cold-boots once and bakes a Firecracker snapshot. Every create after that is a restore of that snapshot rather than a boot — roughly 179ms p50 to get a machine that already believes it has been running for a while. What you put in the image is free at job time in a way that nothing you do at job time ever is.

Which raises the question of how much of the store to bake in. My answer, and I will flag it as an opinion: bake the toolchain closure and nothing else. The things that are identical across every job — the compiler, the standard library, the base nixpkgs paths your flake pins — belong in the image, because they are pure resident weight with no invalidation story. The things that change per commit do not, because baking them means rebaking the image on every dependency bump, and you have just reinvented a slow CI image pipeline to avoid a fast download.

# /etc/nix/nix.conf -- baked into the image, not written at job time.

experimental-features = nix-command flakes

# On Linux this is the default. Set it explicitly anyway: this file doubles as
# the documentation of what the machine promises, and a default you inherited
# is a default someone can change without noticing.
sandbox = true

# Substituters are the only reason a fresh store is survivable. Order matters:
# your own cache first, upstream second. Public keys are how Nix decides a
# substituted path is trustworthy -- an unsigned cache is a supply chain.
substituters = https://cache.example.internal https://cache.nixos.org
trusted-public-keys = cache.example.internal:AAAA0000...= cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=

# A VM waiting on one substituter connection at a time is a VM you are paying
# for by the second. Tune these against your cache, not against this post.
http-connections = 50
max-substitution-jobs = 32

# Do not let a per-job GC eat the closure you just warmed.
keep-outputs = true
keep-derivations = true

# Leave allowed-users/trusted-users tight. A "trusted" user can add its own
# substituters and override sandbox settings, which in a build VM means the
# build can opt itself out of the confinement you configured.
trusted-users = root
allowed-users = root builder

# Option names and defaults move between Nix releases. Check the manual for
# the version you actually ship rather than trusting a config you copied.

Two lines in there matter more than the rest. `trusted-public-keys` is the one people get casual about: a substituter is a machine that hands you prebuilt binaries which you then execute, and the signature check is the only thing standing between "fast build" and "someone else's binary in your artifact." And `trusted-users` is the one people get generous with, because loosening it makes an error message go away — but a trusted user can override substituters and sandbox settings, which is to say it can undo the config file above from inside the build.

Three ways to warm the store, and what each costs

Cost three — getting the closure into a fresh machine — has three real answers. They are not mutually exclusive and the good setups use at least two.

A binary cache you control

This is the load-bearing one and everything else is an optimisation on top of it. A binary cache serves prebuilt store paths so a build that would have compiled something instead downloads it. `cache.nixos.org` covers upstream nixpkgs; it does not cover your code, your overlays, or your patched dependencies, which is exactly the expensive part. So you run your own — Cachix if you want it hosted, attic if you want to self-host, or a plain S3-compatible bucket serving a signed binary cache, which is less magic than either and works fine.

The mechanism to wire in is the post-build hook: after a build produces a path, push it to your cache so the next machine gets it as a download. Without that, your VMs are read-only consumers of a cache nobody populates, and the first build of anything is always the slow one. With it, the fleet compounds — one machine pays for a new dependency and every subsequent machine, in every subsequent job, does not.

Snapshotting a warm store

A binary cache still means a download per job, and a large closure over a network is not nothing. The stronger move is to make the warm store part of the machine you restore. Boot one VM, realise the closure you care about, snapshot the whole machine, and create every subsequent job from that snapshot. The store is not downloaded, it is already there — memory and disk restored together, no substitution step at all for anything in the baked set.

This is where a snapshot-restore platform earns its keep, and it is the same trick as pre-baking an image except that you can do it cheaply and often, keyed to whatever granularity you like: per flake, per branch, per nixpkgs pin. The tradeoff is honest — a snapshot is a point in time, so a snapshot warmed against last week's lockfile is warm for last week's dependencies and cold for whatever changed. Re-warm on a schedule, or on lockfile change, and treat a stale snapshot as a cache miss rather than a correctness problem, which is exactly what it is.

A shared store volume

The third option is to keep `/nix/store` on a durable volume and mount it into each job. It is the fastest to hit and the one I would think hardest about, because a writable store shared across jobs is precisely the cross-job channel the disposable VM was supposed to eliminate. Nix's store is not a naive shared directory — the daemon mediates writes, and paths are keyed by hash — but you are still handing every job a mutable surface that every other job reads. If the jobs are all first-party and mutually trusting, that is a reasonable trade for real speed. If any job is untrusted, mount it read-only or do not mount it.

The shape: a flake build in a fresh machine

Put the pieces together and the per-job code is unremarkable, which is the goal. A machine restored from a snapshot that already has Nix and a warm store, a build that reads from the cache but cannot write to it, no network for the build itself beyond what fixed-output derivations need, and a teardown that is a fact rather than a cleanup script.

# One untrusted flake build, one microVM, no residue.
# The machine is restored from a snapshot whose /nix/store is already warm
# for the pinned nixpkgs -- so this is a build, not an installation.
import json
import shlex

from pandastack import Sandbox

WARM = "snap_nixpkgs_2026_09_01"   # re-baked on lockfile change, see below


def build_flake(repo_url: str, rev: str, attr: str = "default"):
    sbx = Sandbox.create(
        from_snapshot=WARM,            # ~179ms p50; store already resident
        ttl_seconds=1800,              # backstop -- the guest reaps itself
        metadata={"repo": repo_url, "rev": rev},
    )
    try:
        # The repo is untrusted input, so it arrives as data. Note --depth 1:
        # you are building a revision, not browsing a history.
        sbx.exec(
            f"git clone --depth 1 {shlex.quote(repo_url)} /work/src",
            timeout_seconds=180,
        )
        sbx.exec(f"cd /work/src && git checkout {shlex.quote(rev)}", timeout_seconds=60)

        # --no-write-lock-file: an untrusted build does not get to decide which
        # inputs it resolves to. If the lock is stale, that is a review problem,
        # not something the build fixes for itself at 3am.
        run = sbx.exec(
            "cd /work/src && nix build .#" + shlex.quote(attr) + " "
            "--print-build-logs "
            "--no-write-lock-file "
            "--option sandbox true "        # belt and braces; already the default
            "--option substitute true "     # read from the cache
            "--option post-build-hook ''",  # ...and write nothing back to it
            timeout_seconds=1500,
        )

        if run.exit_code != 0:
            return {"ok": False, "log": run.stderr[-20000:]}

        # Read out only what you declared you wanted. "Tar up /work" is how a
        # build smuggles a file into the place your release pipeline trusts.
        paths = json.loads(
            sbx.exec("cd /work/src && nix path-info --json ./result", timeout_seconds=60).stdout
        )
        artifact = sbx.filesystem.read("/work/src/result/bin/app")
        return {"ok": True, "store_paths": paths, "artifact": artifact}
    finally:
        sbx.kill()   # the only cleanup step, and it cannot half-succeed

The `post-build-hook ''` line is the small one that matters. It is the cache boundary drawn separately from the execution boundary: this build may read every prebuilt path in the world and contribute none of its own. An isolated build that still writes into the cache your release pipeline consults has not been isolated, it has been given a slower, better-audited path to the same outcome.

The re-warm side is the other half, and it is a scheduled job rather than a per-build one:

from pandastack import Sandbox

# Run this on lockfile change (or nightly). It pays the cold-store cost once
# so that every job created from the resulting snapshot pays none of it.
def rewarm(flake_url: str) -> str:
    sbx = Sandbox.create(template="nix-builder", ttl_seconds=3600)
    try:
        # Realise the *dependencies* without building the project itself: this
        # is the part that is identical across every commit on the branch.
        sbx.exec(f"nix flake prefetch {flake_url}", timeout_seconds=600)
        sbx.exec(
            f"nix build {flake_url}#default.inputDerivation --no-link",
            timeout_seconds=3600,
        )
        # Trim what the warm set does not need before freezing it: every
        # gigabyte in the snapshot is a gigabyte in every restore.
        sbx.exec("nix store gc --max 2G", timeout_seconds=600)
        sbx.exec("nix store optimise", timeout_seconds=900)

        snap = sbx.snapshot()   # memory + disk; the store rides along
        return snap.id
    finally:
        sbx.kill()

`inputDerivation` is the useful trick there: it gives you a derivation whose inputs are your package's build inputs, so realising it warms the dependency closure without building the thing you are actually going to build per commit. Check the current behaviour against the manual before you lean on it — this area of nixpkgs has moved before — but the principle survives any specific attribute name. Warm what does not change; build what does.

Network policy: the sandbox already did half of this

Nix's build sandbox gives the build no network, and that is a genuinely strong default that most CI systems do not have. But it constrains the *build*, not the machine the build runs on. Between the sandbox's boundary and the internet sits everything else: the flake evaluation that fetches inputs, the substituter downloads, the `git clone`, the fixed-output derivations that are allowed out by construction, and anything a compromised build manages to run outside a derivation — a hook, a `nix-shell` script, a shell command in your own job wrapper.

So the VM's egress policy is a separate control from `sandbox = true`, and it is the one that decides whether a build can reach your cloud metadata endpoint. Each PandaStack sandbox lives in its own pre-allocated network namespace, which makes "this machine may reach the binary cache, the git remote, and nothing else" a rule on a namespace that exists for one job rather than a policy you hope is applied to the right container. Allowlist the substituters, allowlist the git host, and drop the rest — including, specifically, link-local metadata addresses.

Fixed-output derivations are the interesting case because they are the sanctioned exception. A build can reach the network through one, but only to fetch bytes whose hash it declared in advance — so the exfiltration channel is narrow and the ingestion channel is pinned. It is still a channel, and if your threat model includes low-bandwidth exfiltration through DNS or request timing, an allowlist at the namespace level is the thing that closes it. For most people, most of the time, the hash pinning is sufficient and the allowlist is about metadata endpoints rather than exotic covert channels.

Remote builders versus per-job VMs

Nix has a native answer to "build somewhere else": remote builders. You list machines the daemon can offload to, mark your local machine with zero build slots so everything goes out, and Nix ships derivations over SSH, builds them remotely, and copies the results back. It is well-worn, it works, and it is often the right answer.

It is worth being clear about what it does and does not change. A remote builder is a *placement* mechanism, not an isolation one. The builder machine is typically long-lived, shared across everyone who can offload to it, and accumulates a store that is exactly the thing you want it to accumulate. Two derivations from two different trust domains land on the same box, run as `nixbld` users on the same kernel, and share a store. For first-party builds that is a feature — the shared store is the whole speedup. For untrusted flakes it is a fleet of persistent machines that anyone who can open a build can execute code on.

• Remote builders — what they give you: native Nix, no orchestration to write, a shared warm store, easy cross-architecture (an aarch64 builder for your x86 laptop). What they do not give you: isolation between the derivations that land on them, or a clean machine per job. Right answer for: trusted first-party builds, cross-platform offload, and teams who want one less system.
• Per-job microVMs — what they give you: a guest kernel per job, a machine that has never seen a previous build, its own network namespace, and a teardown that cannot partially fail. What they cost: you own the warm-store problem, because there is no long-lived box quietly accumulating one for you. Right answer for: untrusted flakes, forked pull requests, agent-authored builds, and anything you would not run on your laptop.
• Both — remote builders whose backing machines are per-job microVMs. The Nix-native interface stays, the persistence goes. This is more moving parts than either, and it is the right shape if you are running a build service rather than a build farm for one team.

The split I would actually deploy mirrors the one I would deploy for any build system: trusted first-party builds go to warm shared builders because throughput is the only thing that matters there and the marginal security value of isolating your compiler from your compiler is approximately zero. Everything else — forks, vendored flakes, anything an agent wrote — gets its own machine. The interesting engineering is not in either lane; it is in making sure the untrusted lane can read the cache that the trusted lane fills, and cannot write to it.

Daemon mode or single-user mode?

A multi-user Nix install runs a daemon as root and executes builds as unprivileged `nixbld` users, so that no user on a shared machine can write into the store directly. A single-user install skips the daemon and runs builds as the invoking user, who owns the store. On a shared workstation, multi-user is obviously correct: the daemon is the thing preventing one user from corrupting everyone's store.

In a disposable single-tenant VM, that reasoning mostly evaporates, and this is one of the few places where the VM genuinely simplifies rather than complicates. There is one tenant. The store dies with the machine. The scary property of single-user mode — "the build can write to the store" — costs you nothing when the store is a per-job artifact that no other job will ever read. Single-user mode has fewer moving parts at boot, one fewer thing to wait for, and no daemon socket to reason about.

One practical wrinkle either way: Nix's sandbox needs the kernel features it is built on, and environments that restrict user namespaces can make sandbox setup fail. There is a fallback option that lets Nix build without the sandbox when it cannot set one up, and in a build fleet that option is a trap — a silent downgrade from hermetic to not-hermetic is exactly the failure you will not notice until a build starts passing for the wrong reason. Fail the build instead, and verify the current flag semantics against the manual for your version.

The honest arithmetic

The reason to do all of this is a specific one, and if it does not apply to you then a warm shared builder is a better use of your afternoon. It applies when the flake is untrusted, when the build is long enough that machine setup disappears into it, or when you need genuine per-job cleanliness for reasons beyond security — investigating a build that only fails on a clean store, for instance, which is a real and miserable class of bug.

The numbers work out roughly like this. A machine restored from a warm snapshot arrives in about 179ms, which against a four-minute build is 0.07% overhead and is not worth another sentence. A cold machine that has to install Nix and realise a toolchain closure from scratch is minutes, which against the same build is most of it. That gap is the entire engineering problem in this post, and it is why the warm-snapshot and binary-cache sections are longer than the isolation ones. The isolation is the easy part; the platform hands it to you. Not making the isolation expensive is the part you actually have to design.

On cost, the shape is worth internalising even if your numbers differ: a build VM is billed for the time it exists, so at $0.054 per vCPU-hour and $0.0162 per GiB-hour, a four-minute build on a 4-vCPU, 8 GiB machine is a few cents. The thing that makes per-job VMs expensive is not the VM. It is the twenty minutes of cold-store rebuild you accidentally put inside it.

Nix already solved reproducibility and it did not ask you for a VM. What it did not solve — because it never claimed to — is what happens when the derivation is written by someone who wants your host. That is a machine boundary, and machine boundaries are cheap now. The store is the expensive thing. Design for the store.