Ephemeral Jenkins Agents on Firecracker MicroVMs

Every long-lived Jenkins fleet eventually produces the same sentence, in the same weary tone: "it works on agent-04." It gets said as a diagnosis, as if it were a scheduling quirk to be worked around with a label. It should be read as an architecture diagram. What that sentence means is that the correctness of your build depends on the accumulated, undocumented history of one specific machine — every apt-get someone ran during an incident, every symlink added to make a plugin work in 2019, every environment variable exported from a root shell by an engineer who has since left the company. The build is not reproducible. The build is a ritual performed at a particular shrine.

Jenkins is the reason this is worth writing about rather than sneering at. It runs an enormous amount of the world's actual software delivery, in places where nobody is going to migrate to a new CI platform this quarter, and its extensibility is precisely why those agents drifted: the plugin ecosystem made it easy to install things onto a running node and hard to prove what is on one. This post is about a targeted change — keep the controller, keep the Jenkinsfiles, and replace the persistent agent with a Firecracker microVM that exists for exactly one build and is then deleted.

The static agent: a museum with a Jenkinsfile

Start with an honest inventory of what an eight-year-old build agent actually contains, because the drift is never one big thing. It is a thousand small ones, each individually reasonable at the time.

• A snowflake toolchain. Three JDKs, two of which are on PATH depending on which shell initialisation file wins, plus a global npm prefix owned by root because someone once installed with sudo. Nobody knows which one a given job resolves until it fails.
• $HOME residue. The jenkins user's home directory is a shared mutable workspace every job has written to for years: stray ~/.npmrc and ~/.netrc files, an ~/.aws/credentials that predates instance roles, and a ~/.config full of tool state that changes build behaviour silently.
• Docker layers and volumes. If builds run containers, the node's image store is a landfill: dangling layers filling the disk, a base image four months stale that a job resolves by tag instead of digest, and volumes from a docker-compose test nobody cleaned up.
• One ~/.m2 and one ~/.gradle for every job on the node. This is the big one. Maven's local repository and Gradle's caches are shared mutable state across every pipeline scheduled here — a job that runs `mvn install` writes its own artifacts into the repository that the next, unrelated job resolves from. Cross-job contamination is not a bug in that design; it is the design, applied to a multi-tenant machine.
• Left-behind processes. A test that forked a database and never reaped it. A Gradle daemon from a build two weeks ago, holding a stale classpath. A port-4000 listener that makes exactly one job flaky, exactly one time in five.

And the fleet has a folk remedy for all of it, which every platform team recognises: the agent is fine, we just reboot it on Fridays. That works, in the sense that a reboot clears the leaked processes and the full tmpfs. It does not touch the toolchain drift or the poisoned caches, which are exactly the parts that make builds non-reproducible. Meanwhile the actual cleanup control most pipelines use is `rm -rf $WORKSPACE` (or its polite Jenkins spelling, `deleteDir()`), which deletes the directory the build was told to use and nothing else. As a hygiene measure it is fine. As a security control it is close to theatre: a build step that wanted to persist would not have chosen the one directory whose entire purpose is to be deleted.

What a Jenkins build actually is, security-wise

The uncomfortable framing is this: a Jenkins job is arbitrary Groovy and arbitrary shell, authored by whoever can merge a change to a Jenkinsfile, executed as the agent user on a machine that also runs everyone else's builds. That is not a criticism of Jenkins — it is what a CI system is for. But it means the population of people who can execute code on agent-04 is the population of people who can land a commit in any repository that builds there, plus anyone who can influence a dependency those repositories resolve.

Jenkins itself is unusually candid about the resulting trust boundary. The project documents agent-to-controller access control as a first-class security concept, precisely because the controller is the high-value target and agents are the less-trusted half of the relationship: it holds the credentials store, the job configuration, the plugin surface and, in many installs, the ability to run code on every other agent. Jenkins ships a subsystem that filters what an agent may ask the controller to do because a compromised agent walking up that channel is a real, anticipated attack — not a hypothetical invented for a conference talk.

• Build steps run as the agent user. Anything that user can read, the build can read: other jobs' workspaces on the same node, the shared caches, the agent's own working directory and its connection secret.
• The controller is the crown jewels. Agent → controller is a documented boundary with its own access-control rules; treat every agent as a machine that will one day be running hostile code and might probe that channel.
• Credentials binding is real secret exposure, by design. When a pipeline binds a credential, Jenkins puts the actual value into the build's environment for the duration of the block. It masks it in the console log, which is a usability feature, not containment — the process, its children, and anything that can read /proc on that node see the real value.
• Shared caches are a lateral-movement path. A job that can write to ~/.m2 or ~/.gradle influences what every later job on that node resolves and executes. That is code execution with a delay fuse.
• Nothing about a persistent agent expires. A planted cron entry, a modified ~/.bashrc, a wrapper script earlier on PATH — all of it survives the build, the reboot on Friday, and quite possibly the engineer who noticed something felt off.

On a persistent agent, the blast radius of a bad build step is not the build. It is every build that machine ever runs afterwards.

The dynamic-agent options you already have, and their honest limits

None of this is news to Jenkins, and the ecosystem has three well-trodden answers. Each of them is genuinely good at something, and each has a limit worth stating plainly. Treat the characterisations below as qualitative — verify current behaviour and configuration options against each plugin's own documentation, since they move.

• The Docker plugin gives you a fresh container per build. This is a large, real improvement: the toolchain comes from an image you can rebuild from a Dockerfile, the filesystem starts clean, and $HOME residue stops accumulating. What it does not give you is a kernel boundary — every build container on a node shares the host kernel, so a container escape or a kernel bug reached by a build step lands you on the host, next to the other builds and the Docker socket.
• The Kubernetes plugin gives you a fresh pod per build plus real scheduling, and is the modern default for good reasons. Same isolation caveat: pods on a node share that node's kernel. It also adds a Kubernetes-shaped operational surface — pod templates that drift from the Jenkinsfile, service accounts to scope, and the classic Docker-in-Docker question, whose usual answers are a privileged container or the host's Docker socket mounted into a build. Both are exactly the boundary you were trying to build.
• The EC2 plugin (and the fleet variants) gives you a fresh virtual machine per build, which is the isolation everyone actually wants: a separate kernel, a separate machine, terminated afterwards. The cost is latency and money — you wait on instance provisioning and boot before a build can start, and you pay by the instance. Which is why almost every team that adopts it sets an idle timeout measured in tens of minutes and keeps a warm pool. The moment an instance is reused across builds, you have re-created the snowflake agent you paid extra to avoid — slowly, on a small scale, but the same shape.

That last point is the trap worth naming clearly. Teams do not lose freshness because they stopped caring about it; they lose it because keeping a VM warm is the only affordable way to hide a slow provision. The freshness was the whole product, and it gets traded away to make the latency tolerable.

The microVM shape: a fresh guest per build

A Firecracker microVM is what sits between those options. It is a real virtual machine — its own guest kernel, its own memory, its own virtual devices, isolated by the CPU's virtualization extensions under KVM — with a device model small enough that the machine starts in the time range you associate with containers rather than instances. The pitch, stated carefully: EC2-plugin freshness at container-plugin latency. Not because microVMs are magic, but because the expensive part of "start a machine" is booting one, and a snapshot restore skips the boot.

The lifecycle for a build is deliberately boring. Bake a template that has the JDK, Maven or Gradle, Node and whatever else your builds need already installed. Snapshot it once, warm. When a build is queued, restore a guest from that snapshot, hand it the Jenkins connection details, and let the inbound agent connect back to the controller and claim the build. The build runs — cloning, compiling, testing, publishing — entirely inside that guest. When it finishes, delete the node on the controller and destroy the VM. Kernel, filesystem, workspace, environment, credentials, forked processes and anything a build step tried to leave behind all cease to exist in the same operation.

On PandaStack the create path is a snapshot restore rather than a boot: roughly 179ms at p50 and 203ms at p99, with the restore-and-resume step itself around 49ms. The genuine cold boot — about 3 seconds — happens once when the template is baked and is then amortised across every build that restores from it. If you want to branch a warmed, prepared guest rather than start from the template, a same-host fork is 400-750ms (1.2-3.5s cross-host). Each guest gets its own network namespace from a pool of 16,384 pre-allocated /30 subnets per host, so "this build's network" is a real segment rather than a shared bridge. Those numbers matter for one reason only: they are what make per-build machines affordable enough that you never need a warm pool, and therefore never trade freshness back.

The Jenkinsfile barely changes. It asks for a label, and the label now means "a disposable machine with this toolchain" instead of "agent-04, and please don't reboot it".

pipeline {
  // Not `label 'agent-04'`. The label names a TEMPLATE: a snapshot with
  // JDK 21, Maven and Node baked in. Every build gets its own restore.
  agent { label 'microvm-jdk21' }

  options {
    // Belt and braces: Jenkins gives up here, and the VM has its own TTL
    // so a hung build reaps its own machine even if the controller doesn't.
    timeout(time: 20, unit: 'MINUTES')
    disableConcurrentBuilds()
  }

  environment {
    // Credentials binding puts the REAL value in this build's environment
    // for the duration. On a persistent agent that value shares a machine
    // with everyone else's builds. Here it shares a machine with nothing.
    NEXUS = credentials('nexus-deploy-ro')
  }

  stages {
    stage('Build') {
      steps {
        // No cache-warming ritual and no `mvn -U` superstition: ~/.m2 came
        // from the snapshot, read-mostly, and dies with the guest.
        sh 'mvn -B -Dmaven.repo.local=$HOME/.m2/repository verify'
      }
    }
    stage('Publish') {
      when { branch 'main' }
      steps {
        sh 'mvn -B deploy -DskipTests'
      }
    }
  }

  post {
    always {
      junit 'target/surefire-reports/*.xml'
      archiveArtifacts artifacts: 'target/*.jar', allowEmptyArchive: true
      // Deliberately NOT deleteDir() as a security control. The control is
      // that this entire machine stops existing a moment from now.
    }
  }
}

Wiring the agent lifecycle

Inside the guest, nothing exotic happens: you run the same inbound agent Jenkins has always shipped. The controller exposes agent.jar, the guest downloads it and connects outbound over WebSocket — which conveniently means the controller never needs a route into your build network, only the other way around. The per-agent secret is a connection secret for this one node, not a credential from the credentials store, and it becomes worthless the moment the node is deleted.

#!/usr/bin/env bash
# Runs INSIDE the guest. The provisioner injected the node name and secret.
set -euo pipefail

CONTROLLER="https://jenkins.internal.acme.dev"
NODE_NAME="microvm-${SANDBOX_ID}"

curl -fsSLO "${CONTROLLER}/jnlpJars/agent.jar"

# -webSocket: the agent dials OUT to the controller, so the controller needs
# no inbound route into the build network. The secret is written to a file by
# the provisioner, not passed as an argument, so it stays out of the process
# table where a build step could read it.
exec java -jar agent.jar \
  -url "${CONTROLLER}/" \
  -secret "$(cat /run/jenkins/agent-secret)" \
  -name "${NODE_NAME}" \
  -webSocket \
  -workDir /home/jenkins/agent

# There is no supervisor and no restart policy. This guest exists to run one
# build. When the agent process exits, the machine is inert and gets deleted.

The piece you own is the provisioner: the thing that notices a build wants a `microvm-jdk21` node, creates the VM, registers the node, and reaps both afterwards. You can write it as a Jenkins cloud implementation if you want it inside the controller, but for a first cut an external script driven by the queue API is easier to reason about and much easier to roll back. Either way the shape is the same five steps, and the important one is the reaping — delete the node on the controller before you destroy the VM, so Jenkins never schedules a second build onto a machine that is on its way out.

import os
import requests
from pandastack import Sandbox

JENKINS = "https://jenkins.internal.acme.dev"
AUTH = (os.environ["JENKINS_USER"], os.environ["JENKINS_API_TOKEN"])


def run_one_build(label: str = "microvm-jdk21") -> None:
    # 1. Fresh guest from the baked snapshot. JDK, Maven, Node and a warmed
    #    ~/.m2 are already in the image -- nothing installs on the hot path.
    sbx = Sandbox.create(
        template="base",
        ttl_seconds=2400,  # backstop: the VM reaps itself if we crash
        metadata={"kind": "jenkins-agent", "label": label},
    )
    node = f"microvm-{sbx.id}"

    try:
        # 2. Register the node on the controller and read back its JNLP
        #    secret. One node, one build; we delete it in `finally`.
        secret = create_jenkins_node(node, label)

        # 3. Deliver the secret as a FILE, so it never appears in argv where
        #    a build step running as the agent user could read it.
        sbx.filesystem.write("/run/jenkins/agent-secret", secret)
        sbx.filesystem.write("/usr/local/bin/connect-agent.sh", CONNECT_SH)
        sbx.exec("chmod 0400 /run/jenkins/agent-secret")
        sbx.exec("chmod +x /usr/local/bin/connect-agent.sh")

        # 4. Launch the inbound agent detached. It dials the controller,
        #    claims the queued build, and runs it inside this guest.
        sbx.exec(
            f"SANDBOX_ID={sbx.id} setsid /usr/local/bin/connect-agent.sh "
            "> /var/log/jenkins-agent.log 2>&1 &"
        )
        wait_for_build_to_finish(node)  # poll the node/queue API

    finally:
        # 5. Delete the node FIRST so Jenkins stops scheduling onto it, then
        #    destroy the machine. Workspace, caches, credentials in the
        #    environment, stray daemons: all gone in one operation.
        requests.post(f"{JENKINS}/computer/{node}/doDelete", auth=AUTH)
        sbx.kill()

Note the two independent stop conditions. `ttl_seconds` on create means a guest your provisioner forgot about deletes itself; the pipeline `timeout` means Jenkins gives up on a build that hangs. Neither depends on the other being correct, which is the property you want in the component whose job is to guarantee nothing outlives its build.

Baking the toolchain, and the honest tradeoff in warm caches

A per-build machine only pays off if nothing installs on the hot path. Every `apt-get install` or `sdk install java` at build time is latency you added back, plus a network dependency that will one day be the reason your CI is red. So the template becomes the artifact you actually maintain: JDK versions, the build tool, Node, any native toolchain your tests need, the CA bundle, the agent user. Build it from a Dockerfile in version control, bake it into a snapshot, and treat a rebuild as a reviewed change with a version number. That single move converts "what is installed on agent-04?" from an archaeology question into a diff.

Dependency caches are the more interesting decision. A cold ~/.m2 on a large Java monorepo is genuinely painful, and the fix is to warm the cache before you snapshot: run a resolve pass in the template build so the snapshot contains a populated ~/.m2, ~/.gradle and npm cache. Because restore is copy-on-write, every build inherits a read-mostly view of that cache and only dirties its own pages, which vanish with the guest. One build cannot poison the next, because no build writes back into the baked layer.

• Bake the toolchain; never install it at build time. Latency and reliability both improve, and the contents become reviewable.
• Warm the caches in the template build, not in the pipeline. A read-mostly baked cache gives you the speed of a shared cache without the cross-job write path that makes shared caches dangerous.
• Treat the template like a lockfile. Rebuild it when dependencies change, version it, and let a job pin an older template if it needs to.
• Keep the guest offline where you can. A build that resolves entirely from the baked cache and an internal artifact mirror needs no general internet access at all — and the per-guest network namespace is where you enforce that.

Credentials, and the workspaces that genuinely need to persist

Credential scoping is where the microVM model earns its keep beyond hygiene. Because the machine is per-build, you hand it exactly the secrets that build needs and nothing else: a read-only artifact-repository credential for a PR build, a deploy credential only in the branch-gated publish stage, a short-lived token minted at provision time rather than a long-lived one from the store. Design toward one rule — a compromised build must not be able to mint more access than it was given. That means no cloud instance role the guest can reach, no metadata endpoint reachable from its network namespace, and no agent secret with any meaning beyond "this one node, right now".

Some jobs genuinely need a persistent workspace, and pretending otherwise is how migrations stall. A long incremental C++ build, a Gradle build cache that meaningfully changes wall-clock time, a job maintaining a large checked-out monorepo — these are real. Handle them explicitly rather than by accident: attach a durable volume for that job and be deliberate about who else touches it. Mount it read-only for anything untrusted and let one trusted job be the sole writer, or give each pipeline its own volume so persistence is scoped to a pipeline rather than to a machine. The failure mode you are avoiding is not persistence — it is persistence everybody shares because it happened to live on the same node.

Static agent vs Docker vs Kubernetes vs EC2 vs microVM

Side by side, with the same caveat as before: everything about the Jenkins plugins, EC2 and Kubernetes here is qualitative and configuration-dependent, so verify the specifics against their own documentation. Only the PandaStack numbers are measured.

• Static long-lived agent — Freshness: none; state accumulates for years and shared ~/.m2, ~/.gradle and Docker layers cross-contaminate jobs. Isolation: none between jobs on the node; every build runs as the same user with access to the others' leftovers. Start latency: zero, which is the entire reason people keep them. Best for: nothing you would design today, and a great deal of what is running right now.
• Docker plugin (container per build) — Freshness: good; a clean filesystem per build from an image you rebuild from a Dockerfile. Isolation: namespaces and cgroups over a shared host kernel; a container escape reaches the host and its neighbours. Start latency: fast — image pull aside, this is a container start. Best for: trusted internal builds where hygiene is the goal and the threat model is mistakes, not adversaries.
• Kubernetes plugin (pod per build) — Freshness: good, as Docker, plus real scheduling and autoscaling. Isolation: still a shared node kernel, and the Docker-in-Docker workaround usually means a privileged container or a mounted host socket. Start latency: fast, subject to pod scheduling and image pull on a cold node. Best for: teams already operating Kubernetes who want elastic capacity more than a hard boundary.
• EC2 plugin (instance per build) — Freshness: excellent when instances are truly single-use — a whole clean machine, terminated after. Isolation: strong; a full VM with its own kernel. Start latency: instance provision plus boot, which is why teams set long idle timeouts and reuse instances, quietly giving the freshness back. Best for: workloads where a slow provision is acceptable and per-instance cost is not the binding constraint.
• Firecracker microVM (guest per build) — Freshness: total; a fresh guest restored from a baked snapshot, destroyed after one build, with a read-mostly warmed cache that no build can write back to. Isolation: hardware-enforced — its own guest kernel under KVM, its own memory, its own network namespace. Start latency: about 179ms p50 / 203ms p99 on PandaStack because create is a snapshot restore, not a boot; the ~3s cold boot is paid once at bake time. Best for: untrusted or externally-contributed builds, compliance boundaries, and the jobs that keep corrupting your agents.

The honest migration cost

This is an integration, not a plugin install. You will write and own a provisioner, decide how nodes get registered and reaped, build a template pipeline, work out how logs and artifacts get off a machine that is about to be deleted, and handle the long tail of jobs that quietly depended on something being present on agent-04. Expect to find at least one pipeline that only passes because of a file nobody knew was there. That discovery is a benefit, but it will not feel like one at the time.

Which is why the sensible move for most shops is not a fleet-wide migration. It is routing a specific, well-chosen slice of jobs to microVM agents first and leaving everything else exactly where it is:

1. Anything that builds untrusted or externally-contributed code — forked pull requests, community contributions, vendor-supplied build steps. This is the slice where the isolation argument is not a preference, and where today's answer is usually "we hope nobody notices those run on the same agents".
2. The jobs that keep corrupting the agent. Every fleet has two or three: the one that leaves daemons running, the one that fills the disk with Docker layers, the one after which that node is mysteriously flaky. Give those a disposable machine and the recurring incident simply stops.
3. Anything under a compliance or contractual boundary that wants separation stronger than 'different directory, same kernel'. A per-build VM is far easier to describe in an audit than a shared node with a cleanup script.
4. Then, only if the first three go well, the long tail — starting with the pipelines whose Jenkinsfiles are already hermetic, because those will migrate by changing a label.

The end state is worth being precise about, because it is modest and that is the point. Same controller, same Jenkinsfiles, same credentials store, same people. The only thing that changed is that a build no longer runs on a machine with a history. It runs on one restored a fraction of a second ago from an image you can read, and deleted before anyone has a chance to log into it and fix something. Nobody says "it works on agent-04" any more, because there is no agent-04 — there is a template, in git, with a version number, and if the build works there it works everywhere.