Buildkite Agents on MicroVMs: One Job, One Machine

Most CI vendors sell you a control plane and the compute underneath it as one indivisible product. Buildkite does something unusual: it hosts the coordination, the pipeline UI, the scheduling and the log storage, and then hands you a small open-source Go binary and says the rest is yours. Your agents run on your machines, in your network, with your kernel, holding your secrets. Nothing about your source code has to cross into someone else's execution environment.

That split is genuinely good architecture, and it is why regulated shops and teams with fleets spread across a data centre and two clouds keep landing on Buildkite. But it relocates a question that other vendors answer for you, and it relocates it onto your plate as the single most consequential decision in the whole setup: what, exactly, does an agent run on? Buildkite is deliberately unopinionated here. It will happily run your build as a plain process on a long-lived box, and the fastest path to a green pipeline does exactly that. My argument in this post is that the right answer for most non-trivial fleets is a fresh Firecracker microVM per job — and that the reason has less to do with security theatre than with the fact that a long-lived Buildkite agent is a machine whose state is a function of every build that ever landed on it.

What the Buildkite agent actually does

It is worth being precise about the mechanics, because they shape every design choice downstream. The agent is not a daemon waiting to be told what to do over an open port. It dials out to Buildkite's agent API over HTTPS, registers itself with an agent token, and then polls for work — the connection is always outbound, initiated from your side. There is no inbound port to open, no NAT hole to punch, no VPN between Buildkite and your build network. An agent behind three layers of firewall in a private subnet works exactly like one on a public host, which is a large part of why the model fits air-gapped-ish environments so well.

When the agent is handed a job, it runs a bootstrap sequence and fires a series of hooks around it. The names are worth memorising because they are your only real injection points: an environment hook, then checkout hooks around the git clone, then pre-command / command / post-command around the actual step, then artifact hooks, and finally a pre-exit hook that runs whether the job passed or failed. Hooks can live at the agent level (a directory on the machine, configured via the agent's hooks path) or in the repository under a .buildkite/hooks directory. Check the current hook list against Buildkite's own documentation before you build on it — the set has grown over time.

During the job, the same binary is the build's interface back to the control plane. The step shells out to buildkite-agent artifact upload to push files, buildkite-agent annotate to render markdown onto the build page, buildkite-agent meta-data set and get to pass values between steps, and buildkite-agent pipeline upload to generate steps dynamically from inside a running build. That last one is the feature people fall in love with — a pipeline that computes its own shape — and it is also the one that should make you think carefully about what a build step is allowed to do.

• Outbound only. The agent polls Buildkite; Buildkite never connects to you. No inbound ports, no ingress rules, no exposure of the build network.
• Hooks are the extension surface. Agent-level hooks run for every job on that machine; repository hooks ship with the code being built. Both are just executables the agent runs.
• The build talks back through the same binary. Artifacts, annotations, metadata and dynamic pipeline uploads all go through buildkite-agent subcommands invoked by your step.
• The agent is open source and self-contained. A single Go binary plus a config file, which is exactly what makes baking it into a VM image trivial.
• Where it runs is entirely your call. That is the product's premise, and the reason this post exists.

Two agent lifecycles, and only one of them is defensible

There are really only two shapes an agent fleet can take, and Buildkite supports both without much preference.

The first is the long-lived agent: install the binary on a box, run it as a systemd service, let it pick up jobs forever. This is the default in every getting-started guide, it is trivially easy, and it is what most fleets are actually running right now. The second is the one-job ephemeral agent: start an agent, let it take exactly one job, and have it disconnect and exit. Buildkite supports this directly — disconnect-after-job is a config key (and a corresponding flag) that tells the agent to stop after it finishes a single job. There is also an idle-timeout setting that stops an agent that has waited a given number of seconds without being offered work, which is the piece that keeps an over-eager autoscaler from leaving machines spinning. And if your supervisor already knows the specific job UUID it wants to service, the agent can be pointed at exactly that one job rather than accepting whatever is next in the queue. Confirm the exact flag spellings against the agent's own documentation; they are stable but the surrounding options move.

The distinction people miss is that a one-job agent process is not the same thing as a one-job machine. If you run a supervisor loop that starts a fresh agent process on the same host after each job exits, you have made the agent ephemeral and left the machine exactly as persistent as it was before. Everything that matters — the filesystem, the caches, the kernel, the leftover processes — is unchanged. The lifecycle you want is one job, one machine, and disconnect-after-job is how you make the machine's lifetime and the job's lifetime the same thing.

An ephemeral agent process on a persistent host gives you a clean process table and a dirty everything else. The unit of disposal has to be the machine.

What accumulates on a long-lived agent

Take an honest inventory of a Buildkite agent box that has been running for a year. None of this is Buildkite's fault; it is what happens to any machine that runs other people's build scripts repeatedly.

• The build path. The agent checks out each pipeline into a directory under its build path and reuses it. That reuse is a deliberate speed feature — it is why the second build of a repo does not re-clone it — and it is also a working tree that every build of that pipeline has written to. The git-clean flags the agent applies between builds are hygiene, not containment.
• The git mirror. If you enabled git mirrors (and on a large monorepo you should have, because it is a big win), there is now a shared object store on that disk that every pipeline cloning that repo reads from. Shared, mutable, on the machine untrusted builds run on.
• Language and package caches. One ~/.npm, one ~/.cargo, one ~/.m2, one pip cache, shared by every pipeline scheduled to that agent. A build that writes into a cache is influencing what a later, unrelated build resolves and executes.
• The Docker state. If builds touch Docker, the machine's image store, build cache and dangling volumes are a landfill with a disk-full incident in its future, and every container that runs there shares the host kernel with every other one.
• Agent-level hooks and plugins. The hooks directory and the plugins directory are code that runs for every job on that machine. Anything that can write there has achieved persistent code execution across all future builds on the box.
• Left-behind processes and ports. A test that forked a database and never reaped it. A dev server holding port 3000. Precisely the class of thing that makes one pipeline mysteriously flaky on one agent.

Now put a fork pull request onto that machine. Buildkite lets you gate whether builds run for pull requests from forks, and if you accept external contributions at all, you eventually turn that on for something. At that instant, the population of people who can execute code on your agent becomes the population of people with a GitHub account. Their code runs as the agent user, on a machine holding a shared git mirror, shared caches, an agent-level hooks directory and — the part that deserves its own section — the agent token.

The agent token is a fleet credential, not a job credential

This is the single most important security difference between Buildkite's model and GitHub Actions', and it is the one I see teams get wrong most often. A GitHub Actions runner registers with a short-lived registration token that is minted per registration and expires quickly. A Buildkite agent token is not that. It is a long-lived credential that authorises a machine to join your fleet and receive jobs, and it is typically written into the agent's config file or passed as an environment variable and left there for the lifetime of the host.

So ask the uncomfortable question: on a long-lived agent, can a build step read the token? If the config file is readable by the agent user — and the agent user is the user your build step runs as — then yes, trivially. A stolen agent token does not give an attacker your source directly; it gives them the ability to register a machine of their own into your fleet and start receiving your jobs, including jobs from pipelines they were never allowed to touch, along with whatever environment those jobs carry. Buildkite's clusters feature exists in part to scope this blast radius — a token belongs to a cluster and can only receive that cluster's work — and you should use it, but scoping a credential is not the same as putting it out of reach.

On a one-job microVM the exposure window shrinks to the life of that job, and you get an additional structural option: hand the token to the guest at provision time as a file the agent reads and then scrub, or better, let the supervisor start the agent process before the job's code exists on the machine at all. The token still lives inside the guest for a moment, but the guest is deleted minutes later and its next job is on a different machine. That is a far smaller promise to keep than the one you make on a host that runs strangers' code for a year.

The Docker-in-Docker problem, in its Buildkite clothes

Because Buildkite's agent is unopinionated about execution, the community answer to isolation has historically been the Docker plugins: run the step inside a container instead of directly on the host. That is a real improvement in hygiene — a clean filesystem per step, a toolchain that comes from an image you can rebuild — and for trusted internal builds it is often enough. But it is a namespace boundary over a shared host kernel, and the shared kernel is precisely the thing you were trying to get away from when you started worrying about fork PRs.

It gets sharper the moment a pipeline needs to build a container image, which on a modern fleet is most of them. Building images inside a containerised step means either mounting the host's Docker socket into the step, or running a privileged nested daemon. Mounting the socket is functionally equivalent to giving the step root on the host: anything that can talk to that socket can start a container with the host filesystem bind-mounted, and now the untrusted build owns the machine, the agent token and every other build on it. The privileged nested daemon drops most of the container isolation you were relying on, which makes it an odd defence to build on. Rootless builders help and are worth adopting, but they come with their own storage-driver and caching sharp edges.

A microVM sidesteps the dilemma rather than mitigating it. Inside a guest with its own kernel, you can just run a normal, unprivileged, ordinary Docker daemon — because the isolation boundary is not the container, it is the virtual machine around it. There is no nesting problem to solve, because nothing is nested.

One job, one microVM

A Firecracker microVM is a real virtual machine — its own guest kernel, its own memory, its own virtual devices, isolated by the CPU's virtualization extensions under KVM — with a device model small enough that starting one is measured against container start times rather than EC2 boot times. The reason a per-job VM is affordable is that creating one is a snapshot restore, not a boot. You bake a template once with the agent binary, your toolchains and your hooks already installed, snapshot it warm, and every subsequent create restores that frozen machine and resumes it.

On PandaStack that create path runs at roughly 179ms at p50 and 203ms at p99. The genuine cold boot — about three seconds — happens once when the template is baked and is amortised across every job that restores from it. Memory is copy-on-write and the rootfs is a reflink clone, so the hundredth guest does not copy gigabytes to exist. Each guest gets its own network namespace from a pool of pre-allocated /30 subnets, so a job's network is a real segment where you can enforce egress policy rather than a shared bridge. And a TTL on create means a guest your supervisor forgot about deletes itself.

The numbers matter for exactly one reason, and it is not bragging rights. Every team that adopts per-build VMs on conventional instances eventually discovers that provisioning is slow enough to hurt, and quietly starts reusing instances to hide the latency. At that point they have paid for freshness and given it back. Sub-second creates are what let you never make that trade.

The agent config inside the guest

Here is what runs inside the microVM. The shape is deliberately dull: configure an agent that takes one job and exits, with tags that make it targetable and a build path that nobody will ever reuse.

#!/usr/bin/env bash
# Runs INSIDE the guest. The supervisor wrote the token file before calling this.
set -euo pipefail

# The agent binary and toolchain came from the baked snapshot -- nothing is
# installed on the hot path. Only the token and the identity are injected.
cat > /etc/buildkite-agent/buildkite-agent.cfg <<'CFG'
# NOTE: no `token=` line. The agent reads BUILDKITE_AGENT_TOKEN from the
# environment, which we source from a 0400 file below -- so the fleet join
# credential is never on a config file the build user can read, and never in
# argv where a build step could find it in the process table.

name="fc-%hostname"

# Tags are how a pipeline targets this machine. `queue` is the special one --
# it selects which queue this agent pulls from. The rest are free-form and
# match against a step's `agents:` block.
tags="queue=fc-untrusted,isolation=microvm,docker=true,os=linux,arch=arm64"

# THE load-bearing setting. Take exactly one job, then disconnect and exit.
# The supervisor deletes the machine when this process returns.
disconnect-after-job=true

# If Buildkite has nothing for us, do not sit here burning a VM. Give up and
# let the machine be reclaimed; the autoscaler will make another when needed.
disconnect-after-idle-timeout=60

build-path=/var/lib/buildkite-agent/builds
hooks-path=/etc/buildkite-agent/hooks
plugins-path=/etc/buildkite-agent/plugins
CFG

chown buildkite-agent:buildkite-agent /etc/buildkite-agent/buildkite-agent.cfg
chmod 0400 /etc/buildkite-agent/buildkite-agent.cfg

# The token file is 0400 root-owned; read it here, then scrub it so it is not
# sitting on a disk that is about to run someone else's build script.
BUILDKITE_AGENT_TOKEN="$(cat /run/buildkite/token)"
export BUILDKITE_AGENT_TOKEN
rm -f /run/buildkite/token

# No systemd unit, no Restart=always, no supervisor loop. This guest exists to
# service one job. When the agent disconnects, the machine is inert.
exec sudo -u buildkite-agent --preserve-env=BUILDKITE_AGENT_TOKEN \
  /usr/bin/buildkite-agent start

# Verify current flag and config-key spellings against the agent's own docs --
# `buildkite-agent start --help` is the authority, not this blog post.

Two things to notice. There is no restart policy, because a restart would be the beginning of a second job on a machine that has already run someone's code. And the token is in a file owned by the agent user with mode 0400, not in the step's environment — the build should never be able to read the credential that let its machine join the fleet.

The supervisor that spawns and reaps machines

The piece you own is the supervisor: something that watches queue depth, creates a guest per pending job, and destroys it afterwards. Buildkite exposes an agent metrics endpoint that reports scheduled and running job counts per queue, and there is a small official metrics collector built for exactly this — that is your scaling signal. The loop below is deliberately simple, and simple is the right first cut, because the failure mode you care about is not inefficiency, it is a machine that survives its job.

import os
import time

import requests
from pandastack import Sandbox

QUEUE = "fc-untrusted"
AGENT_TOKEN = os.environ["BUILDKITE_AGENT_TOKEN"]  # fleet join credential
MAX_IN_FLIGHT = 40


def scheduled_jobs(queue: str) -> int:
    """Queue depth from Buildkite's agent metrics endpoint. This is the
    scaling signal: how many jobs are waiting for a machine right now."""
    r = requests.get(
        "https://agent.buildkite.com/v3/metrics",
        headers={"Authorization": f"Token {AGENT_TOKEN}"},
        timeout=10,
    )
    r.raise_for_status()
    return r.json()["jobs"]["queues"].get(queue, {}).get("scheduled", 0)


def spawn_one_job_machine() -> None:
    # 1. Restore a fresh guest from the baked snapshot: agent binary, hooks,
    #    toolchains and warmed caches already inside. ~179ms p50, because this
    #    is a snapshot restore rather than a boot.
    sbx = Sandbox.create(
        template="buildkite-agent",
        ttl_seconds=3600,  # backstop: the guest reaps itself if we crash
        metadata={"kind": "buildkite-agent", "queue": QUEUE},
    )
    try:
        # 2. Deliver the agent token as a file, never as argv. A build step
        #    that can read the fleet's join credential can join the fleet.
        sbx.filesystem.write("/run/buildkite/token", AGENT_TOKEN)
        sbx.exec("chmod 0400 /run/buildkite/token")

        # 3. Start the agent. disconnect-after-job means it claims exactly one
        #    job and exits; the untrusted checkout, every hook and every build
        #    command run inside THIS guest and nowhere else.
        result = sbx.exec(
            "/usr/local/bin/start-one-job-agent.sh",  # reads + scrubs the file
            timeout_seconds=3300,
        )
        print(result.stdout[-2000:])  # tail for our own observability

    finally:
        # 4. Destroy the machine. Build path, git mirror, package caches,
        #    Docker state, hooks directory, token file, forked daemons and
        #    anything the job tried to plant all cease to exist together.
        #    There is no cleanup script to get wrong.
        sbx.kill()


def supervise() -> None:
    in_flight = 0
    while True:
        want = min(scheduled_jobs(QUEUE), MAX_IN_FLIGHT - in_flight)
        for _ in range(max(want, 0)):
            spawn_one_job_machine()  # run these on a thread/task pool
        time.sleep(5)

Note the two independent stop conditions, which is the property you want in the component whose job is to guarantee nothing outlives its build. The exec timeout is a circuit breaker for a job that hangs or deliberately stalls. The create-time TTL is a backstop for a supervisor that crashes between spawning a guest and reaping it. Neither depends on the other being correct.

Queues, tags, and routing only what needs it

You do not have to move the whole fleet, and you should not try. Buildkite's targeting model makes a partial migration genuinely easy, which is unusual and worth exploiting. An agent advertises tags; a step selects agents with an agents block matching those tags. The queue tag is special — it partitions work, and an agent only receives jobs from the queue it is in — while every other tag is free-form matching.

So the migration is a routing change, not a rewrite. Point the pipelines that build fork PRs, or the ones that keep corrupting an agent, at a queue served by microVMs. Leave everything else on the fleet you have. A step that says it wants queue: fc-untrusted and isolation: microvm gets a disposable machine; a step that says nothing keeps getting whatever it got yesterday.

• Route untrusted work first. Anything that builds fork pull requests, community contributions or vendor-supplied build steps. This is where the isolation argument is not a preference.
• Route the agent-corrupting pipelines second. Every fleet has two or three: the one that leaves daemons running, the one that fills the disk with image layers, the one after which that box is mysteriously flaky. Give those disposable machines and the recurring incident just stops.
• Use tags for capability, not identity. isolation=microvm and docker=true describe what the machine can do. Tagging by hostname re-creates the snowflake you are trying to delete.
• Give each queue its own agent token and cluster. Then a token leaked from the untrusted queue cannot receive jobs from the deploy queue.
• Keep a separate, small, trusted queue for deploys. Deploy steps want long-lived credentials and should never share a machine class with fork PRs.

Autoscaling against queue depth

Because agents pull work rather than being pushed to, autoscaling a Buildkite fleet is refreshingly direct: read the number of scheduled and running jobs per queue and make that many machines exist. Buildkite publishes an agent metrics endpoint for precisely this, ships a small metrics collector that can forward those counts into a monitoring system, and maintains an elastic stack for AWS that wires the whole loop together with an autoscaling group. If you are on AWS and happy with instances, that stack is a perfectly reasonable place to start and I would not talk anyone out of it.

The microVM version changes the arithmetic in one important way. An instance-backed autoscaler is fighting provisioning latency: a scheduled job waits for an instance to boot, so you keep a warm buffer and accept some idle spend, and you set scale-in cooldowns long enough that you are not paying to boot the same machine repeatedly. When a machine appears in under two hundred milliseconds, the buffer stops earning its keep. You scale the host capacity — the boxes that run the microVMs — on a slow, sane loop, and you create and destroy the per-job guests on demand with no pool at all. Idle cost approaches zero not because you tuned a scale-in policy well, but because there is nothing idle to pay for.

Keeping the caches, losing the contamination

The obvious objection to a disposable machine is that you throw away the git mirror and the package caches that make your builds fast, and re-cloning a monorepo per job is not a trade anyone will accept. The answer is to move the cache from the machine into the image.

Bake it. Run the clone and the dependency resolve during the template build, then snapshot. Because restore is copy-on-write, every job inherits a read-mostly view of that warmed git mirror and package cache and only dirties its own pages, which vanish with the guest. You get the speed of a shared cache without the write path that lets one job poison the next — a fork PR can scribble on its own copy-on-write pages all it likes and cannot write back into the baked layer that the next job restores from. Refresh the snapshot when your lockfile or default branch moves; treat the template version as a cache key.

If a pipeline genuinely needs persistence across jobs — a large incremental build cache, say — attach a durable volume deliberately and scope it to that pipeline, with untrusted work mounting it read-only and one trusted job as the sole writer. The failure mode you are avoiding was never persistence. It was persistence that everybody shares because it happened to live on the same box.

Long-lived agent vs containerised step vs microVM per job

Side by side, with the usual caveat: everything about Buildkite's own features and the container plugins here is qualitative and configuration-dependent, so verify specifics against their documentation. Only the PandaStack numbers are measured.

• Long-lived agent, steps on the host — Freshness: none; build path, git mirror, package caches and Docker state accumulate indefinitely. Isolation: none between jobs; every build runs as the agent user with access to the others' leftovers and, usually, the agent token. Start latency: zero, which is the entire reason people keep them. Best for: trusted internal pipelines on a fleet you fully control, and a great deal of what is running right now.
• Long-lived agent, containerised step — Freshness: good; a clean filesystem per step from an image you rebuild from a Dockerfile. Isolation: namespaces and cgroups over a shared host kernel; a container escape reaches the host, its neighbours and the agent token. Image builds force you into a mounted socket or a privileged daemon, both of which undo the boundary. Start latency: fast, image pull aside. Best for: trusted internal builds where hygiene is the goal and the threat model is mistakes rather than adversaries.
• Instance per job (elastic stack style) — Freshness: excellent when instances are genuinely single-use — a whole clean machine, terminated after. Isolation: strong; a full VM with its own kernel. Start latency: instance provision plus boot, which is why teams keep a warm buffer and reuse instances, quietly giving the freshness back. Best for: workloads where a slow provision is acceptable and per-instance cost is not the binding constraint.
• MicroVM per job — Freshness: total; a fresh guest restored from a baked snapshot, destroyed after one job, with a read-mostly warmed cache no job can write back into. Isolation: hardware-enforced — its own guest kernel under KVM, its own memory, its own network namespace for egress policy. Start latency: about 179ms p50 / 203ms p99 on PandaStack, because create is a snapshot restore; the roughly 3s cold boot is paid once at bake time. Best for: fork PRs, third-party code, compliance boundaries, image builds without a privileged daemon, and the pipelines that keep corrupting your agents.

The honest cost

This is an integration you own, not a plugin you install. You will write and maintain the supervisor, decide how the agent token reaches a guest, build a template pipeline and keep it current, work out how logs get off a machine that is about to be deleted, and discover the long tail of pipelines that only pass because of something that happened to be present on an agent. That last discovery is a benefit; it will not feel like one on the day. Buildkite also now offers hosted agents of its own, which is a legitimate answer if what you want is to stop running compute entirely — check their current details rather than mine.

So do not migrate the fleet. Stand up one queue, point the pipeline that builds fork PRs at it, and leave everything else where it is. The end state is deliberately modest, and that is the point: same Buildkite organisation, same pipeline definitions, same hooks, same people. The only thing that changed is that a job no longer runs on a machine with a history. It runs on one restored a fraction of a second ago from an image you can read, and deleted before anyone has the chance to log into it and fix something by hand.