CircleCI Self-Hosted Runners on MicroVMs

There is a small irony at the centre of every CircleCI self-hosting project. On CircleCI's own cloud, if you pick the machine executor, you get a dedicated virtual machine that is created for your job and destroyed afterwards. Nothing survives. That is the default, you paid for it, and most teams never think about it because it simply works. Then a requirement arrives — a GPU, a licensed toolchain, an on-prem database, a compliance reviewer who does not want source code leaving the network — and you stand up a self-hosted runner. And in the very same move, without anyone writing it down as a decision, you swap a fresh machine per job for a long-lived box that accumulates the residue of every build it has ever run.

Nobody chooses that. It is just what the getting-started path produces: install the runner, run it as a service, watch jobs land. The rest of this post is about why that default is a worse security posture than the cloud you moved off, and how to get back to one clean machine per job using Firecracker microVMs — cheaply enough that you never feel tempted to reuse a host to hide the latency.

I'm Ajay; I build PandaStack, a Firecracker microVM platform where every sandbox create is a snapshot restore rather than a boot. The CircleCI-specific details here are from their documented behaviour and are worth re-checking against their current docs, because the runner product has been through more than one generation. The latency numbers are mine and measured.

Two flavours of self-hosted runner, and why one fits

CircleCI ships two quite different self-hosted runner products, and choosing between them is the first real fork in the road. They are not tiers of the same thing; they have different execution models and different failure modes.

The container runner runs your jobs as pods in a Kubernetes cluster you operate. You install it with a Helm chart, it watches for tasks on your resource classes, and for each task it creates a pod from the image your job declares. It is the right answer if you already run Kubernetes, want per-job image selection, and are content with the isolation a container gives you. Its ceiling is the ceiling of every container CI system: the job runs on a shared host kernel, and if a step needs to build a container image you are back in the socket-mounting, privileged-daemon conversation that has eaten a decade of CI engineering.

The machine runner runs jobs directly on a machine — a VM, a bare-metal box, a Mac mini in a rack. The current generation is machine runner 3.0, which replaced the older launch-agent; if you find a config file at a path with 'launch-agent' in it, you are looking at the previous generation and its keys differ. The machine runner is deliberately dumb about isolation: it does not create anything to run your job in, it just runs the job where it is. Which sounds like the weaker option, and is in fact the reason it maps perfectly onto what we want. If the runner does not manage the isolation boundary, then the machine itself can be the isolation boundary — and if the machine's lifetime equals the job's lifetime, you have a clean VM per job with no cleverness required.

• Container runner — isolation unit is a pod on a shared kernel; per-job image is declared in the config; you operate a Kubernetes cluster; image builds need a privileged or rootless workaround.
• Machine runner — isolation unit is whatever machine you point it at; the runner adds no boundary of its own; no cluster to operate; image builds are just image builds, because the guest has its own kernel.
• The consequence — with container runner, the vendor owns the boundary and you inherit its limits. With machine runner, you own the boundary, which is worse if you do nothing and much better if you make the machine disposable.
• One task at a time. A machine runner instance services a single task and then looks for the next one, so concurrency comes from running more runner instances — which, for our purposes, means more VMs.

The claim loop, which is the hook you need

The mechanics matter here more than usual, because the design depends on exactly where you can intervene. A CircleCI runner never listens for inbound connections. It dials out over HTTPS to CircleCI's runner API, authenticates with a runner token bound to a resource class, and polls for available tasks. When a task is available it claims it. Claiming is the moment the job becomes that runner's responsibility; CircleCI hands over a task token, the runner fetches and executes the task agent, and the task agent runs the job's steps, streams logs, uploads artifacts and reports the result.

Three properties fall out of that shape, and all three are load-bearing.

1. Outbound only. There is no port to open, no ingress rule, no tunnel between CircleCI and your build network. A runner inside a private subnet behind three firewalls behaves identically to one on a public host — which is why this model survives contact with security teams that would never approve inbound access.
2. The runner is a small binary plus a config file. That is the whole install. Something that small is trivial to bake into a VM image, which is exactly what we are going to do.
3. The claim is the event. A job does not exist as work-on-your-infrastructure until a runner claims it. So the question 'when do I create the machine?' has a precise answer, and the answer determines whether your fleet is disposable or not.

There are two ways to arrange the claim, and they look similar in a diagram while being completely different in what they promise. The obvious arrangement is to keep a runner process alive on a host, let it claim tasks forever, and clean the working directory between them. The arrangement you want is to make the machine's lifetime the same as the task's: a supervisor decides a machine should exist, that machine starts a runner configured to service exactly one task and exit, and when the runner process returns the supervisor deletes the machine.

Machine runner 3.0 supports the second arrangement directly through its run mode: instead of running continuously, it can take a single task and terminate. That is the setting the whole design hangs on, so confirm the current key name in CircleCI's runner configuration reference rather than trusting a blog post — including this one. There is also an idle timeout, which is the piece that stops an over-eager supervisor from leaving a VM spinning forever waiting for work that already got claimed elsewhere.

An ephemeral runner process on a persistent host gives you a clean process table and a dirty everything else. The unit of disposal has to be the machine.

What this looks like in .circleci/config.yml

The pipeline side is refreshingly small, and that is the point: the disposability is an infrastructure property, not something every pipeline author has to remember to opt into. A resource class is a namespace-qualified name you create once with the CircleCI CLI, and a job targets it by name.

# .circleci/config.yml
#
# One-time setup, outside this file:
#
#   circleci runner resource-class create acme/fc-untrusted \
#       "disposable Firecracker microVM, one job per machine"
#   circleci runner token create acme/fc-untrusted "supervisor"
#
# The token is a LONG-LIVED credential that authorises a machine to join this
# resource class and receive its jobs. It is not a job secret. Treat it the way
# you would treat a fleet join key -- see the secrets section below.

version: 2.1

jobs:
  build-fork-pr:
    # `machine: true` is mandatory for a machine-runner resource class: there is
    # no container for CircleCI to place you in, because the machine IS the
    # boundary. On the cloud this line would name a docker image instead.
    machine: true
    resource_class: acme/fc-untrusted

    steps:
      - checkout

      # Content-addressed restore INTO a fresh rootfs. The cache is data that
      # arrives over the network, not state left behind by the previous job on
      # this box -- because there was no previous job on this box.
      - restore_cache:
          keys:
            - v3-deps-{{ checksum "package-lock.json" }}

      - run:
          name: install
          command: npm ci

      - save_cache:
          key: v3-deps-{{ checksum "package-lock.json" }}
          paths: ["~/.npm"]

      - run:
          name: test
          # No docker-in-docker gymnastics: the guest has its own kernel, so an
          # ordinary unprivileged daemon inside it is just an ordinary daemon.
          command: npm test

workflows:
  pr:
    jobs:
      # Untrusted work goes to the disposable class. Deploys keep whatever
      # trusted, credential-holding executor they already use -- migrating is a
      # routing change, not a rewrite.
      - build-fork-pr:
          context: [ci-readonly]

Reusing a runner host is the actual security bug

People frame per-job VMs as defence in depth, a nice-to-have you get around to. I think that framing is wrong and it is why the work never gets prioritised. Reuse is not a missing mitigation. It is an affirmative decision to let one job's output become the next job's input, across a trust boundary you did not design and cannot see.

Take an inventory of a machine runner box that has been serving a busy resource class for a year. None of this is CircleCI's fault; it is what happens to any machine that runs other people's build scripts on repeat.

• The toolchain. Every `npm i -g`, every `pip install --user`, every `curl | sh` that a build step ran is still there. A poisoned job does not need persistence tricks — it just installs a wrapper earlier on PATH than the real binary and waits for the next tenant's build to call it.
• The dotfiles. A modified ~/.npmrc pointing a scope at an attacker's registry. A ~/.gitconfig with an insteadOf rewrite. A ~/.docker/config.json with a helper. Small files, enormous leverage, and no CI system diffs them between jobs.
• The working directory. The runner has a cleanup option for it and you should absolutely turn that on, but understand what it is: hygiene, applied by the same user the untrusted job ran as, on a machine that job may have already modified. Cleanup is not containment.
• Background processes. A test that forked a database and never reaped it. A dev server holding a port. A cron entry, a systemd user unit, an at-job. These are the ones that make a pipeline mysteriously flaky on exactly one runner, and the reason nobody ever finds it is that nobody is looking at the machine.
• Package and layer caches. One ~/.npm, one ~/.cargo, one ~/.m2, one Docker image store, shared by every job the resource class ever received. A job that writes into a cache is choosing what a later, unrelated job resolves and executes.
• The runner token itself, sitting in a config file on disk on a machine where strangers' code runs as a local user.

Now point that resource class at a pipeline that builds pull requests from forks, which is the whole reason a lot of teams are reading this. The set of people who can execute code on that box becomes the set of people with a GitHub account. Their code runs as the runner user, on a machine holding shared caches, a shared toolchain and the credential that lets a machine join your fleet.

The honest thing to say about a container runner here is that it fixes the filesystem half of that list and none of the kernel half. A pod gets a fresh root filesystem, so the dotfiles and the planted binaries go away, which is a genuine improvement. But a container is a polite suggestion to a kernel that is shared with every other job on that node, and the moment a pipeline needs to build an image you will be mounting a socket or running something privileged, at which point the polite suggestion is withdrawn.

Secrets, contexts, and why OIDC changes the calculation

CircleCI's secret story has two halves and they age very differently on a self-hosted runner. The first half is contexts: named bundles of environment variables, optionally restricted to a security group, injected into the job's environment. Contexts are a good access-control mechanism CircleCI-side, and they do exactly nothing about the fact that the value lands as a plaintext environment variable inside a process on your machine. On a disposable machine that is fine — the value exists for the duration of the job and then the machine that held it stops existing. On a reused host it is one more thing that was, at some point, in the memory and possibly the shell history of a box that later ran somebody else's code.

The second half is OIDC, and it is the one worth building around. CircleCI mints a signed identity token into the job environment, issued by CircleCI for your organisation, with claims that identify the org, the project and the actor. Your cloud provider trusts that issuer and exchanges the token for short-lived credentials scoped to exactly what that project may do. The static AWS key that used to live in a context stops existing.

Put OIDC and a one-job VM together and you get a property that neither gives you alone. The credential is short-lived and narrowly scoped, so its value decays on its own. The machine that held it is deleted at the end of the job, so there is no window in which a later tenant can go looking for it in a file, a process environment, an agent socket or a leftover credential cache. Compare that with the long-lived-credentials-on-a-persistent-runner arrangement, where the interesting question for an attacker is not 'can I exfiltrate this token' but 'what will be sitting on this machine an hour from now?' — and the answer is the next team's deploy job.

1. Prefer OIDC over stored credentials wherever the target supports it, and scope the trust policy by project claim rather than by organisation, so one compromised project is not every project.
2. Keep the runner token out of the job's reach. A file readable only by the process that starts the runner, never an environment variable the step inherits, never an argument on a command line.
3. Give untrusted work its own resource class, its own token, and its own contexts. Fork PRs should not be able to target the resource class your deploys run on.
4. Set a job timeout that is shorter than your patience. A claimed task that hangs is a machine held open, and on a disposable fleet that is the main way you leak money.
5. Verify the OIDC claim shape and availability for the runner flavour you chose against CircleCI's current docs before you write a trust policy against it.

Wiring claim, boot, run, destroy

The pieces are: a template that already contains everything a job needs, a runner config that takes one task and exits, and a supervisor that creates a guest, waits, and deletes it. Here is the guest side.

# /etc/circleci-runner/circleci-runner-config.yaml -- INSIDE the guest.
#
# Everything here came from the baked snapshot except the token, which the
# supervisor writes in as a 0400 root-owned file a moment before starting the
# runner. Nothing is installed on the hot path.

api:
  # NOT the literal token. The systemd unit that starts the runner reads
  # /run/circleci/token, exports it, and scrubs the file, so the join
  # credential is never in a config file the job user can read.
  auth_token: ${CIRCLECI_RUNNER_AUTH_TOKEN}

runner:
  # The sandbox id, so a claimed task is traceable to exactly one machine that
  # exists for exactly as long as that task does.
  name: "fc-${PANDASTACK_SANDBOX_ID}"

  # THE load-bearing setting: service one task, then terminate. The supervisor
  # deletes the machine when this process returns. Confirm the current key and
  # accepted values against CircleCI's runner configuration reference -- this
  # is the one line that must be right.
  mode: single-task

  # If CircleCI has nothing for us, do not burn a VM waiting. Exit and let the
  # supervisor reclaim it; a new one costs a fraction of a second.
  idle_timeout: 1m

  # A hard ceiling on a task that hangs or deliberately stalls. This is the
  # first of two independent stop conditions -- the second is the create-time
  # TTL on the sandbox itself.
  max_run_time: 45m

  working_directory: /var/opt/circleci/workdir

  # Belt and braces. On a disposable machine this is redundant, which is the
  # nicest possible thing to be able to say about a cleanup routine.
  cleanup_working_directory: true

logging:
  file: /var/log/circleci-runner.log

Note that there is no restart policy anywhere in that file or in the unit that starts it. A restart would be the beginning of a second job on a machine that has already run somebody's code, which is precisely the thing we are buying our way out of. When the runner exits, the guest is inert and waiting to be deleted.

Now the supervisor. It watches for demand, creates a guest per task, and reaps it. Keep it boring; the failure mode you care about is not inefficiency, it is a machine that outlives its job.

import os
import concurrent.futures as futures

from pandastack import Sandbox

RESOURCE_CLASS = "acme/fc-untrusted"
RUNNER_TOKEN = os.environ["CIRCLECI_RUNNER_TOKEN"]  # resource-class join key
MAX_IN_FLIGHT = 40


def run_one_task() -> None:
    """Create a machine, let it claim exactly one CircleCI task, destroy it."""

    # 1. Restore a fresh guest from the baked template: runner binary, git,
    #    language toolchains and a warmed package cache are already inside.
    #    ~179ms p50 / ~203ms p99, because this is a snapshot restore, not a
    #    boot. The ttl is a backstop for a supervisor that crashes between
    #    creating a guest and reaping it.
    sbx = Sandbox.create(
        template="ci-runner",
        ttl_seconds=3600,
        metadata={"kind": "circleci-runner", "resource_class": RESOURCE_CLASS},
    )

    try:
        # 2. Deliver the join credential as a file, never as argv. A step that
        #    can read this token can register its own machine into the class.
        sbx.filesystem.write("/run/circleci/token.staged", RUNNER_TOKEN)
        sbx.exec("install -m 0400 -o root -g root /run/circleci/token.staged "
                 "/run/circleci/token && rm -f /run/circleci/token.staged")

        # 3. Start the runner. In single-task mode it polls, claims ONE task,
        #    runs it to completion and exits. The checkout, every step and
        #    anything the job decides to install happen inside THIS guest.
        result = sbx.exec(
            "systemd-run --wait --unit=circleci-runner-oneshot "
            "/usr/local/bin/start-single-task-runner.sh",
            timeout_seconds=2760,  # inside the guest's own max_run_time
        )
        print(result.stdout[-2000:])  # tail for our own observability

    finally:
        # 4. Destroy the machine. Working directory, package caches, installed
        #    toolchain, dotfiles, background processes, the token file and
        #    anything the job tried to plant for its successor all cease to
        #    exist at the same instant. There is no cleanup script to get wrong,
        #    because there is nothing left to clean.
        sbx.kill()


def supervise(pending: int) -> None:
    """`pending` comes from your own signal -- CircleCI's Insights API, a
    webhook on job creation, or simply a fixed floor of warm claimers."""
    want = min(pending, MAX_IN_FLIGHT)
    with futures.ThreadPoolExecutor(max_workers=MAX_IN_FLIGHT) as pool:
        for _ in range(want):
            pool.submit(run_one_task)

Two independent stop conditions is the property to preserve as this grows. The exec timeout is a circuit breaker for a task that hangs. The create-time TTL is a backstop for a supervisor that dies mid-flight. Neither depends on the other being correct, which is what you want from the component whose entire job is guaranteeing that nothing outlives its build.

One design choice worth calling out. Because the runner claims work rather than being pushed it, you do not need a perfect demand signal. A supervisor that keeps a handful of single-task claimers alive at all times, each of which exits after its idle timeout if nothing arrives, degrades gracefully: too few and jobs queue for a fraction of a second longer, too many and you throw away some sub-second creates. Neither mistake is expensive, which is a nice change from tuning an autoscaling group where being wrong costs minutes.

The cold-start objection, and the number that answers it

Here is the objection, and it is a fair one: a VM per job sounds slow. Every team that has tried per-build instances on conventional cloud VMs has met the same wall — provisioning takes long enough to be annoying, someone adds a warm pool to hide it, the warm pool gets reused to stay warm, and now you are back to a persistent runner wearing a disguise. Freshness was paid for and quietly given back.

The way out is to stop booting. A Firecracker microVM can be snapshotted while running — memory, vCPU state, device state — and restored later. So you boot the template once, with the runner binary and the toolchain and your caches already in place, freeze it warm, and every subsequent machine is a restore of that frozen state rather than a kernel coming up from scratch.

The difference is not incremental. On PandaStack the genuine cold boot — the first spawn of a template that has no snapshot yet — takes about three seconds, and it happens once, at bake time, amortised across every job that ever restores from it. The restore step itself is around 49 milliseconds. End to end, the create path measures roughly 179ms at p50 and 203ms at p99, and that includes allocating the network, cloning the root filesystem, starting the VMM, loading the snapshot, resuming and confirming the guest is answering.

Underneath, the reason a hundredth guest is as cheap as the first: memory is mapped copy-on-write, so a restore does not copy gigabytes, it maps them and pays only for pages the guest actually dirties. The root filesystem is a reflink clone, which is an O(metadata) operation — the data blocks are shared until something writes. Same-host forks of an already-running guest land in the 400 to 750 millisecond range, which is what you would reach for if you wanted a per-matrix-entry machine branched off a common post-install state.

Caching without cross-contamination

The reflex objection to disposable machines is that you throw away the caches that make builds fast. It is worth separating two things that get called 'cache' and behave completely differently.

The first is CircleCI's own caching: restore_cache and save_cache keyed by a checksum of your lockfile, and workspaces persisted between jobs in a workflow. These are content-addressed and delivered over the network into whatever machine the job is running on. They are perfectly compatible with a fresh VM per job, because they were never machine state in the first place — the cache key is derived from your inputs, the object is fetched, and if the key does not match you do the work. A fresh rootfs plus a keyed restore is the good arrangement, and it is the one CircleCI already nudges you toward.

The second is a shared cache directory or a shared volume mounted into every job on the runner, which teams add when the first kind feels too slow. This is the arrangement to be suspicious of. A writable directory that every job on a resource class can modify is a shared attack surface with a very short exploit: write a malicious package into the cache under a name a later job will resolve, and wait. It is the persistent-runner problem with extra steps, and mounting it into otherwise-isolated VMs re-imports exactly the contamination you built the VMs to prevent.

The microVM answer is to move the cache from the machine into the image. Do the clone and the dependency resolve during the template build, then snapshot. Because restore is copy-on-write, every job inherits a read-mostly view of that warmed cache and dirties only its own pages, which vanish with the guest. You get the speed of a shared cache without the write path that lets one job poison the next: a fork PR can scribble on its copy-on-write pages all it likes and cannot write back into the baked layer the next job restores from. Treat the template version as a cache key and rebuild it when your lockfile or default branch moves.

If a pipeline genuinely needs write-through persistence — a large incremental compiler cache, say — attach a durable volume deliberately, scope it to that one pipeline, mount it read-only for untrusted work and let exactly one trusted job be the writer. The thing you are avoiding was never persistence. It was persistence that everybody shares because it happened to live on the same box.

Concurrency and what actually limits it

Because a machine runner instance services one task at a time, concurrency is a count of machines. Twenty concurrent jobs means twenty guests. The question people ask next is what the ceiling is, and the answer is almost never the thing they expect.

It is not network slots. Each guest on our agents gets its own network namespace and a dedicated /30 subnet from a pool of 16,384 pre-allocated per agent, pre-built precisely because creating namespaces, veth pairs and firewall rules on the hot path costs far more than patching a MAC address on a slot that already exists. That pool is large enough that you will meet every other limit first. It is also the mechanism that makes per-job egress policy real rather than aspirational: a job's network is a segment you can write rules against, not a shared bridge everyone sits on.

The binding constraint is host RAM, and then CPU. A guest's memory is set when the template is baked and cannot be changed at restore time, so your per-job footprint is a template design decision made once. Our general-purpose base template is 4 GiB with 8 vCPU of burst, and lighter templates are 2 GiB. Divide your host's usable memory by that figure and you have your realistic per-host concurrency; everything else is arithmetic about how many hosts you want.

The vCPU number is burst capacity rather than a reservation. Eight vCPU on a guest does not mean eight cores are idle waiting for it; CPU shares are weighted so a job that wants the whole machine can have it when nobody else does, and gets a fair slice when the host is busy. This is the right shape for CI, where load is spiky and the median job spends a lot of its time waiting on the network. It also means the capacity conversation is about memory first, memory second and CPU third.

1. Size the template, not the job. Memory is baked at snapshot time, so a 4 GiB template is a 4 GiB job whatever your supervisor asks for.
2. Divide host memory by template memory for per-host concurrency, then leave headroom for the host agent and the page cache.
3. Scale host capacity on a slow loop measured in minutes, because adding a host is slow and getting it wrong is expensive.
4. Create and destroy per-job guests with no pool at all, because a sub-second create does not need one and a pool is just reuse waiting to happen.
5. Watch queue wait time, not host utilisation. High utilisation on a CI fleet is a success; a growing wait time is the signal that you need another host.

Cloud executors vs long-lived self-hosted runner vs microVM per job

Side by side, with the usual caveat that everything about CircleCI's own products here is qualitative, configuration-dependent and worth verifying against their current documentation. Only the PandaStack latency figures are measured.

• Isolation boundary — CircleCI cloud: a container on shared infrastructure for the docker executor, a dedicated VM for the machine executor. Long-lived self-hosted runner: none beyond the OS user, unless you add one yourself. MicroVM per job: a hardware-virtualized guest with its own kernel, one per job.
• What survives a job — CircleCI cloud: nothing on the machine executor; a fresh container on the docker executor. Long-lived self-hosted runner: the toolchain, the dotfiles, the caches, any background process, and anything deliberately planted. MicroVM per job: nothing, because the machine is deleted.
• Time to a machine — CircleCI cloud: managed, generally quick, and not something you tune. Long-lived self-hosted runner: effectively zero, which is exactly why the reuse is tempting. MicroVM per job: about 179ms at p50 and 203ms at p99 via snapshot restore; a genuine cold boot is about 3s and happens once at bake time.
• Building container images — CircleCI cloud: supported, with remote-docker or the machine executor. Long-lived self-hosted runner: a shared daemon and image store on the host, or a mounted socket, which is effectively root on the box. MicroVM per job: an ordinary unprivileged daemon inside the guest, because nothing is nested.
• Fork pull requests — CircleCI cloud: run in someone else's disposable environment; the blast radius is a job. Long-lived self-hosted runner: strangers' code on a box holding your join token and your caches. MicroVM per job: strangers' code on a machine that exists for the length of their build and then does not.
• Secrets exposure window — CircleCI cloud: the job. Long-lived self-hosted runner: the job, plus whatever lingers on disk and in memory for every subsequent tenant. MicroVM per job: the job, ending when the guest is destroyed — which is what makes short-lived OIDC credentials genuinely short-lived.
• Data residency and network access — CircleCI cloud: execution happens on the vendor's infrastructure. Long-lived self-hosted runner: your network, your kernel, your VPC — the reason you self-hosted. MicroVM per job: same as self-hosted, since the hosts are yours; the guests just do not outlive their jobs.
• Idle cost — CircleCI cloud: metered per job. Long-lived self-hosted runner: you pay for the box whether or not it is building. MicroVM per job: you pay for hosts, sized on a slow loop, with no warm pool of idle guests to fund.
• What you operate — CircleCI cloud: nothing. Long-lived self-hosted runner: a box, its configuration drift, and a cleanup story you will keep rewriting. MicroVM per job: a host fleet plus a supervisor loop, which is more than nothing and less than a Kubernetes cluster.

Migrating without a big-bang project

You do not have to move everything, and trying to is how this becomes a quarter-long project that gets cancelled. Resource classes make a partial migration genuinely easy: a job names a class, so moving a pipeline is a one-line change and moving it back is the same line.

• Move untrusted work first. Anything that builds fork pull requests, community contributions or vendor-supplied build steps. This is where the isolation argument is not a preference but a finding.
• Move the runner-corrupting pipelines second. Every fleet has two or three: the one that leaves daemons running, the one that fills the disk with image layers, the one after which that runner is mysteriously flaky. Give them disposable machines and the recurring incident simply stops happening.
• Give the disposable class its own token and its own contexts. A leak from the untrusted class must not be able to receive jobs from the deploy class.
• Keep a small, separate, trusted class for deploys. Deploy jobs want long-lived credentials and privileged network reach, and they should never share a machine class with fork PRs.
• Leave the rest alone until you have a reason. A pipeline that builds internal code from an internal repo on an internal runner is not the problem you are solving today.

The summary I would give a sceptical reviewer is short. CircleCI's cloud machine executor already established the standard: a job gets a machine, the machine dies with the job. Self-hosting is a decision about where compute runs, not a licence to abandon that property — and the only reason anyone abandons it is that on conventional infrastructure, creating a machine per job is slow enough to hurt. Snapshot restore removes that reason. Once a fresh, fully-provisioned guest costs a fifth of a second, reusing a build host stops being a performance trade-off and becomes what it always was underneath: an unexamined decision to let every job inherit the last one's leftovers.