Docker-in-Docker vs microVMs for CI builds

Every platform team hits this problem in roughly week three. Your CI jobs need to build container images. Your CI runners are themselves containers, scheduled onto a Kubernetes node or a fleet of shared hosts. So the build needs a Docker daemon, and the container it's running in doesn't have one. Now what.

There are four real answers, and the internet will confidently give you the first because it's a single flag. I'm Ajay; I build PandaStack, which runs workloads as Firecracker microVMs, so you can guess where I land — but I'd rather you land there for the right reasons. Three of these four are legitimately correct for some teams, and I'll say which.

The setup: a container that needs to build containers

Building an OCI image is not a normal userspace task. The classic builder unpacks layers as root, creates device nodes, sets arbitrary file ownership, mounts overlay filesystems, and drives a container runtime for each `RUN` step. That's a pile of privileged kernel operations, and a stock unprivileged container is specifically designed not to have them.

So you have to give the job something. The four ways: point it at a daemon that already has privilege (the socket mount), run a second privileged daemon inside the job (true DinD), use a builder that avoids needing most of that privilege (rootless/daemonless builders), or give the job its own kernel so "privileged" stops being a question about your host at all (a microVM per build).

Option 1: mount the Docker socket

This is the one that shows up in every StackOverflow answer, every internal wiki, and a distressing number of production pipelines. You bind-mount the host's Docker socket into the job, and suddenly `docker build` works.

# The convenient version. Works instantly. Cache is warm. Everyone is happy.
docker run --rm \
  -v /var/run/docker.sock:/var/run/docker.sock \
  ci-image:latest sh -c 'docker build -t app:$CI_COMMIT_SHA .'

# What "just build access" also grants, from inside that same job:
docker run --rm -v /:/host alpine cat /host/etc/shadow
docker run --rm -v /:/host alpine sh -c 'cat /root/.ssh/id_ed25519'
docker run --rm --privileged --pid=host alpine nsenter -t 1 -m -u -i -n sh
# ^ that last one is a root shell in the host's namespaces.

It's worth being precise about why this is bad, because "container escape" is the wrong mental model — it makes people think it's an exotic risk. Nothing escaped; no bug is involved. The Docker API includes, by design, the ability to start a container with `--privileged`, with the host root filesystem bind-mounted, in the host PID namespace. Any process that can reach the socket can ask for exactly that. You didn't leave a hole; you handed over the remote control.

The practical translation: mounting the Docker socket into a CI job gives that job's code root on the host. Not "root inside a sandbox" — root on the machine, with access to every other job's files, the runner's credentials, the kubelet's certificates. If your CI ever builds a pull request from outside your org, you have published a root shell and put a webhook on it.

And yet: it's fast, it's one line, and it gives you a warm shared layer cache for free, because the host daemon already has every base image pulled. That combination is precisely why it's everywhere. If your CI runs only first-party code from committers you'd hand an SSH key to, the socket mount is a defensible call. It stops being defensible the instant an outsider's code can reach it.

Option 2: true Docker-in-Docker

The next rung is running a real, second Docker daemon inside the job. GitLab CI popularised this with the `docker:dind` service; the shape is familiar:

# .gitlab-ci.yml — the classic true-DinD job.
# NOTE: the runner itself must be configured with privileged = true
# in config.toml for this service to start at all.
build-image:
  image: docker:27-cli
  services:
    - name: docker:27-dind
      alias: docker
      command: ["--mtu=1450"]   # nested networking and MTU are old enemies
  variables:
    DOCKER_HOST: tcp://docker:2376
    DOCKER_TLS_CERTDIR: "/certs"
    DOCKER_CERT_PATH: "/certs/client"
    DOCKER_TLS_VERIFY: "1"
    DOCKER_DRIVER: overlay2
  script:
    - docker info      # confirm you're talking to the nested daemon, not the host's
    - docker build --pull -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA" .
    - docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"

This is better than the socket mount in one important respect: the job talks to its own daemon, not the host's. Images, containers, and networks it creates are its own — there's no ambient handle to the host's container inventory.

What `--privileged` actually drops

But that nested daemon needs privilege to do its job, and `--privileged` is a flag whose name is refreshingly honest. It is not a dial; it's a switch that turns off most of what makes a container a container:

• Capabilities — instead of the default trimmed set, the process gets the full capability set, including `CAP_SYS_ADMIN` (mount, namespace manipulation, and a long tail of everything-else) and `CAP_MKNOD`.
• Seccomp — the default syscall filter is not applied, so the full host syscall surface is reachable rather than the curated subset.
• AppArmor/SELinux confinement — the default profile is not applied, removing the mandatory-access-control layer that would otherwise block a lot of opportunistic misbehaviour.
• Device access — the device cgroup is opened up, so host block devices under `/dev` become visible and often writable. A raw disk is a filesystem you can read around every permission check above it.
• The consequence — a privileged container is a root process on the host kernel with the rails removed. It is closer to "a root shell on the node" than to "a container." Escaping it is not an exploit; it's an afternoon.

So true DinD fixes the API-surface problem and does nothing for the kernel problem. The nested daemon's containers are still processes on your host kernel, shared with every other job on the node. Against a malicious build script, privileged DinD is a smaller target than the socket mount but the same category of target.

The operational friction nobody warns you about

DinD is also annoying in ways that consume real engineering weeks:

• Storage-driver nesting. Overlayfs on overlayfs has historically been a source of pain, and which combinations work depends on your kernel version, host storage driver, and distro patches. Teams end up on `vfs` (correct and appallingly slow, since it copies every layer) or fighting to get `overlay2`-on-`overlay2` working. Verify what your kernel supports rather than trusting a blog post — including this one.
• Cold layer cache, every single job. The nested daemon starts empty. Every job re-pulls your base images and rebuilds every layer, because there is no cache to hit. This is the number-one reason teams quietly revert to the socket mount.
• Cache-warming hacks. The workarounds are all a bit sad: mounting a host directory as the nested `/var/lib/docker` (which reintroduces shared mutable state, and therefore cache poisoning), a pull-through registry mirror, or `--cache-from` against a registry, trading local disk reads for network round-trips. Nested bridge networks also inherit the wrong MTU surprisingly often, producing builds that hang on `apt-get update`.

Option 3: rootless and daemonless builders

This is the option most teams should try before reaching for VMs, and it's genuinely good engineering. BuildKit in rootless mode, Kaniko, and Buildah attack the problem from the other end: instead of granting privilege, they rearchitect the build so most of it isn't needed — user namespaces (so "root" inside the build is an unprivileged UID outside), userspace filesystem handling, and in Kaniko's case an in-process layer extractor rather than a container runtime.

What you get is meaningful: no exposed daemon socket, no `--privileged` on the job, and a much smaller set of things the build can ask the kernel for. For a platform team building first-party images at scale this is frequently the right answer, and BuildKit's cache export/import story is better thought-out than anything you'll bolt onto DinD.

Where it gets awkward, honestly:

• Feature gaps versus the classic builder. Some Dockerfile patterns, mount types, and networking behaviours differ or need extra flags. Builds that do privileged things inside a `RUN` step (loop mounts, `mknod`, some package postinstall scripts) are the usual casualties.
• Rootless networking and storage quirks. User-namespace networking typically routes through a userspace path with its own performance and MTU characteristics, and rootless storage-driver support depends on kernel features being enabled on your nodes. It works well on modern kernels and less well on the LTS one you're pinned to.
• The big one: daemonless is not the same as isolated. Kaniko executes the build's `RUN` steps in its own container and has always been explicit that it doesn't sandbox them. Rootless BuildKit reduces starting privilege, but the build's code still executes against your host kernel through a user namespace — and user namespaces have themselves been a productive source of kernel bugs. Less privilege is a real win. It is not a separate kernel.

Option 4: a microVM per build

The fourth option changes the question. Give the build its own kernel and the whole "how much host privilege does the builder need" debate evaporates — the privilege it takes is over a kernel that exists solely for this build and gets deleted in ten minutes.

Inside a Firecracker microVM you can run a bog-standard Docker daemon. `--privileged` inside the guest means privileged in the guest. The build can mount overlayfs, create device nodes, and do all the ugly things image builds legitimately need, and none of it is a statement about your host kernel. The isolation boundary is hardware virtualization (KVM) plus a very small virtio device surface — the same model AWS Lambda uses to run untrusted code from millions of customers.

# ci/build_in_microvm.py
# The CI runner never builds. It delegates to a VM that gets destroyed after.
import sys
from pandastack import Sandbox

REPO, SHA = sys.argv[1], sys.argv[2]

build = f"""#!/usr/bin/env bash
set -euo pipefail

# A plain dockerd. It owns this kernel; this kernel owns exactly one build.
dockerd --host=unix:///var/run/docker.sock >/var/log/dockerd.log 2>&1 &
for _ in $(seq 1 40); do docker info >/dev/null 2>&1 && break; sleep 0.5; done

git clone --depth 1 {REPO} /workspace/repo
cd /workspace/repo
git fetch --depth 1 origin {SHA} && git checkout {SHA}

docker build --pull -t app:{SHA} .
docker save app:{SHA} | gzip > /workspace/image.tar.gz
"""

# ttl_seconds is a backstop: if this orchestrator dies, the VM still reaps itself.
with Sandbox.create(template="base", ttl_seconds=1800) as sbx:
    sbx.filesystem.write("/workspace/build.sh", build)
    r = sbx.exec("bash /workspace/build.sh", timeout_seconds=1500)
    print(r.stdout)
    if r.exit_code == 0:
        sbx.exec("curl -sf -T /workspace/image.tar.gz $ARTIFACT_SINK")
    sys.exit(r.exit_code)
# VM destroyed here, along with anything the build did to it.

(You'd bake `docker` into the template rather than installing it per job — a template is just an image you snapshot once. The guest is Ubuntu 24.04 on a Linux 5.10 kernel, so anything your build needs is a normal `apt` line at bake time.)

What this buys you:

• Kernel-level blast radius. A build that compromises its container, its daemon, and its guest kernel has compromised a VM you were about to delete. No host kernel is shared with the next tenant's build.
• No `--privileged` anywhere on the host. The host runs Firecracker processes with a small, auditable interface. It does not run privileged build containers.
• Per-build clean state by construction. Not a wipe script you hope ran — the machine ceases to exist. Toolchain drift, leftover cron jobs, and "it passed on runner 7" go away together.
• Normal tooling inside. Nobody has to port Dockerfiles to a builder with different semantics — it's the classic builder, behaving classically, in a box.

And the honest costs:

• You are booting a VM. Which is why boot time is the entire ballgame — see below.
• You need hardware virtualization — bare metal or an instance type exposing nested virtualization. That rules out some managed Kubernetes node pools and constrains your fleet design.
• A microVM does not fix your layer cache. A fresh VM has a cold daemon exactly like a fresh DinD service. Solvable, but you have to actually solve it.

On boot time: the old objection was that nobody waits seconds to boot a machine for a two-minute build. The answer is to stop cold-booting. PandaStack creates a sandbox by restoring a baked Firecracker snapshot — around 179ms p50 (p99 ~203ms), with the restore step itself near 49ms. Only the first-ever boot of a never-used template pays the full ~3s; every create after takes the snapshot path. At those numbers a fresh isolated machine per build is scheduling noise, not a pipeline stage.

The layer cache is what actually decides this

The thing I'd tell any team evaluating these four: you will not lose this argument on security. You'll lose it on build times. Somebody migrates one pipeline off the socket mount, watches a five-minute build become fifteen because every base image is re-pulled, and the migration dies in the retro. Cold caches kill adoption. Plan for the cache first.

There are two workable answers with a microVM per build, and they compose.

Answer one: bake the warm state, then fork it

This is genuinely hard to do with containers, and it's why the VM model turns out to be an advantage rather than a tax. Do the expensive warm-up once — pull base images, prime the dependency cache, start the daemon — then snapshot that machine. Every build forks the snapshot, getting the populated layer store and an already-running daemon, because it's a copy-on-write clone of a machine where those things were already true.

# warm_pool.py — pay for the cold cache once, per day, not per build.
from pandastack import Sandbox

WARM = """#!/usr/bin/env bash
set -euo pipefail
dockerd --host=unix:///var/run/docker.sock >/var/log/dockerd.log 2>&1 &
for _ in $(seq 1 40); do docker info >/dev/null 2>&1 && break; sleep 0.5; done
docker pull node:22-bookworm
docker pull python:3.12-slim
docker pull our-registry.internal/base/runtime:stable
"""

warm = Sandbox.create(template="base", ttl_seconds=3600)
warm.filesystem.write("/workspace/warm.sh", WARM)
warm.exec("bash /workspace/warm.sh", timeout_seconds=900)
warm.snapshot()          # freeze a running daemon with a populated layer store

# Per build: fork instead of booting. Same-host forks land in 400-750ms and
# share memory + disk copy-on-write; cross-host is 1.2-3.5s.
job = warm.fork()
job.filesystem.write("/workspace/build.sh", build_script)
print(job.exec("bash /workspace/build.sh", timeout_seconds=1500).stdout)
job.destroy()            # the "did that build poison my cache" question dies here

The security property is the part I like: the warm cache is read-only in effect, not by policy. A build can happily write garbage into its layer store — into its own copy-on-write view, discarded when the fork dies. The parent snapshot is never mutated. You get a warm cache and a poison-proof cache from one mechanism, which is not a trade you can normally make.

Answer two: an explicit shared cache, scoped deliberately

The complementary approach is a cache the build reaches over the network rather than a directory it can corrupt: a pull-through registry mirror, a BuildKit remote cache backend, or `--cache-from` against a registry tag. Slower than local disk, but it survives across hosts and snapshot rebakes, and degrades gracefully.

The rule: shared caches are fine as long as untrusted builds can read them and cannot write them. A trusted pipeline on your default branch populates the cache; PR builds from strangers consume it and push nothing. The failure mode you're avoiding is a hostile PR writing a backdoored layer that your next production build reuses.

Side by side

• Isolation boundary — Socket mount: none; the job commands the host daemon. DinD: a nested daemon, still on the host kernel. Rootless builder: user namespaces on the host kernel. microVM: hardware virtualization, own guest kernel.
• Effective privilege — Socket mount: root on the host, by design. DinD: `--privileged` — root on the host with the rails off. Rootless builder: an unprivileged host UID. microVM: root in a guest that gets deleted.
• Safe for code from strangers — Socket mount: no, not close. DinD: no. Rootless builder: better, but still your host kernel. microVM: yes — the model Lambda uses.
• Layer cache out of the box — Socket mount: warm and shared, half its appeal. DinD: cold every job, the top complaint. Rootless builder: cold locally, strong registry-based import/export. microVM: cold unless you snapshot-and-fork a warm VM.
• Cache poisoning — Socket mount: high; one shared mutable daemon. DinD: high if you mount a host `/var/lib/docker` to fix the cold cache. Rootless builder: depends who can write your remote cache. microVM: none via fork — writes land in a copy-on-write view that's destroyed.
• Dockerfile compatibility — Socket mount: perfect, it's the real daemon. DinD: perfect, modulo storage-driver nesting. Rootless builder: mostly, with edge cases around privileged `RUN` steps. microVM: perfect — the classic builder in a box.
• Startup cost — Socket mount: zero. DinD: seconds to start the nested daemon, plus the cold-cache tax. Rootless builder: near zero. microVM: ~179ms p50 via snapshot-restore, or 400-750ms for a same-host fork.
• Operational burden — Socket mount: trivial to set up, career-ending to explain in an incident review. DinD: constant storage-driver and MTU fiddling. Rootless builder: real setup work, then stable. microVM: nested-virt-capable hosts plus a template bake pipeline.

Which one should you actually pick

It depends almost entirely on who can cause code to run in your pipeline.

Solo project or a small trusted team, private repo, no fork PRs. Use the socket mount and don't feel bad about it — the privilege you're granting, you're granting to yourself. Just make the call deliberately, write it down, and revisit it the day you accept your first outside contributor. The failure mode here isn't the choice; it's forgetting you made it.

Internal platform team, first-party code, many services. Rootless BuildKit or Buildah is the sweet spot, and I'd try it before anything heavier: no privileged daemon, no exposed socket, a good remote-cache story, and no new hypervisor in your fleet. The work is up-front (subuid ranges, cgroup delegation, the handful of Dockerfiles doing something exotic in a `RUN` step) and then it's boring, the highest compliment available in infrastructure. Reach past it only for a specific reason: builds that genuinely need privileged operations, a hardware-isolation compliance requirement, or untrusted code in the mix.

Multi-tenant CI accepting builds from strangers. This one isn't really a debate. If you're building a CI product, a preview-deploy service, an AI agent that runs generated code, or an OSS project that builds fork PRs, your tenants' build scripts are hostile by assumption. The socket mount and privileged DinD put that code on your host kernel with elevated privilege; rootless builders reduce the privilege but keep the shared kernel. Every company that has actually shipped this converged on VM-level isolation, mostly microVMs, because it's the only option where a tenant's worst day is confined to a machine you were going to throw away.

Ask the question in reverse: if a build script did the single worst thing you can imagine, what's the largest thing it could reach? For the socket mount and DinD, the answer is the node. For rootless, it's the host kernel. For a microVM, it's a machine with a ten-minute lifespan.

The shortest version of all of it: privilege has to live somewhere for image builds to work. Your only real choice is whether it lives on a kernel other people's workloads share, or on a kernel that exists for one build and then doesn't. Snapshot-restore made the second option cheap enough to stop being a theoretical preference. The layer cache decides whether your team lets you do it — solve that first, and the security argument makes itself.