BuildKit vs Kaniko vs microVMs for Untrusted Image Builds

Somewhere in your pipeline there is a line that reads RUN npm ci, and it is the most trusting line in your entire infrastructure. A Dockerfile is not a manifest. It is a script. Every RUN is arbitrary code, executing as root, with network access, at build time — and whatever you built to execute it is not really a build tool. It is a code-execution engine wearing a build tool's clothes.

That does not matter much when you wrote the Dockerfile. It matters enormously the day your product lets customers supply one — a PaaS, an internal developer platform, an AI agent that generates images, a CI system that builds arbitrary forks from arbitrary repos. At that point "how do we build images" stops being a tooling question and becomes an isolation question, and the three serious answers are BuildKit, Kaniko, and a microVM per build.

I build PandaStack, which runs workloads as Firecracker microVMs, so you can guess which one I use. But two of the other options are genuinely correct for most teams most of the time, and pretending otherwise would waste your afternoon. Here is the fair version.

The shared problem: RUN is a shell, not a spec

Whichever builder you pick, the same thing happens: a process reads instructions from a file the build's author controls and executes them. It downloads whatever URLs that file names. It runs whatever install scripts those downloads contain. It does this as root, because package managers want to be root. And it does it with whatever credentials, network reach, and mounted secrets the builder happened to have available.

So the real comparison is not "which builder is fastest." It is: when the code inside RUN turns hostile, what is standing between it and everything else on that machine? The three tools answer that question at three different layers — user namespaces, a container filesystem, and a hypervisor — and everything else about them follows from that choice.

BuildKit: the modern default, and mostly the right one

BuildKit is what Docker builds with today, and what most other tools are quietly re-implementing. Instead of walking a Dockerfile line by line, it compiles the build into LLB — a low-level directed acyclic graph of build steps — and then solves that graph. Because it knows the dependency structure rather than just the line order, it can execute independent stages in parallel, skip stages nothing depends on, and reason precisely about what a cache hit means.

That DAG is also where its best feature lives. A cache mount lets a step keep a persistent, non-layer directory across builds — your npm store, your Go module cache, your pip wheels — so dependency installation stops re-downloading the internet on every run without those files bloating the final image. Combined with the export and import cache options, BuildKit will push its build cache to a registry, which is the single change that makes ephemeral CI runners economically sane: a brand-new runner with an empty disk still starts warm.

# BuildKit through buildx, with a registry-backed cache so a fresh CI
# runner starts warm instead of starting from scratch every single time.
docker buildx create --name ci --driver docker-container --use

docker buildx build \
  --builder ci \
  --file Dockerfile \
  --platform linux/amd64,linux/arm64 \
  --cache-from type=registry,ref=registry.example.com/app:buildcache \
  --cache-to   type=registry,ref=registry.example.com/app:buildcache,mode=max \
  --tag registry.example.com/app:"$GIT_SHA" \
  --push .

# The rootless variant. Same Dockerfile, no root on the host.
# Note what it does not change: the RUN steps still execute, still as
# root inside the build, still against the one kernel this box has.
docker buildx create --name ci-rootless --driver kubernetes \
  --driver-opt image=moby/buildkit:rootless,rootless=true --use

Deployment is flexible in a way that matters operationally. You can run buildx against a local container-driver builder on a developer laptop, run a long-lived buildkitd daemon on a beefy build host that many jobs share, or schedule builders in-cluster on Kubernetes and let them scale. The shared-daemon shape is a real advantage: the cache lives with the daemon, so consecutive builds of the same project are fast without any registry round trip at all.

Rootless mode: we removed the gun, the shooter is still in the room

BuildKit's rootless mode is a genuine security improvement and it is widely misread. It uses user namespaces so the builder runs as an unprivileged host user while the build believes it is root. That removes an entire class of problem — the daemon no longer needs the privileged flag, no longer needs to be root on the host, and a bug in the builder itself is much less catastrophic.

What it does not do is stop the build from executing attacker-supplied code. The RUN steps still run. They still run against the host's one shared kernel, reachable through the full Linux syscall surface, alongside every other tenant's build on the same machine. Rootless narrows the blast radius of a compromise; it does not remove the thing doing the compromising. If your threat model is "a customer's Dockerfile is actively trying to reach another customer's build," a user namespace is a speed bump on a road that still goes there.

There is also a practical cost. Some builds legitimately want privileges that rootless cannot grant — mounting loop devices, some FUSE-based workflows, certain nested-container tests, tooling that wants specific capabilities. In rootless mode those builds fail, and the failure modes are not always legible. Teams then reach for the privileged flag, which is the flag whose documentation is a warning, and the whole exercise unwinds. Verify current rootless limitations against the BuildKit documentation before you promise your users that everything will just work.

Kaniko: no daemon, no privileged flag, no host to secure

Kaniko exists because of a specific, extremely common constraint: you are on a managed Kubernetes cluster, you cannot set privileged on a pod, you cannot mount the node's Docker socket, and you still need to produce images. Kaniko builds the image entirely in userspace inside an ordinary container. It extracts the base image into its own root filesystem, executes each Dockerfile command against that filesystem, snapshots what changed after each step, and turns those diffs into layers that it pushes to a registry.

# Kaniko as a Kubernetes Job: no daemon, no privileged: true, no
# docker.sock mounted from the node. That is the whole pitch, and in a
# locked-down cluster it is a very good pitch.
apiVersion: batch/v1
kind: Job
metadata:
  name: build-app
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: kaniko
          image: gcr.io/kaniko-project/executor:latest  # pin a real tag in prod
          args:
            - --dockerfile=Dockerfile
            - --context=git://github.com/example/app.git#refs/heads/main
            - --destination=registry.example.com/app:$(GIT_SHA)
            - --cache=true
            - --cache-repo=registry.example.com/app/cache
            - --snapshot-mode=redo
          volumeMounts:
            - name: docker-config
              mountPath: /kaniko/.docker
      volumes:
        - name: docker-config
          secret:
            secretName: registry-credentials

The operational appeal is real. There is no build daemon to run, patch, monitor, or secure. There is no shared builder whose cache one tenant can poison for another, because there is no shared builder. Every build is a pod, the pod is scheduled by the thing that already schedules everything you run, and when it exits its filesystem goes away. For a platform team whose cluster policy forbids privileged workloads outright, that is often the difference between shipping and not shipping.

Now the honest limits. The first is the one people misunderstand most: Kaniko executes RUN steps in its own container's filesystem, as its own process. It is not sandboxing the build — it is being the build. Whatever the Kaniko pod has, the Dockerfile has: its service account token if one is projected, its network policy, its mounted registry credentials, its view of the cluster. Kaniko's design goal was to avoid needing a privileged daemon, not to contain a hostile Dockerfile. Those are different goals, and only the first one is achieved.

The second limit is performance shape. Snapshotting a filesystem after each command is inherently different work from BuildKit's content-addressed DAG solving, and on images with very large filesystems the snapshot step is where the time goes. There are knobs — snapshot modes that compare only file metadata rather than full contents, ignore paths, cache repositories — and tuning them is a real part of adopting Kaniko. Cache semantics are also coarser: layer caching against a cache repository works, but there is no equivalent of BuildKit's persistent cache mounts, so the package-manager download problem you solved elegantly in BuildKit reappears. Benchmark on your own images rather than trusting anyone's numbers, mine included.

Third, and this is a governance point rather than a technical one: check the project's current maintenance status before you adopt it. Build tooling sits on the critical path of every deploy you will ever do, and the question of who is fixing its bugs next year is a legitimate input to the decision. Look at recent release cadence and issue triage yourself; do not take a blog post's word for it, including this one.

microVMs: move the boundary, then stop crippling the builder

The third option refuses the premise. Instead of asking which builder is safe enough to run hostile code against a shared kernel, give the build its own kernel. A Firecracker microVM boots a guest kernel under KVM with a minimal virtio device model; the guest reaches the outside world through that narrow interface and nothing else. Inside it, you run whatever builder you actually wanted — a fully privileged BuildKit, a real dockerd, Buildah, Kaniko if you like — because the isolation no longer depends on the builder behaving.

That inversion is the whole point. BuildKit and Kaniko both spend design effort making the builder less dangerous, and both pay for it in capability. With a hypervisor boundary you stop paying that tax. The privileged flag inside a VM that exists for one build and is then destroyed is not a scary flag; it is a flag scoped to a kernel nobody else is using.

# build_in_microvm.py -- the Dockerfile arrived from a customer.
# It gets its own guest kernel, and that kernel is destroyed afterwards.
import sys
from pandastack import Sandbox

dockerfile = sys.stdin.read()   # untrusted: attacker-controlled RUN steps

script = """#!/usr/bin/env bash
set -euo pipefail

# A fully privileged BuildKit daemon. It is privileged inside a kernel that
# exists only for this build, so the scary flag costs us nothing we mind.
buildkitd --addr unix:///run/buildkit/buildkitd.sock >/var/log/buildkitd.log 2>&1 &
for _ in $(seq 1 40); do buildctl debug workers >/dev/null 2>&1 && break; sleep 0.5; done

buildctl build \\
  --frontend dockerfile.v0 \\
  --local context=/workspace/ctx \\
  --local dockerfile=/workspace/ctx \\
  --import-cache type=registry,ref=registry.example.com/app/cache \\
  --export-cache type=registry,ref=registry.example.com/app/cache \\
  --output type=docker,name=app:build,dest=/workspace/image.tar
"""

# ttl_seconds is the backstop: if this orchestrator dies, the VM reaps itself.
sbx = Sandbox.create(template="base", ttl_seconds=1800)
try:
    sbx.filesystem.write("/workspace/ctx/Dockerfile", dockerfile)
    sbx.filesystem.write("/workspace/build.sh", script)
    r = sbx.exec("bash /workspace/build.sh", timeout_seconds=1500)
    print(r.stdout, r.stderr)
    if r.exit_code == 0:
        sbx.exec("curl -sf -T /workspace/image.tar $ARTIFACT_SINK")
finally:
    sbx.kill()   # the blast radius of that Dockerfile ends here

The costs are honest and worth stating plainly. You are now operating virtual machines, which is a different operational discipline from scheduling pods. You need KVM, which means bare metal or a cloud instance type that exposes nested virtualisation — so this is simply not available on many managed CI providers, and that constraint alone rules it out for some teams. And every VM starts with an empty layer cache unless you deliberately arrange otherwise, which is the next section.

The startup cost people fear is mostly historical. On PandaStack a sandbox create runs around 179ms p50 and 203ms p99, because every create restores a baked snapshot rather than cold-booting; the restore step itself is near 49ms, and only the very first boot of a template pays the full ~3s. Against a build that takes minutes, VM startup is not the number worth optimising. What is worth optimising is the cache.

What are you actually defending against?

This is the question that collapses the decision, and most teams skip it. There are two genuinely different situations and they have different right answers.

Case one: your own engineers wrote the Dockerfile

The code in RUN is code you already run in production. A malicious dependency is a real risk, but it is a supply-chain risk you carry regardless of your builder, and the correct mitigations are pinned digests, a vetted registry mirror, restricted build-time egress, and provenance attestation — not hypervisor isolation. Here, all three options are defensible and you should pick on speed and operational fit. That almost always means BuildKit, or Kaniko if your cluster policy forbids anything else.

Case two: someone else wrote it, and you cannot read it first

A customer pushed it. An agent generated it. A fork opened a pull request. Now the build is deliberate, adversarial code that runs before any of your scanning or policy gates get a look at the result — and it runs concurrently with other tenants' builds. Isolation stops being one criterion among several and becomes the criterion. A shared host kernel with thousands of syscalls of attack surface is not the boundary you want between two paying customers. Here the microVM answer stops being an indulgence and starts being the design.

Pick your builder for speed when the code is yours. Pick your boundary for isolation when the code is someone else's. Confusing those two is how multi-tenant build services end up in incident reports.

Cache strategy, which is where real-world speed actually lives

Whatever you pick, the difference between a two-minute build and a twelve-minute one is almost never the builder's raw execution speed. It is whether the dependency install step had to re-download everything. All three models can be made fast; they get there differently.

• BuildKit, shared daemon — the fastest arrangement available. Cache lives on the builder's disk, cache mounts persist package-manager stores across builds, and repeat builds of the same project barely touch the network. The trade is that tenants share a cache, so cache poisoning is now part of your threat model.
• BuildKit, ephemeral runner — use registry-backed cache export and import so a fresh runner pulls warm layers instead of rebuilding. Slower than a warm local daemon, dramatically faster than nothing, and it keeps tenants separated by using a per-tenant cache repository.
• Kaniko — layer caching against a cache repository, plus a tuned snapshot mode to keep the per-step snapshot cost down. There is no persistent cache-mount equivalent, so plan for dependency installs to be genuinely repeated and structure your Dockerfile so that the expensive steps sit above anything that changes per commit.
• microVM, per build — a fresh VM inherits nothing by default, which is exactly the property you wanted for isolation and exactly the property that hurts here. Import from a registry cache inside the guest, and warm the base images into the template snapshot itself so the common layers are already on disk when the VM starts.
• microVM, forked from a warm builder — boot a VM, populate its layer store, snapshot it, then fork per build. Same-host forks land in 400-750ms and share memory and disk copy-on-write; cross-host forks are 1.2-3.5s. Each build gets a warm cache and its own kernel, and whatever it did to that cache dies with it.

That last row is the interesting one, because it is the only arrangement where cache warmth and tenant isolation are not in tension. Everywhere else you are choosing: share a cache and accept that a hostile build can influence a later one, or isolate fully and pay the cold-cache cost every time.

Head to head

• Isolation boundary — BuildKit rootless: user namespaces, shared host kernel. Kaniko: container filesystem, shared host kernel. microVM: guest kernel under KVM.
• Needs a privileged daemon — BuildKit: rootless mode avoids it, some builds still want privileges it cannot grant. Kaniko: no, that is the entire design goal. microVM: irrelevant, the guest can be privileged safely.
• Dockerfile fidelity — BuildKit: reference implementation, everything works including newer frontend syntax. Kaniko: broadly compatible, verify edge cases against its docs. microVM: whatever the builder inside supports, because you can run the reference implementation.
• Cache semantics — BuildKit: strongest, with cache mounts plus registry import and export. Kaniko: layer caching to a cache repo, no persistent cache mounts. microVM: inherits the builder's cache model, plus snapshot and fork for VM-level warmth.
• Parallelism — BuildKit: DAG-aware, independent stages run concurrently. Kaniko: sequential per Dockerfile command by construction. microVM: whatever the inner builder does, at full privilege.
• Where it runs — BuildKit: laptops, build hosts, in-cluster, managed CI. Kaniko: anywhere a pod runs, including locked-down clusters. microVM: hosts with KVM only, which excludes most managed CI runners.
• Operational weight — BuildKit: a daemon or builder fleet to run and secure. Kaniko: a pod, nothing else. microVM: a VM fleet, snapshots, networking, and the scheduling to go with it.
• Multi-tenant untrusted input — BuildKit: not the design goal, rootless narrows but does not close it. Kaniko: not the design goal, the build inherits the pod's identity. microVM: this is the design goal.

Pick this if

Pick BuildKit if your Dockerfiles are first-party. It is the fastest, has the best cache story by a distance, has the highest Dockerfile fidelity because it is the reference implementation, and it is what the ecosystem is converging on. Turn on rootless mode anyway — it is a real hardening win and costs most teams nothing. This is the default and you should need a reason to leave it.

Pick Kaniko if your constraint is the environment rather than the threat model: a managed Kubernetes cluster that forbids privileged pods and node socket mounts, where running and securing a builder daemon is not something your team wants to own. Accept slower builds on large images and coarser caching as the price of fitting the policy. Structure Dockerfiles so the expensive layers are stable, and check the project's maintenance status before you commit your deploy path to it.

Pick microVMs when the Dockerfile is not yours. Customer-supplied builds, agent-generated builds, arbitrary forks in public CI, anything where one tenant's build must not be able to reach another's. You take on VM operations and a KVM requirement, and you have to warm caches deliberately instead of inheriting them — but you get to run an uncrippled builder behind a hypervisor, and "the build did something horrible" becomes a VM you delete rather than an incident you investigate.

There is also a perfectly reasonable hybrid, and plenty of platforms run it: BuildKit on a shared builder for your own images, microVMs for anything a customer supplied. The two paths have different security postures because they have different inputs, and one pipeline pretending both are the same is how the interesting failures happen.

The one answer that is always wrong is the one that gets chosen by default — mounting the Docker socket because it was a single flag and the build went green. That is not a point on this spectrum. It is root on the host, handed to whoever can open a pull request.