Tekton Steps, MicroVM Bodies: Isolating Untrusted Tasks

Tekton's founding idea is that continuous integration should be Kubernetes objects rather than a system that happens to run on Kubernetes. A Task is a custom resource. A TaskRun is a custom resource. A controller reconciles the second into a Pod. Everything you already built for the cluster — RBAC, admission control, network policy, GitOps, your observability stack, your cost attribution — applies to CI for free, because CI is now just more objects. If your platform team's instinct is to reach for a CRD and a controller, Tekton fits that worldview more completely than anything else in the field.

The consequence people internalise more slowly is the second half of that sentence: a TaskRun is a Pod. Not a Pod per Step — a Pod per TaskRun, with each Step as a container inside it, sharing the Pod's volumes, the Pod's service account, the Pod's network identity, and the node's kernel. That is a deliberate and good design. It is what makes Steps cheap, what makes passing a workspace between them trivial, and what makes a Tekton build inspectable with kubectl. It is also, precisely, a single trust boundary drawn around a group of containers, one of which may be running a build script that arrived in a pull request from a person you have never met.

For first-party builds by employees on repositories you control, that boundary is completely fine and I would not lift a finger to change it. For fork pull requests, customer-supplied build configuration, catalog Tasks resolved from a shared registry, and image builds that keep asking for privilege, it is uncomfortable in a structural way that no amount of securityContext tuning fully resolves. This post is about the specific mechanics of why, and about a pattern that keeps Tekton exactly as it is — orchestrator, control plane, source of truth — while moving the untrusted body of a step out of the Pod and into a microVM that is created for that step and deleted after it.

How a TaskRun actually becomes a Pod

You cannot reason about the blast radius without knowing the machinery, and Tekton's machinery is unusually visible if you look. Here is what happens, in the order it happens.

The object model, briefly

A Task declares params, workspaces, results and an ordered list of Steps. A Step is essentially a container spec: an image, and either a command plus args or a script block that Tekton writes to a file and executes with the shebang you gave it. A TaskRun is a request to run one Task with concrete parameter values, workspace bindings, a service account and a timeout. A Pipeline strings Tasks together with runAfter ordering and result plumbing, and a PipelineRun executes one. Newer versions add StepAction as a reusable, individually-referenceable step, which is a genuinely nice unit and does not change the isolation story at all.

The controller turns each TaskRun into exactly one Pod. Steps become containers in that Pod's containers list, in declaration order. Sidecars — long-running helpers like a Docker daemon or a test database — become additional containers that start alongside the steps and are stopped when the steps finish. Init containers handle setup: placing Tekton's own tooling into a shared volume, and running the credential initialiser.

The entrypoint binary, or how sequential containers are faked

Kubernetes has no notion of "run these containers one after another". All containers in a Pod start together. Tekton's answer is charmingly low-tech: it copies a small entrypoint binary into a volume shared by every step container, rewrites each step's command to invoke that binary instead of your command, and passes it a wait file and a post file. Step 2's entrypoint blocks until step 1's entrypoint writes its completion marker under Tekton's own directory tree, then execs your actual command. Every container really does start at once; all but the first are sitting in a loop watching a file.

This is worth knowing for three reasons. First, it explains why a Step that kills the entrypoint process, or writes the wrong marker file, does strange things to the ordering — the sequencing is cooperative, not enforced by the kubelet. Second, it explains where results and step state live: under a Tekton-owned directory inside the Pod, on volumes every step can see. Third, it means that a step running untrusted code is a peer of the mechanism that sequences the trusted steps, not a subordinate of it. The exact paths under that tree have moved between Tekton versions, so check your version's docs rather than hard-coding what a blog post told you.

Workspaces, PVCs, and the affinity assistant

A workspace is Tekton's abstraction for "a directory that shows up in my steps". The Task declares it by name; the TaskRun or PipelineRun binds it to something concrete — an emptyDir, a PersistentVolumeClaim, a volumeClaimTemplate that creates a PVC for the run's lifetime, a ConfigMap, a Secret, or a CSI volume. It is mounted into every step of the Task at a path you can reference as a variable.

In a Pipeline, a workspace backed by a PVC is how one Task hands work to the next: clone into it, build from it, publish out of it. That is enormously convenient and it is also why Tekton ships the affinity assistant, a placeholder Pod that pins the TaskRuns sharing a workspace onto one node, because a ReadWriteOnce volume cannot be mounted by pods on two nodes at once. The scheduling behaviour is configurable and the flag names have changed over releases — verify against your install — but the shape is fixed: a shared PVC is a shared, writable, cross-Task filesystem, and every step of every Task bound to it can write anywhere in it.

• One TaskRun, one Pod. Steps are containers in it, run in order by a cooperating entrypoint binary, sharing the Pod's volumes and network namespace.
• The service account is per-Pod, not per-Step. Every step in the Task runs under the same identity, with the same projected token, the same imagePullSecrets and the same credential-initialiser output.
• Workspaces are shared and usually writable. A PVC workspace is a filesystem that persists across Tasks in the Pipeline; there is no per-step view of it.
• Results are files. A step writes a value to a path under Tekton's results directory, and the controller lifts it into the TaskRun's status where later Tasks can reference it.
• Sidecars run in parallel for the whole Task. A sidecar Docker daemon is reachable by every step, including the one you did not write.
• Kubernetes primitives still apply. securityContext, resource limits, network policy, seccomp, runtime classes — Tekton mostly gets out of the way and lets you set them. That is the good news in this post.

What one step can reach

Now put a fork pull request's build script into one of those step containers and take an inventory of what it is standing next to. None of this is a Tekton defect; it is the direct, honest consequence of Pod-shaped execution, and every Kubernetes-native CI system inherits some version of it.

• The node's kernel. Every container on that node — your steps, other teams' steps, whatever else the cluster scheduled there — is making syscalls into one Linux kernel. Seccomp and AppArmor shrink that surface; they do not replace it. A container boundary is, on a bad day, a polite suggestion to the kernel.
• The Pod's service account token. Unless you turned automounting off, there is a projected token on disk in every step container, and it is the identity Tekton itself uses to do things like create PVCs. Whatever RBAC you granted it, the untrusted step has it too.
• The cloud identity attached to that node or pod. IRSA, workload identity, an instance profile reachable at the metadata endpoint. This is usually the biggest prize in the room and the one people forget is in the room at all.
• The whole workspace. Not just the checkout — the shared PVC that the clone Task wrote to and that the publish Task will read from. A step that can write there can influence what a later, trusted step builds and ships.
• Every other step's results directory. Results are files in a shared location. An untrusted step that writes a plausible value into a later step's result file has just influenced a Pipeline's when expressions and downstream parameters.
• Credentials materialised by the credential initialiser. Secrets annotated onto the service account are turned into a git credentials file and a Docker config inside the step containers' home directory. That is the feature working as designed — it is how git-clone Tasks authenticate — and it means registry push credentials are sitting in the same filesystem as the untrusted build.
• Any sidecar in the Task. Including, memorably, a Docker daemon someone added so that the image build would work.

Steps are containers in one Pod. That is the design. It means the isolation between your trusted publish step and someone else's build script is the same isolation you would get between two processes you wrote yourself — which is to say, an organisational assumption wearing a technical costume.

Why privileged steps show up anyway

You can harden a Tekton step quite far with plain Kubernetes: runAsNonRoot, a non-zero runAsUser, allowPrivilegeEscalation false, all capabilities dropped, a read-only root filesystem, the runtime default seccomp profile. Do all of that and the container is a genuinely unpleasant place to be an attacker. Then someone adds a Task that builds a container image, and the hardening quietly comes off.

The reason is mechanical. Building an OCI image means unpacking layer filesystems, which means creating device nodes, setting ownership across UIDs, and generally doing the things that root-in-a-namespace was invented to allow and that the hardened profile above forbids. Kaniko's whole design is to unpack image layers into its own root filesystem, so it wants to run as root inside its container and it explicitly asks not to be run on a machine you care about. BuildKit's daemon has a rootless mode that leans on user namespaces and needs the cluster to be configured for them, plus the right seccomp and AppArmor posture; the non-rootless path is usually run privileged. Anything that talks to a mounted Docker socket has effectively been handed root on the node, because a process that can create containers can create one with the host filesystem bind-mounted.

So the step that looked like this in the design doc:

# The step everybody writes first -- and the reason this post exists.
# Task .spec.steps[]
- name: build-image
  # Unpinned tag: whatever is in the registry the morning your build runs.
  image: gcr.io/kaniko-project/executor:latest
  args:
    - --dockerfile=$(params.dockerfile)
    - --context=$(workspaces.source.path)
    - --destination=$(params.image)
  securityContext:
    # Kaniko unpacks image layers into its own root filesystem, so it wants
    # to be root. That part is inherent to how it works.
    runAsUser: 0
    # This part is not inherent. It was added on the Tuesday a build failed
    # with a permissions error, it made the build pass, and nobody removed
    # it. It also removes most of the boundary the rest of the file assumes.
    privileged: true
  volumeMounts:
    # Push credentials, mounted into the container that executes a Dockerfile
    # supplied by whoever opened the pull request.
    - name: docker-config
      mountPath: /kaniko/.docker

# Verify the current requirements of whichever builder you use against its own
# docs -- kaniko, BuildKit rootless and Buildah all have different and moving
# stories about what privileges they actually need.

…ends up shipping with a line in it that undoes the isolation everything else in the file was carefully establishing. And I want to be fair here: nobody added that line out of carelessness. They added it because the alternative was a two-week detour into user namespace configuration on a cluster they do not own, and the build needed to be green today. This is the most common way a Kubernetes CI setup's security posture degrades — not a decision, an accumulation.

The catalog is code you pull

One more piece of the picture, because it is the part that surprises people who thought their exposure was limited to their own repositories. Tekton's remote resolvers let a taskRef point somewhere other than your cluster: an OCI bundle in a registry, a file in a git repository, an entry in a catalog. This is a great feature. It is how you stop copy-pasting the same git-clone Task into forty namespaces.

It is also a supply chain. A resolved Task is a container image reference plus a script, and when your PipelineRun executes it, that script runs in your Pod, under your service account, with your workspace mounted and your credential-initialiser output on disk. A Task that reads a directory it has no business reading looks exactly like a Task that does not. The mitigations are the ordinary supply-chain ones and they are worth doing on the day you set this up rather than the day after an incident: pin bundles by digest rather than tag, mirror the catalog Tasks you use into a registry you control, review them like dependencies because that is what they are, and use Tekton Chains if you want signed provenance for what your Pipelines produced.

The pattern: Tekton orchestrates, a microVM executes

Here is the shape I want to argue for. Do not move off Tekton. Do not try to make the Pod boundary into something it was never designed to be. Instead, change what one specific step does.

The untrusted step stops being a container that runs the repository's build. It becomes a driver: a small, pinned, hardened image whose entire job is to create a fresh microVM, push the checkout into it, run the build command inside it, stream the logs back so they land in the TaskRun's logs exactly like any other step, extract a small number of validated results, and delete the machine. The driver executes nothing that came out of the repository. The guest executes nothing that came out of your cluster.

That inversion is the whole trick, and it has a pleasing property: everything Tekton is good at stays. The Pipeline is still a Pipeline. Results still flow through when expressions. The Dashboard still shows step logs. RBAC, GitOps, admission control, Chains provenance, retries, timeouts, finally blocks — all unchanged, because from Tekton's point of view this is just a Task whose step happens to make network calls. What changed is that the arbitrary code moved across a hypervisor boundary onto a kernel that is not your node's kernel.

The reason this is affordable is that creating the guest is a snapshot restore, not a boot. You bake a template once with your toolchains and caches already inside, snapshot it warm, and every create restores that frozen machine. On PandaStack that path runs at roughly 179ms at p50 and 203ms at p99; the genuine cold boot of about three seconds happens once, at bake time. Each guest gets its own network namespace out of a pool of pre-allocated /30 subnets, so egress policy is enforced on a real segment rather than by asking the build nicely. Memory is copy-on-write and the rootfs is a reflink clone, so the hundredth guest of the day does not copy gigabytes to exist. A TTL set at create time means a guest your driver forgot about deletes itself.

The Task that calls a sandbox

The Task below is deliberately boring, which is the point. It declares what crosses the boundary in each direction — params in, two results out, one read-only workspace — and hardens the driver container as far as Kubernetes will let it, because the driver genuinely does not need any of the privileges it is giving up.

apiVersion: tekton.dev/v1
kind: Task
metadata:
  name: sandboxed-build
spec:
  description: >-
    Runs an untrusted build inside a fresh Firecracker microVM. This Pod is a
    driver: it holds credentials and executes none of the repository's code.
  params:
    - name: build-command
      type: string
      default: "make ci"
    - name: template
      type: string
      default: "base"
  workspaces:
    - name: source
      description: A previous Task cloned here. This Task only reads it.
      readOnly: true
  results:
    - name: exit-code
      description: Exit status of the build command inside the guest.
    - name: artifact-digest
      description: Digest the guest reported, re-validated by the driver.
  stepTemplate:
    securityContext:
      # The driver makes HTTPS calls and reads a directory. It needs nothing
      # else, so it gets nothing else -- and unlike a build step, it will
      # never be handed a workload that suddenly requires more.
      runAsNonRoot: true
      runAsUser: 65532
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities:
        drop: ["ALL"]
      seccompProfile:
        type: RuntimeDefault
  steps:
    - name: drive-sandbox
      # Pinned by digest. This image is the ONLY code that runs in this Pod.
      image: ghcr.io/acme/pandastack-driver@sha256:3f9c0d1e...  # truncated
      env:
        - name: PANDASTACK_API_KEY
          valueFrom:
            secretKeyRef:
              # Mounted as an env var on THIS step only. Deliberately not
              # annotated onto the ServiceAccount, so the credential
              # initialiser never materialises it into a shared home dir.
              name: pandastack-ci
              key: api-key
        - name: BUILD_COMMAND
          value: $(params.build-command)
        - name: TEMPLATE
          value: $(params.template)
        - name: WORKSPACE_PATH
          value: $(workspaces.source.path)
        - name: TASKRUN_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.labels['tekton.dev/taskRun']
      computeResources:
        # A driver, not a builder. If this Pod is using a core, something is
        # running here that should be running in the guest.
        requests:
          cpu: "100m"
          memory: "128Mi"
        limits:
          cpu: "500m"
          memory: "256Mi"
      script: |
        #!/usr/bin/env sh
        set -eu
        exec /usr/local/bin/drive-sandbox

Note what is absent. No sidecar. No Docker socket. No privileged flag, and no plausible future request for one, because the container that would have needed it is not in this Pod any more. The image build, if there is one, happens inside the guest with an ordinary unprivileged daemon, because the isolation boundary there is the virtual machine rather than the container — there is no nesting problem to solve because nothing is nested.

The step body

And here is the driver. It is short on purpose: every line of logic in this file is logic that runs next to your credentials, so there should be as few as possible, and none of it should be conditional on anything the guest said.

#!/usr/bin/env python3
"""Tekton step body. Runs in the TaskRun Pod and executes nothing it is given.

The Pod holds credentials. The guest holds the untrusted code. They never meet:
the only things that cross are a tarball going out, and log text plus two
validated strings coming back.
"""
import io
import json
import os
import pathlib
import sys
import tarfile

from pandastack import Sandbox

WORKSPACE = pathlib.Path(os.environ["WORKSPACE_PATH"])
RESULTS = pathlib.Path("/tekton/results")  # confirm for your Tekton version
BUILD_COMMAND = os.environ["BUILD_COMMAND"]
DIGEST_LEN = 71  # len("sha256:") + 64 hex chars


def pack(src: pathlib.Path) -> bytes:
    """Only the checkout crosses the boundary -- not the whole shared PVC,
    and not the .git directory with its credential helper config."""
    buf = io.BytesIO()
    with tarfile.open(fileobj=buf, mode="w:gz") as tar:
        tar.add(
            src,
            arcname="src",
            filter=lambda ti: None if "/.git/" in ti.name else ti,
        )
    return buf.getvalue()


def main() -> int:
    sbx = Sandbox.create(
        template=os.environ["TEMPLATE"],
        # Backstop. If this Pod is OOM-killed, preempted or evicted between
        # create and delete, the guest still reaps itself. The finally block
        # below is the happy path; this is the one that survives our bugs.
        ttl_seconds=1800,
        metadata={
            "tekton_taskrun": os.environ["TASKRUN_NAME"],
            "trust": "untrusted",
        },
    )
    try:
        sbx.filesystem.write("/work/src.tar.gz", pack(WORKSPACE))
        sbx.exec("mkdir -p /work && tar -xzf /work/src.tar.gz -C /work")

        # The untrusted part. It runs on a guest kernel, in its own network
        # namespace, as a user with no route back to this Pod, this node, or
        # the cluster's API server. There is no service account token in
        # there, no kubeconfig, no instance metadata endpoint to curl.
        result = sbx.exec(
            f"cd /work/src && {BUILD_COMMAND}",
            timeout_seconds=1500,  # shorter than the TaskRun timeout, on purpose
        )

        # Stream it into the step log so it shows up in the Tekton Dashboard,
        # `tkn taskrun logs`, and whatever ships your pod logs. As far as the
        # rest of your platform is concerned this was a normal step.
        sys.stdout.write(result.stdout)
        sys.stderr.write(result.stderr)

        RESULTS.joinpath("exit-code").write_text(str(result.exit_code))

        if result.exit_code == 0:
            # Results are values WE compute or validate -- never a blob the
            # guest handed us that we forward into a `when` expression.
            meta = json.loads(sbx.filesystem.read("/work/src/build-meta.json"))
            digest = str(meta.get("digest", ""))
            if not digest.startswith("sha256:") or len(digest) != DIGEST_LEN:
                raise ValueError(f"guest returned a malformed digest: {digest!r}")
            RESULTS.joinpath("artifact-digest").write_text(digest)

        return 0 if result.exit_code == 0 else 1
    finally:
        # Filesystem, package caches, forked daemons, anything the build tried
        # to plant, and the kernel it planted it on all cease to exist at the
        # same instant. There is no cleanup script to get subtly wrong.
        sbx.kill()


if __name__ == "__main__":
    sys.exit(main())

Three details are load-bearing. The exec timeout is shorter than the TaskRun timeout so that a hanging build is killed by the thing holding the sandbox handle rather than by Kubernetes killing the driver mid-teardown. The TTL on create is an independent backstop that does not depend on the driver being alive to work. And the digest is validated before it is written to a result file, because a result is not a log line — it feeds a when expression that decides whether the trusted publish Task runs.

How workspaces, results and artifacts cross the boundary

This is where most designs of this shape either work or quietly become a data pipe with a security theme. Three separate flows, three different answers.

Keeping the results contract intact

Tekton results are small strings a step writes to files, which the controller lifts into the TaskRun status so a Pipeline can wire them into a later Task's params or a when expression. They are surfaced through a mechanism with a real size limit — historically the container's termination message, which Kubernetes caps at a few kilobytes across a container's results — and newer Tekton versions add larger results through a different transport behind a feature flag. Check what your version supports; the important thing either way is that results are for values, not payloads.

The offload pattern preserves the contract exactly, because the driver is a normal step writing normal result files. What changes is who computes them. The guest reports; the driver validates and writes. Keep results to things like an exit code, a digest, a version string, a count. If a downstream Task needs more than that, it needs a file, and a file is not a result.

Artifacts should not go through the Pod

The tempting design is to pull the build output back into the workspace so the rest of the Pipeline can consume it as usual. Resist it for anything large. You would be moving gigabytes out of a guest, through a driver Pod with a 256Mi limit, onto a ReadWriteOnce PVC that is pinning your TaskRuns to one node — three constraints, all of them yours, none of them helping.

Push directly from the guest to object storage instead, using a short-lived, tightly scoped credential the driver mints and injects at create time — write-only, prefixed to that run, expiring in minutes. Then the only thing that comes back through the Pod is the object key and its digest, as results. The trusted publish Task fetches by digest with its own credentials. This is more moving parts on a diagram and considerably fewer on the day something goes wrong, because the untrusted build never held a credential that could overwrite anything but its own prefix.

Which direction the workspace flows

Bind the workspace read-only to the sandboxed Task and push a tarball of just the checkout, not the workspace root. A shared PVC accumulates things across a Pipeline — a previous Task's output, cached dependencies, whatever a catalog Task decided to leave there — and none of that needs to be inside a guest running someone else's code. Excluding the .git directory is worth doing specifically: it carries remote URLs and, if the credential initialiser has been at work, a credential helper configuration you do not want travelling.

Secrets: the sandbox never gets the pipeline's identity

The clearest way to state the security win is as a list of things that are simply not present inside the guest. There is no projected service account token, so there is no path to the Kubernetes API even if RBAC were generous. There is no node metadata endpoint, because the guest's network namespace does not route to one. There is no imagePullSecret, no credential-initialiser output, no kubeconfig, and no PandaStack API key — the driver has that, and the driver is in a different machine.

Getting there takes three deliberate choices, and the first is the one people skip. Give untrusted Pipelines their own service account with automountServiceAccountToken set to false, and nothing annotated onto it. Mount the driver's own credential as a secret env var on that one step rather than annotating it onto the account, so the credential initialiser never writes it into a home directory that a future step might share. And run the publish Task under a different service account entirely, so the identity that can push to your registry is never in the same Pod as the code that produced the thing being pushed.

That last one is a Tekton feature rather than something you have to build: a PipelineRun can set a default service account for its TaskRuns and then override it per Pipeline Task. Which brings us to the Pipeline itself.

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: tekton-untrusted
# The single highest-value line in this file. Most CI steps never call the
# Kubernetes API; a token they can read is pure downside. (Tekton also exposes
# this on the pod template -- check which your version honours.)
automountServiceAccountToken: false
---
apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
  name: fork-pr
spec:
  params:
    - name: repo-url
    - name: revision
  workspaces:
    - name: source
  tasks:
    - name: clone
      taskRef:
        # Catalog Tasks are dependencies. Pin the bundle by digest and mirror
        # it into a registry you control; a tag is a promise someone else
        # can rewrite while you are asleep.
        resolver: bundles
        params:
          - name: bundle
            value: ghcr.io/acme/catalog/git-clone@sha256:9b1e77c2...
          - name: name
            value: git-clone
          - name: kind
            value: task
      params:
        - name: url
          value: $(params.repo-url)
        - name: revision
          value: $(params.revision)
      workspaces:
        - name: output
          workspace: source

    - name: build
      runAfter: ["clone"]
      taskRef:
        name: sandboxed-build
      # A hostile build cannot make this Task run forever, and this bound is
      # longer than the driver's own exec timeout so the driver gets to clean
      # up rather than being killed halfway through it.
      timeout: "30m"
      params:
        - name: build-command
          value: "make ci"
      workspaces:
        - name: source
          workspace: source

    - name: publish
      runAfter: ["build"]
      # Only on a clean exit, and only ever with a digest -- never with the
      # tree the untrusted build produced.
      when:
        - input: "$(tasks.build.results.exit-code)"
          operator: in
          values: ["0"]
      taskRef:
        name: sign-and-publish
      params:
        - name: digest
          value: "$(tasks.build.results.artifact-digest)"

  finally:
    # Runs whether the Pipeline succeeded, failed, timed out or was cancelled.
    # Belt and braces with the create-time TTL: this is the fast path, the TTL
    # is the one that works when this Pod never got to run.
    - name: reap-orphans
      taskRef:
        name: pandastack-reap
      params:
        - name: pipelinerun
          value: "$(context.pipelineRun.name)"
---
apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  generateName: fork-pr-
spec:
  pipelineRef:
    name: fork-pr
  taskRunTemplate:
    # Default identity for every TaskRun: no token, no annotated secrets.
    serviceAccountName: tekton-untrusted
  taskRunSpecs:
    # ...except the publish Task, which is the only place registry credentials
    # exist, and which never shares a Pod with untrusted code.
    - pipelineTaskName: publish
      serviceAccountName: tekton-publisher
  workspaces:
    - name: source
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 5Gi

The finally block deserves a word. Tekton runs it whether the Pipeline succeeded, failed, timed out or was cancelled, which makes it the natural place for a reaper that deletes any sandbox tagged with this PipelineRun's name. Do it anyway even though you set a TTL, and set the TTL anyway even though you have a finally block. Two independent stop conditions, neither depending on the other being correct, is the whole design philosophy of the thing you are building.

Step in a Pod vs step that shells out vs a per-job VM runner

Side by side, with the standard caveat: everything about Tekton's own behaviour here is version-dependent and configuration-dependent, so verify specifics against the current documentation. Only the PandaStack numbers are measured.

• Tekton Step in the Pod (the default) — Isolation: container over the node's shared kernel; the untrusted step sits beside the Pod's service account token, the workspace PVC, the results directory, any sidecar and the node's cloud identity. Image builds pull you toward privileged or a mounted socket, which collapses the boundary entirely. Ergonomics: perfect — one Task, one Pod, everything native, kubectl debuggable. Ops cost: none beyond the cluster you already run. Best for: first-party builds by employees on repositories you control, which is most builds.
• Kata or gVisor runtime class under the same Pod (worth naming) — Isolation: a real upgrade without changing a single Pipeline definition; gVisor interposes a user-space kernel on the syscall path, Kata runs the Pod inside a lightweight VM. Ergonomics: you set a runtimeClassName and Tekton keeps working. Ops cost: node configuration, a compatibility tail for exotic syscalls, and it is the whole Pod that moves — trusted and untrusted steps together, still sharing the workspace and the service account. Best for: the cheapest genuine boundary if you are already on Kubernetes and can change node config.
• Tekton step that shells out to a microVM — Isolation: hardware-enforced; the build runs on a different kernel under KVM, in its own network namespace, with no service account token, no metadata endpoint and no registry credential in reach. The driver Pod holds the secrets and runs none of the repository's code. Ergonomics: Pipelines, results, when expressions, retries, finally, Dashboard logs and Chains all keep working, because Tekton still sees a normal step. Latency: about 179ms p50 / 203ms p99 to create the guest on PandaStack, against a Pod that already spent seconds scheduling. Ops cost: a driver image you own and keep current, hosts with /dev/kvm, a template bake pipeline, and a decision about how artifacts leave the guest. Best for: fork PRs, customer-supplied build configuration, agent-generated branches, image builds without privilege, and any Task where the answer to 'who wrote this code' is 'we do not know'.
• A dedicated per-job VM runner outside Tekton — Isolation: equally strong; a whole machine per job, terminated after. Ergonomics: you leave the Kubernetes-native model behind for those jobs — separate control plane, separate scheduling, separate log path, separate RBAC story, and results no longer flow through your Pipelines. Latency: instance provision plus boot, which is why teams keep a warm pool and then start reusing machines to hide the wait, quietly giving the freshness back. Ops cost: a second CI system to operate and reconcile with the first. Best for: teams already running a VM-based fleet for other reasons, or workloads that were never a good fit for Pod-shaped execution to begin with.

When the extra hop is not worth it

I would rather you skip this pattern than adopt it everywhere, because adopting it everywhere is how it acquires a bad reputation. It is a boundary with a cost, and the cost is only worth paying where a boundary is actually load-bearing.

• Trusted first-party builds. Employees, protected branches, code that went through review. The Pod boundary is proportionate here, and adding a hop buys you an extra failure mode and an extra thing to page someone about.
• Steps that are supposed to touch the cluster. Deploy Tasks, kubectl apply, Helm releases, anything that legitimately needs the service account token. Moving those into a guest means shipping the credential in there, which is the exact opposite of the point.
• Very short steps. A lint pass that takes four seconds does not want a create and a delete around it. Group the cheap checks into one sandboxed Task rather than one per check, or leave them in the Pod and sandbox only the part that executes repository code.
• Large artifacts you have not solved yet. If you have not built the object-storage path and your plan is to pipe six gigabytes back through a driver Pod, build that path first. This design is much worse than the Pod version until artifacts stop travelling through the middle.
• Clusters where nobody will own the driver image. This is an integration you maintain — a pinned image, an SDK version, a template bake pipeline, a credential rotation. If there is no owner, it will rot into a stale image with a two-year-old CVE list, and you will have added complexity in exchange for a boundary you stopped trusting.
• When a runtime class gets you far enough. If Kata or gVisor is available on your nodes and your real threat model is 'a dependency's postinstall script does something stupid' rather than a targeted attacker, set runtimeClassName and go home. It is a smaller change and it is a genuine boundary.

What to move first

The good news about Tekton specifically is that this is a per-Task decision, not a migration. Nothing about the pattern requires you to change your other Pipelines, your Dashboard, your GitOps repo, or the way anyone on your team works. You add one Task to the cluster and change one taskRef.

1. Start with the Pipeline that builds fork pull requests, if you have one. This is where the argument is not a preference — the population of people who can execute code in your cluster is currently the population of people with a source-forge account.
2. Then the image-build Task, because it is the one carrying privileged or a mounted socket, and because moving it into a guest lets you run an ordinary unprivileged daemon and delete that line for good.
3. Then anything driven by a model. An agent that opens a branch, runs the tests, reads the failure and tries again is running unreviewed code in a loop, and 'the tests passed' is not a statement about intent. The generated rm -rf is not malice; it is a plausible next token, and a shared node kernel is a bad place to find out.
4. Turn off automountServiceAccountToken for those Pipelines on the same day, whether or not the rest lands. It is one line and it removes the most valuable thing in the Pod.
5. Leave everything else exactly where it is. Your trusted builds are fine in a Pod. They were fine yesterday and this post did not change that.

The end state is deliberately unremarkable: the same cluster, the same Tekton install, the same Pipeline definitions with one Task swapped, the same logs in the same Dashboard. The only thing that changed is that when a stranger's build script runs, it runs on a kernel that was created a fraction of a second earlier and is destroyed a few minutes later — and the machine holding your registry credentials was never the machine running their code.