Per-Tenant ETL Pipeline Isolation With MicroVMs

Every data platform eventually ships "custom transforms." Customers want to define their own SQL, drop in a Python script to reshape a column, write a Jinja-templated model, or wire up a connector that parses their weird CSV dialect. It's the feature that turns a rigid product into a platform. It's also the feature that quietly turns your ETL workers into a shared execution host for code written by everyone you've ever onboarded.

Here's the shape of the problem. Your orchestrator — Airflow, Dagster, a Spark cluster, your own scheduler — pulls a job off the queue and runs it in a worker. That worker has the warehouse credentials, the connection strings, the object-storage tokens it needs to move data. When the job is "run tenant B's transform," tenant B's arbitrary Python now executes in a process that can reach tenant A's tables. Nobody designed a data-exfiltration path; one emerged from running untrusted code next to trusted credentials.

I'm Ajay; I built PandaStack. This post is about the fix: one Firecracker microVM per pipeline run, with that tenant's credentials — and only that tenant's — scoped into the guest, a hard TTL so a runaway transform can't run all night, and an immutable baseline so run N+1 isn't polluted by run N. Let me walk through the failure modes, then the model.

A custom transform is untrusted code with your keys in scope

It's tempting to think of a transform as a pure function — rows in, rows out. In practice it's a script, and a script does whatever it wants. The moment you let a tenant supply the transform, you've accepted arbitrary code execution; the only question left is what that code can reach.

• Custom Python/Jinja is arbitrary code — a transform can `import os`, open sockets, read environment variables, and enumerate the filesystem. If your warehouse password is in the worker's env, the transform can read it and POST it out.
• "Just SQL" isn't just SQL — user-defined functions, database links, `COPY ... PROGRAM`, and extension hooks all reach beyond the query. And a transform that builds SQL from tenant input is an injection surface into whatever connection the worker holds.
• Connectors parse hostile formats — a connector reading tenant-supplied CSV, Parquet, or JSON runs a parser against attacker-controlled bytes, with the parser's own bug surface.
• Credentials are the crown jewels — ETL workers are, by design, the most credential-rich machines you own. They hold the keys to every source and sink. That's the room you're inviting arbitrary transforms into.
• A runaway job is a DoS — a transform that writes an accidental cartesian join, or allocates until the OOM killer wakes up, starves every other tenant's job sharing that worker. No malice required, just a bad query.

The model: one microVM per pipeline run

The structural fix is to stop running tenant transforms in your trusted orchestrator process. Keep the orchestrator — the part that holds the master credentials and decides what runs — on a trusted host. For each pipeline run, spin up a Firecracker microVM, inject only the credentials that run needs (scoped, short-lived, that tenant's), run the transform inside the guest, collect the output, and destroy the VM. The worker that touches tenant code holds tenant-scoped secrets and nothing else, and it ceases to exist when the run ends.

This works because create is cheap. A cold Firecracker boot is a couple of seconds, but PandaStack creates sandboxes by restoring a baked snapshot on demand — the restore step is around 49ms, and end-to-end create is p50 179ms (p99 ~203ms). A hardware-isolated worker per run costs a fifth of a second. With 16,384 pre-allocated network slots per agent, running hundreds of tenant pipelines concurrently is bounded by CPU and memory, not by isolation overhead.

from pandastack import Sandbox

def run_pipeline(tenant_id: str, transform_code: str, scoped_creds: dict) -> dict:
    # A fresh guest for THIS run. It gets ONLY this tenant's scoped creds.
    sbx = Sandbox.create(
        template="code-interpreter",
        ttl_seconds=1800,          # a runaway transform can't run all night
        metadata={"tenant": tenant_id, "kind": "etl-run"},
    )
    try:
        # Inject the untrusted transform and a NARROW, per-run credential set.
        # The orchestrator's master keys never enter this guest.
        sbx.filesystem.write("/work/transform.py", transform_code)
        sbx.filesystem.write("/work/creds.json", json_dumps(scoped_creds))

        # Run the transform inside the guest. If it tries to read a secret it
        # wasn't given, there's nothing there to read.
        out = sbx.exec(
            "cd /work && python3 run_transform.py transform.py creds.json",
            timeout_seconds=1500,
        )
        return {"ok": out.exit_code == 0, "logs": out.stdout}
    finally:
        sbx.delete()   # the worker, and the scoped creds, disappear with it

The credential-scoping is the whole game. In a shared worker, "least privilege" is aspirational — the worker needs the union of every tenant's access. In a per-run microVM, the guest gets exactly the source and sink credentials for this one run, ideally short-lived tokens minted just-in-time. A malicious transform in tenant B's guest can enumerate the environment all it likes; tenant A's warehouse password was never there.

Reproducibility and parallel partitions

Isolation is the reason to reach for microVMs. Reproducibility and parallelism are the reasons to stay. Two properties fall out of the per-run model for free.

An immutable baseline per run

Shared workers accumulate state — cached packages, leftover temp files, a global install from three jobs ago — so run N+1 is subtly not run N. That's how a pipeline "works on the retry" and no one can explain why. With microVMs, every run forks the same immutable baseline snapshot: the dependencies your platform installed, at the versions you pinned, in a byte-identical state. Reproducibility stops being a hope and becomes a property of how runs start.

Fork per partition for parallelism

When a transform runs over a partitioned dataset, you can fan out: snapshot the "deps installed, transform loaded" baseline once, then fork it per partition. A same-host fork lands in roughly 400–750ms and shares the baseline's disk copy-on-write, so ten partitions don't cost ten fresh installs — they share the baseline's blocks until each writes. (Cross-host forks are 1.2–3.5s.) Each partition runs in its own guest with its own memory budget, so a bad partition can't starve the others.

# Build the trusted baseline ONCE: install deps + stage the transform.
base = Sandbox.create(template="code-interpreter", ttl_seconds=1800)
base.exec("pip install pandas pyarrow duckdb", timeout_seconds=600)
base.filesystem.write("/work/transform.py", transform_code)
baseline = base.snapshot()
base.hibernate()

# Fan out one fork per partition -- each in its own hardware-isolated guest.
for part in partitions:                 # e.g. date shards, key ranges
    with baseline.fork(ttl_seconds=900) as sbx:   # ~400-750ms same-host
        sbx.filesystem.write("/work/part.json", json_dumps(part))
        sbx.exec("cd /work && python3 transform.py part.json",
                 timeout_seconds=800)
    # fork destroyed here; the baseline is untouched, ready to re-fork

Shared worker vs container-per-run vs microVM-per-run

• Isolation strength — Shared Airflow/Spark worker: none; every tenant's transform runs in a process holding the union of all credentials. Container per run: shares the host kernel, so an escape or kernel bug reaches the host and neighbors. MicroVM per run: hardware-virtualized guest with its own kernel — a compromised transform stays in one disposable VM.
• Credential blast radius — Shared worker: the worker holds every source and sink key; least privilege is impossible. Container: env-injected secrets still sit next to shared-kernel escape risk. MicroVM: only this run's scoped, short-lived credentials enter the guest; other tenants' keys were never present.
• Reproducibility — Shared worker: state leaks between runs (caches, global installs), so retries diverge. Container: better, but reused images and layer caches drift. MicroVM: every run forks the same immutable baseline snapshot — byte-identical starting state.
• Runaway jobs — Shared worker: one cartesian join or memory hog starves all tenants. Container: cgroup limits help, kernel is shared. MicroVM: fixed vCPU/RAM per guest plus ttl_seconds; a runaway saturates only itself and is reclaimed.

The transform still does its job at full speed — pandas is pandas, DuckDB is DuckDB. What changes is that the untrusted part runs in a guest scoped to one tenant's credentials and destroyed at the end of the run, while your orchestrator holds the master keys on a trusted host it never lets a transform touch. At ~179ms per create and ~400ms per fork, per-tenant isolation costs you almost nothing and buys you a data platform where a customer's transform can't read another customer's rows.

For adjacent patterns: log-processing pipelines get the same treatment in /blog/microvm-per-tenant-log-processing-isolation, database-level tenant isolation is in /blog/per-tenant-database-isolation, and the agent-driven data-pipeline case is in /blog/ai-agent-data-pipeline-sandbox.