Per-Tenant Analytics Queries in Isolated microVMs

You built an analytics SaaS, and the feature everyone asked for is the one that scares you: let each customer write their own transforms. A SQL model, a pandas cleanup step, a dbt-style derived table, a custom scoring UDF, a whole notebook — against their data, in your platform. It's a great feature. It's also the moment your product stops running only code you wrote. Now you run code your customer wrote, or worse, code your customer's end-user wrote, and you run it next to every other tenant's data.

That code has two properties that don't mix well with a shared worker pool. It's untrusted — you didn't review it, you can't review it at the rate customers submit it, and a valid-looking transform can read an environment variable or open a socket just as easily as it can group by region. And it's expensive in ways you can't predict: a forgotten join condition becomes a cartesian product, a `df = pd.read_parquet(...)` becomes a 40GB allocation, a `while True` becomes a pinned core. The failure isn't malice. It's Tuesday.

I'm Ajay — I build PandaStack, a Firecracker microVM platform, so I spend my days thinking about exactly where the isolation boundary sits when you run other people's code. This post is about running per-tenant analytics queries and notebooks the honest way: one microVM per tenant, or per run, so a runaway query or a nosy transform is contained to one disposable machine instead of everyone's afternoon.

The shared worker pool is a noisy-neighbor and data-leak disaster

The tempting first design is a pool of warm workers — processes or containers — that pull tenant jobs off a queue. It's cheap, it's warm, it scales horizontally. It also fuses every tenant's fate together in two ways, and both of them will find you.

The first is the noisy neighbor. Tenant A submits a transform with a join that fans out to a cartesian product, or a pandas step that tries to materialize a hundred million rows in memory. That worker's CPU pins and its RSS balloons. On a shared box, that pressure doesn't stay politely inside tenant A's job — page cache thrashes, the OOM killer starts making decisions on everyone's behalf, and tenant B's perfectly reasonable dashboard query, which happened to land on the same worker, times out. One customer's `SELECT *` cross join becomes every customer's incident, and your on-call gets paged for a query they didn't write and can't see.

The second is worse, because it's silent. That worker process has, at some point, held tenant A's data in memory and tenant B's data in memory. It has your platform's credentials in its environment — the warehouse connection, the object-store keys, the API tokens. Tenant-supplied code running in that process can read all of it: another tenant's dataframe still resident in the heap, a temp file left in `/tmp`, the environment variable holding your master DSN. A shared kernel makes it worse still — a container escape or kernel bug turns "tenant A's transform" into "root on the host that runs everybody." You didn't leak the data. Your architecture did.

One microVM per tenant, or per run

The fix is to stop sharing the thing you shouldn't share: the machine. Run each tenant's transform in its own Firecracker microVM. Each VM boots its own guest kernel and is confined by KVM hardware virtualization — the only way out is a tiny set of emulated virtio devices, not the full Linux syscall surface a container sees. This is the same isolation model AWS Lambda and Fargate use to run untrusted code from thousands of customers on shared fleets. Tenant A's runaway cartesian join, or their attempt to read the filesystem, is contained to one VM. Its memory, its filesystem, its network are its own. When it misbehaves, you kill one machine and nobody else notices.

The classic objection is startup cost — nobody wants a three-second VM boot on the front of every query. PandaStack sidesteps that with snapshot-restore: every create restores a baked snapshot of an already-booted machine rather than cold-booting. The restore step is around 49ms; an end-to-end create is p50 179ms and p99 about 203ms. A true cold boot, only on the very first spawn of a template, is around 3s — after that, you're paying restore prices. That's the trade that makes a VM-per-query practical: hypervisor-grade isolation at roughly container-grade latency.

Here's the shape in the Python SDK — run one tenant's query in its own sandbox, give it only that tenant's data, cap it with a timeout, read the result back, and destroy the VM:

from pandastack import Sandbox

def run_tenant_query(tenant_id: str, user_sql: str, scoped_data_csv: str) -> dict:
    """Run ONE tenant's user-defined query in its own microVM, then destroy it.
    The VM never sees another tenant's data or our platform secrets."""
    with Sandbox.create(
        template="code-interpreter",
        ttl_seconds=300,                       # backstop: a leaked VM reaps itself
        metadata={"tenant_id": tenant_id},     # for per-tenant audit + billing
    ) as sbx:
        # Inject ONLY this tenant's data into the guest. No warehouse creds,
        # no platform env — the query runs against a local, scoped extract.
        sbx.filesystem.write("/workspace/data.csv", scoped_data_csv)

        # The customer's transform. We never reviewed it and never trust it.
        runner = (
            "import duckdb, json\n"
            "con = duckdb.connect()\n"
            "con.execute(\"CREATE TABLE t AS SELECT * FROM '/workspace/data.csv'\")\n"
            "rows = con.execute(open('/workspace/query.sql').read()).fetchdf()\n"
            "rows.to_json('/workspace/result.json', orient='records')\n"
        )
        sbx.filesystem.write("/workspace/query.sql", user_sql)
        sbx.filesystem.write("/workspace/run.py", runner)

        # timeout_seconds is the circuit breaker for the infinite loop /
        # cartesian-product class of failure. It kills THIS vm, no one else's.
        result = sbx.exec("python3 /workspace/run.py", timeout_seconds=60)
        if result.exit_code != 0:
            # Hand stderr back to the customer so they can fix their query.
            raise RuntimeError(result.stderr)

        return {"tenant": tenant_id, "result": sbx.filesystem.read("/workspace/result.json")}
    # VM (and everything the tenant's code touched) is gone here.

The load-bearing details: `metadata={"tenant_id": ...}` so your audit log and billing can attribute the run; `timeout_seconds` on exec as the hard circuit breaker against the loop that never ends; and — most importantly — the only data in the guest is the scoped extract you wrote in. The VM has no route to your warehouse and no platform credentials in its environment, so even if the tenant's code is actively hostile, the worst it can do is compute a wrong answer about its own data.

Give each VM only that tenant's data — and none of your secrets

Isolation of the machine is necessary but not sufficient. A perfectly isolated VM that you hand a superuser warehouse DSN is a well-contained way to let a tenant read everyone's rows. The VM boundary contains the code; it does not sanitize the powers you grant the code. So the second rule is: the guest gets exactly the data the run is allowed to see, and nothing else.

• Inject a scoped extract, not a connection. The cleanest pattern for a bounded query is to pull that tenant's slice out of your store yourself, write it into the guest filesystem (`filesystem.write`), and let the query run against the local copy. The VM has no network route to your warehouse at all, so a nosy transform has nothing to be nosy about.
• Keep platform secrets out of the guest. Your warehouse master DSN, object-store keys, and internal API tokens must never be exported into the sandbox environment. If the running code can read an env var, treat that env var as leaked. Pass only run-scoped, least-privilege credentials — never the platform's own.
• If the query genuinely needs live SQL, give it a database that only contains that tenant's data (next section), reached with a read-only, run-scoped credential — not a filter on a shared warehouse. 'The customer will always add the tenant_id filter' is not a security control.
• Attribute everything. Tag the sandbox with the tenant id so a runaway run, a suspicious egress attempt, or a billing line traces back to exactly one customer.

Resource limits and timeouts: cap the query to one VM you can kill

The whole reason a runaway query is survivable here is that it's fenced inside one machine with a finite ceiling. A microVM has a fixed memory and CPU budget baked into its template — a 40GB pandas allocation doesn't eat the host, it hits the VM's ceiling and the guest OOM-kills the offending process, which is exactly the blast radius you want. The tenant's job dies; the host and every other tenant's VM keep humming.

Layer a timeout on top for the failure that isn't about memory — the infinite loop, the query that would technically finish sometime next quarter. The `timeout_seconds` on `exec` cancels the run; the `ttl_seconds` on the sandbox is the backstop that reaps the whole VM even if your orchestration process crashes mid-run and forgets about it. Two independent cutoffs, because the expensive query is the common case, not the exception, when customers write their own transforms.

• Shared worker pool — a 40GB allocation triggers the host OOM killer, which evicts whatever it evicts; a tenant's neighbors get killed for a job they didn't submit. A `while True` pins a shared core. Cleanup means hoping the worker reset left no temp files or open connections behind.
• microVM per tenant/run — the allocation hits the VM's baked memory ceiling and OOM-kills only that guest's process; a runaway loop is cancelled by the exec timeout or reaped by the sandbox TTL. Teardown is `kill()` — the VM's memory, filesystem, and any half-written state vanish with it. The blast radius is one machine you were going to throw away anyway.

When the query needs real SQL: a managed database VM per tenant

A local extract in the guest is perfect for a bounded transform, but sometimes the customer's model needs an actual SQL engine — joins across several of their tables, a warehouse-scale scan, a dbt run that expects a live Postgres. Handing the sandbox a role on your shared warehouse re-introduces exactly the cross-tenant leak you just designed out: now the query's blast radius is every tenant in that warehouse, and a hallucinated or hostile join is one missing filter away from reading it all.

The stronger answer is a database that physically contains only one tenant's data. On PandaStack a managed database is a dedicated Firecracker microVM plus its own durable volume — not a schema carved out of a shared instance — created in 30–90s (the create blocks until Postgres is genuinely accepting connections). A per-tenant analysis database is then a hardware-enforced boundary: a query that forgets its filter returns that tenant's rows and nobody else's, because there's no one else in the database. Give the sandbox a read-only, run-scoped role on that database, add a statement timeout, and even a hostile query can't mutate anything, can't run forever, and can't see across the boundary.

# One isolated managed Postgres per tenant: its own microVM + durable volume.
# Create takes 30-90s (it blocks until Postgres is accepting connections),
# so you provision this ONCE per tenant, not once per query.
curl -sS -X POST https://api.pandastack.ai/v1/databases \
  -H "Authorization: Bearer $PANDASTACK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"label": "analytics-tenant-acme"}'

# Response returns a TLS-only connection_url scoped to THIS tenant's VM alone:
#   postgres://pandastack:<pw>@<id>.db.pandastack.ai:5432/pandastack
#
# Do NOT hand the tenant's query this superuser URL. Connect once as admin,
# create a read-only role with a statement_timeout, and give the SANDBOX a
# DSN built from that scoped role instead — never the platform's own.

For truly per-run isolation — where you want each notebook execution to start from a known-good copy with zero chance of state bleeding between runs — fork a seeded database VM instead of provisioning a fresh one. A same-host fork lands in 400–750ms (cross-host 1.2–3.5s) and shares memory and disk copy-on-write, so each run gets a private, disposable copy of exactly the data it's allowed to see without paying the 30–90s full-create every time. The run finishes, you destroy the fork, and there's nothing left to leak.

Fresh-per-run vs. long-lived-per-tenant

There are two ways to hold the VM, and the right one depends on the workload. Neither is wrong; picking the wrong one just costs you either latency or money.

• Fresh microVM per run — create a VM for the query, run it, kill it. This is the strongest posture: nothing survives between runs, so there's no residual state for one execution to leak into the next, and idle tenants cost nothing because there's no VM sitting around. At p50 179ms to create, a VM-per-query is cheap enough to be the default for one-shot transforms, scheduled jobs, and API-driven analysis. This is where most per-tenant query workloads should live.
• Long-lived microVM per tenant — for an interactive session (a hosted notebook a customer pokes at all day, a warm workspace with libraries and data already loaded), a fresh VM per cell is wasteful and slow. Keep one persistent VM per tenant, mark it `persistent=True` so the idle reaper leaves it alone, and route that tenant's cells to it. The hard rule: one tenant per persistent VM, forever. The VM is the boundary; the instant you reuse one persistent sandbox across two tenants, you've erased the boundary and rebuilt the shared-pool leak by hand.
• The hybrid most platforms land on — fresh-per-run for the long tail of one-shot queries and scheduled models, and a long-lived per-tenant VM only for the interactive-notebook feature where warmth is the product. Hibernate the interactive VMs between bursts and auto-wake them on the next request, so an idle notebook trends toward zero cost while a returning user gets a warm resume.

Teardown is the feature, not an afterthought

The reason this architecture stays safe under load is that destruction is total and cheap. When a run finishes — or when a query blows its timeout, or when a tenant churns — you kill the VM, and its memory, its filesystem, its scoped data extract, and any half-written temp files go with it. There's no worker to reset, no `/tmp` to sweep, no lingering connection to reap, no chance that the next tenant's job inherits a stale dataframe. Fresh-per-run VMs teardown automatically via `ttl_seconds` and the `with` block; long-lived per-tenant VMs you `kill()` explicitly when the account closes. Either way, the leak surface between one run and the next is zero, because there is no next run on the same machine.

Putting it together

Letting customers write their own transforms is a great feature and a genuine liability, and the liability is entirely about where that code runs. Don't run it in a shared worker pool, where one tenant's cartesian join OOM-kills a neighbor and one tenant's code can read another's data and your secrets. Run each tenant's query in its own Firecracker microVM: give the guest only that tenant's scoped data and none of your platform credentials, cap it with a timeout and the VM's own memory ceiling, reach for a per-tenant managed database VM when the query needs live SQL, choose fresh-per-run for one-shots and a long-lived VM only for interactive notebooks — and let snapshot-restore keep the whole thing fast enough that isolation isn't a tax. The runaway `SELECT *` cross join still happens. It just takes down one disposable VM instead of your Tuesday.