Building a data clean room on microVMs

Two organisations have datasets that would be worth a lot if joined, and both are legally and commercially unable to send them to each other. An advertiser wants to know which of its conversions came from a publisher's impressions. A hospital wants to know which of its patients on a drug had the outcome the pharma company's registry is tracking. A bank and a fintech want to know how much of their customer base overlaps before signing a partnership. In every case the answer is a small number and the inputs are the most regulated data either side owns.

The thing that computes that small number is a clean room. Most of the industry's clean rooms are, architecturally, a shared job runner with a contract stapled to it: both parties upload, a query runs somewhere in a multi-tenant cluster, an aggregate comes back, and the privacy property rests on the operator's word plus an audit report from eighteen months ago. That works right up until the first person asks what would actually happen if the operator, or a bug, or another tenant, wanted the rows.

I'm Ajay; I built PandaStack. This post is about building the compute half of a clean room on ephemeral Firecracker microVMs: one disposable guest per query, both parties' data mounted only inside it, no outbound network so results can only leave through a path you audit, output thresholds enforced outside the guest, and the whole thing destroyed when the job ends. It also covers, plainly, the thing microVM vendors tend to skip: this is not a hardware enclave, and there are threat models where you need one.

What a clean room actually has to guarantee

"Clean room" is a marketing term attached to four separate technical guarantees. They fail independently, and most products that call themselves clean rooms have two of the four. Write them down before you design anything, because the architecture falls out of which ones you're actually promising.

1. Input privacy. Party A's raw rows are never readable by party B, by B's code, or by any human at B. Same in reverse. This is the guarantee everybody assumes and the one most casually broken — usually by a support engineer with production access rather than by an attacker.
2. Code review. The job that runs against both datasets is not written by one party and trusted by the other. It's a specific artifact both sides have seen, approved, and pinned by hash. "Party B submits arbitrary SQL against party A's table" is not a clean room; it's party A handing party B a query interface to its own data.
3. Output privacy. The result that comes out is an aggregate, and an aggregate that can't be run backwards. A count over a cohort of one is a row lookup wearing a hat. Repeated queries with slightly different filters are a row lookup with extra steps.
4. No exfiltration. The job cannot send data anywhere except the one audited output path. Not to an S3 bucket, not to a DNS resolver, not to an attacker's server, not to a model API someone imported for "data cleaning."

Notice that only the first has anything to do with encryption. The other three are about what the running code can do — which is an isolation problem, an egress problem, and a policy problem. Encryption at rest and in transit is the easy part and the part everyone's compliance page leads with.

Why a container plus a promise is not a clean room

The default implementation is a pod in a Kubernetes cluster: mount both datasets, run the job, collect the output, delete the pod. It is not a stupid design. It's just that the privacy guarantee it provides is "we didn't do anything bad," and the whole point of a clean room is to be able to describe a guarantee that doesn't depend on that sentence.

• A container is a polite suggestion to the kernel. Namespaces and cgroups are enforced by the same kernel the job's code is calling into. One kernel bug and the boundary between "party A's data" and "whatever else is on that node" is a footnote. The full version of this argument is in /blog/why-docker-is-not-a-sandbox.
• The neighbours are visible. Shared hosts leak through /proc, through mount tables, through timing. A clean room job on the same node as another clean room job is one information channel away from being a very awkward incident report.
• Egress is open unless someone closed it. Container networking defaults to "can reach the internet," and the person who set up the cluster did that so the base image could pull packages. A job that can resolve DNS can exfiltrate a dataset at roughly the speed of a subdomain-per-record loop, and it will look like normal traffic.
• The job is whatever the image contained. If either party can influence the image, or the job is arbitrary user-submitted SQL, the code-review guarantee is void regardless of how good the isolation is.
• Deleting the pod is not destroying the data. The volume, the page cache, the layer cache, the node's disk, the log aggregator that scraped stdout — all of these outlive the pod, and one of them has the rows in it.

Here's the same comparison across the substrates people actually choose between, against the four guarantees.

• Shared container/pod — Input privacy: enforced by namespaces on a kernel the job can call into, plus operator discipline; a kernel bug or a curious admin ends it. Output control: whatever the job writes to the volume, so the job decides what leaves. Verdict: a job runner with a contract, not a clean room.
• Ephemeral microVM per job — Input privacy: a hardware-virtualized guest with its own kernel, one job's data inside, destroyed at the end; the boundary is KVM, not a namespace. Output control: default-deny egress plus a single audited output file the control plane reads out, so the job can produce a result but cannot deliver one itself. Verdict: a real structural boundary, with the host operator still inside the trust model.
• Hardware enclave (SEV-SNP / TDX / Nitro Enclaves) — Input privacy: guest memory is encrypted against the hypervisor and the host operator, and remote attestation lets each party verify the exact code before releasing their decryption key. Output control: same policy layer you'd build anyway, now anchored to an attestation. Verdict: the strongest option and the only one that removes the operator from the trust boundary; costs you hardware constraints, attestation plumbing, and a slower iteration loop.
• Multi-party computation / homomorphic encryption — Input privacy: cryptographic, no party or operator ever holds the other's plaintext at all. Output control: the protocol only computes the agreed function, so exfiltration isn't a category. Verdict: the best guarantee on paper and a severe restriction in practice — the computation has to fit the protocol, and general-purpose analytics mostly doesn't.
• Trusted third party doing it manually — Input privacy: a person with a laptop and an NDA. Output control: a spreadsheet emailed on a Thursday. Verdict: more common in regulated industries than anyone likes to admit.

The microVM row is the pragmatic middle: it's a hard boundary you can build on ordinary infrastructure this quarter, it makes exfiltration structurally hard rather than contractually forbidden, and it's honest about what it doesn't do. The rest of this post is how to build it and how to describe its limits without overclaiming.

The architecture: one ephemeral guest per job

The design has one organising idea: the clean room is not a service that runs continuously and processes jobs. It's a machine that is created for one query, holds both parties' data for the duration of that query, and then stops existing. Nothing persists between jobs, so there is no accumulating pile of two parties' joined data waiting for someone to find it.

1. Both parties upload encrypted inputs to storage under keys neither the other party nor the platform's application tier holds in the clear.
2. Both parties approve a job: a specific query or script, pinned by hash, plus an output schema and an aggregation threshold. Approval is a two-sided signature, recorded before anything runs.
3. The control plane creates a fresh microVM on the approved template and applies default-deny egress to that guest's network namespace before anything is written into it.
4. The per-job decryption keys are delivered into the guest, both parties' inputs are written in, the approved job runs, and it writes exactly one output file to a known path.
5. The control plane reads that one file out, applies the output policy — minimum cohort size, suppression, rounding — outside the guest, and only then releases the result to whoever is entitled to it.
6. Optionally snapshot the guest for dispute resolution, then destroy it. The inputs, the intermediate join, the temp files, and the page cache go with it.

Step 3 is the one to be pedantic about: egress policy goes on before data goes in, not after. A guest that has both datasets and a working default route for even a second has, for that second, been a data-exfiltration appliance. Order of operations is a security control here.

The reason per-job VMs are affordable at all is that a create doesn't boot anything. On PandaStack every create restores a pre-baked Firecracker snapshot — the restore step is around 49ms, end-to-end create is p50 179ms with p99 around 203ms. The ~3s cold boot happens once, at bake time. If you're running a job that scans a few million rows, provisioning is a rounding error; the point is that per-job isolation doesn't cost you a pool of long-lived machines with two parties' data on them. The mechanics are in /blog/snapshot-restore-boot-path.

import json

from pandastack import Sandbox


def run_clean_room_job(job) -> dict:
    """One microVM per approved job. Created for this query, destroyed after it.

    `job` carries the two-sided approval: the pinned job artifact, both parties'
    encrypted inputs, and the output policy. If it isn't signed by both sides,
    this function is never called.
    """
    assert job.approved_by_a and job.approved_by_b, "unapproved job"

    sbx = Sandbox.create(
        template="code-interpreter",
        ttl_seconds=1800,               # hard ceiling: a wedged job cannot sit on
                                        # two parties' data all night
        metadata={
            "kind": "clean-room-job",
            "job_id": job.id,
            "job_sha256": job.artifact_sha256,   # what both sides approved
            "party_a": job.party_a, "party_b": job.party_b,
        },
    )
    try:
        # 1. Egress policy FIRST, before either dataset exists in the guest.
        apply_default_deny_egress(sbx)

        # 2. Both parties' inputs, decrypted only inside the guest. The wrapping
        #    keys are per-job and are useless five minutes from now.
        sbx.filesystem.write("/inputs/a/rows.csv.enc", job.input_a_ciphertext)
        sbx.filesystem.write("/inputs/b/rows.csv.enc", job.input_b_ciphertext)
        sbx.filesystem.write("/run/keys.json", json.dumps(job.per_job_keys))

        # 3. The approved artifact, pinned by hash. Not "whatever SQL B sent us."
        sbx.filesystem.write("/job/job.py", job.artifact_source)

        r = sbx.exec(
            "cd /job && timeout -s KILL 1500 python3 job.py "
            "--keys /run/keys.json --out /job/out/aggregate.json",
            timeout_seconds=1560,
        )
        if r.exit_code != 0:
            # Never echo the job's stderr to a party verbatim -- a traceback can
            # carry a row in it. Log it internally, return a reference.
            log_internal(job.id, r.stderr[-8000:])
            return {"status": "failed", "job_id": job.id}

        # 4. Exactly one file crosses the boundary, and it is read OUT by the
        #    control plane. The guest never pushed anything anywhere.
        raw = json.loads(sbx.filesystem.read("/job/out/aggregate.json"))

        # 5. Thresholds are applied out here, where the job's code cannot see or
        #    influence them. See the policy block below.
        return apply_output_policy(raw, job.policy)
    finally:
        sbx.kill()   # inputs, the joined intermediate, temp files, page cache

Two details worth defending. The result is pulled out by the control plane rather than pushed out by the job — that inversion is what makes "no exfiltration" enforceable rather than aspirational, because the guest has no way to deliver anything even if its code wanted to. And the job's stderr does not go back to a party unfiltered; a Python traceback happily includes the value that caused it, and the value is somebody's row.

Default-deny egress: the result leaves one way

This is the control that turns "the job shouldn't send data out" into "the job cannot." Each PandaStack sandbox gets its own network namespace and routing, so the policy is per-job rather than a fleet-wide firewall someone might relax for an unrelated reason. For a clean room the correct policy is unusually simple, because the answer to "what does this job legitimately need to reach?" is nothing.

#!/usr/bin/env bash
# Clean-room egress policy. Applied to ONE job's guest, in its own netns,
# BEFORE either party's data is written into it.
#
# The correct allowlist for a clean-room job is empty. It has both datasets and
# a CPU; it does not need to talk to anything. The result is collected by the
# control plane reading a file out, not by the job pushing bytes anywhere.
set -euo pipefail

iptables -P OUTPUT DROP
iptables -P FORWARD DROP
iptables -A OUTPUT -o lo -j ACCEPT          # loopback only: DuckDB, local sockets

# Cloud metadata -- first stop for any credential-stealing attempt.
iptables -A OUTPUT -d 169.254.0.0/16 -j DROP

# No lateral movement: your control plane, your storage tier, the next job.
iptables -A OUTPUT -d 10.0.0.0/8     -j DROP
iptables -A OUTPUT -d 172.16.0.0/12  -j DROP
iptables -A OUTPUT -d 192.168.0.0/16 -j DROP

# DNS is exfiltration. A resolver the guest can reach is a covert channel that
# moves a dataset one subdomain label at a time and looks like ordinary traffic.
# There is no allowed resolver here, deliberately.
iptables -A OUTPUT -p udp --dport 53 -j DROP
iptables -A OUTPUT -p tcp --dport 53 -j DROP

iptables -S OUTPUT

If your job genuinely needs a dependency at runtime, the answer is to bake it into the template, not to open a hole. Dependencies fetched at job time are also a supply-chain problem: a package that resolves differently on Tuesday means the artifact both parties approved is not the artifact that ran. Bake, pin, snapshot, and let the guest be offline. The general treatment of egress policy is in /blog/controlling-network-egress-untrusted-code.

A clean room whose compute can reach a DNS resolver is a clean room with a data pipe attached that nobody put on the architecture diagram.

Output guards belong outside the guest

Input privacy without output privacy is theatre. If party B can run "count conversions where postcode = X and age = 34 and device = Y" and get back "1," they have performed a row lookup through an aggregate interface, and every subsequent query narrows it further. The classic attack isn't one query, it's two: run the same aggregate with and without one predicate and subtract.

So you need a policy layer, and its most important property is where it lives. It runs in your control plane, on the aggregate you read out of the guest, after the guest is dead or about to be. It never runs inside the job. A threshold enforced by the job's own code is a threshold enforced by code one of the parties wrote, and "the query politely declines to return small cohorts" is a comment, not a control.

from dataclasses import dataclass


@dataclass(frozen=True)
class OutputPolicy:
    min_cohort: int = 50        # suppress any cell below this count
    round_to: int = 10          # blunt the last digits of what survives
    max_cells: int = 200        # a 10,000-cell "aggregate" is a data dump


SUPPRESSED = None


def apply_output_policy(raw: dict, policy: OutputPolicy) -> dict:
    """Runs in the CONTROL PLANE, on the file read out of the guest.

    Never inside the job: a threshold the job enforces on itself is a threshold
    enforced by code one of the parties wrote.
    """
    cells = raw.get("cells", [])
    if len(cells) > policy.max_cells:
        return {"status": "rejected", "reason": "output_too_granular"}

    out, suppressed = [], 0
    for cell in cells:
        n = int(cell["count"])
        if n < policy.min_cohort:
            # Suppress the VALUE, and report only that suppression happened --
            # "suppressed" already tells you the cohort is under the threshold,
            # which is information. Don't also tell them how far under.
            out.append({**cell, "count": SUPPRESSED})
            suppressed += 1
        else:
            out.append({**cell, "count": round(n / policy.round_to) * policy.round_to})

    # Complementary suppression: one lonely suppressed cell in a row whose total
    # is published can be recovered by subtraction. If only one cell in a group
    # got suppressed, suppress the next-smallest too.
    out = suppress_complements(out, policy)

    return {
        "status": "ok",
        "cells": out,
        "suppressed_cells": suppressed,
        "policy": {"min_cohort": policy.min_cohort, "round_to": policy.round_to},
    }

Thresholds and rounding are the floor, not the ceiling. If your clean room lets a party run many jobs over the same data, you also need a query budget: track cumulative queries per party per dataset, refuse overlapping differential queries, and log every job so a pattern of one-predicate-at-a-time probing is visible after the fact. Formal differential privacy — calibrated noise against a spent epsilon budget — is the rigorous version of this, and if your outputs are genuinely sensitive it's worth the extra work. A minimum cohort size is what most clean rooms actually ship, and it is a great deal better than nothing.

Auditability: keep the job, not the data

Clean room results get disputed. Six months later a partner insists the overlap number was wrong, or a regulator asks what exactly ran against a patient cohort, or your own team needs to know whether a bug affected results from a specific week. "We deleted the VM" is the correct privacy answer and an unhelpful forensics answer.

The pieces you should always retain are cheap and non-sensitive: the job artifact and its hash, both parties' approval signatures, the template identity the guest was baked from, the egress policy that was applied, the output policy parameters, and the released result. That set reconstructs the decision. None of it contains a row.

For the harder disputes there's sbx.snapshot() before sbx.kill(), which preserves the entire working state of the job — intermediate joins included — so an investigator can later snap.fork() a private copy and step through it. A same-host fork lands in 400–750ms and shares memory copy-on-write until written (cross-host 1.2–3.5s), so several investigators can each work in isolation; see /blog/snapshot-and-fork-explained.

Be extremely careful with that one. A snapshot of a clean room job is a durable copy of both parties' raw inputs plus the joined result, which is precisely the artifact the entire architecture exists to avoid creating. If you keep them: encrypt them under a key that requires both parties to release, put them behind a short retention window that actually deletes, restore forks under the same default-deny egress as the original job, and write the whole thing into the agreement rather than discovering it during an audit. The default should be no snapshot; the exception should require a reason.

Honest limits: this is not an enclave

Here is the sentence most clean room vendors work hard not to write. In a microVM-based clean room, the host operator is inside the trust boundary. Firecracker gives you a hardware-virtualized guest with its own kernel, which is a genuinely strong boundary against the code running inside the guest and against other guests on the machine. It does not protect the guest from the machine underneath it. Whoever controls the host controls the hypervisor, and the hypervisor can read guest memory.

• Against the job's code and the other party: strong. The job runs in its own kernel, with no network path out, and both parties' plaintext exists only for the life of one VM.
• Against other tenants on the host: strong. A separate guest kernel and its own network namespace, which is the boundary AWS chose for Lambda for the same reason.
• Against the platform operator: not protected. A host root can inspect guest memory or the memory snapshot. Your controls here are organisational — access management, audit logging, separation of duties — not cryptographic.
• Against a malicious or compromised hypervisor: not protected. VM escapes are rare, not mythical; /blog/vm-escape-attacks-explained is the honest version of that risk.
• Against statistical inference from outputs: only as protected as your policy layer. Isolation contributes nothing here — a perfectly isolated job answering a thousand narrowing queries leaks the dataset one number at a time.

If your threat model includes the operator — because the parties are direct competitors, because a regulator requires it, or because you are the operator and would rather be structurally incapable of reading the data than merely trustworthy — you need confidential computing. AMD SEV-SNP, Intel TDX, or AWS Nitro Enclaves encrypt guest memory against the host and, crucially, provide remote attestation: each party can verify the exact code and configuration running before releasing their decryption key into it. That attestation step is the real prize, because it converts "we promise this is the approved job" into something a party's own client can check. /blog/firecracker-vs-aws-nitro-enclaves covers the comparison.

The costs are real too: specific hardware, a slower and more finicky build-and-attest loop, constrained tooling, and a much smaller pool of engineers who have done it before. Which is why the honest recommendation is a ladder rather than a single answer. Start with per-job microVMs, default-deny egress, and an output policy in the control plane — that already eliminates the shared-kernel, open-egress, and unbounded-output failure modes that break most real clean rooms. Add attestation-backed enclaves when the operator is genuinely in the threat model. Reach for MPC or homomorphic encryption when the computation is narrow enough to fit and the guarantee has to be cryptographic.

What the microVM layer buys you is worth being precise about: both parties' data exists in one place, for one query, on a machine with no way to send anything anywhere, and that machine is gone afterwards. That's not a promise not to look. It's an architecture where looking requires a different set of privileges than running the clean room does — and for a great many two-party joins, that is the guarantee that was actually missing.

For related patterns: /blog/microvm-saas-multi-tenant-isolation covers the general per-tenant isolation model, /blog/microvm-per-tenant-analytics-query covers query isolation without the two-party wrinkle, and /blog/controlling-network-egress-untrusted-code is the deep version of the egress half.