Per-Tenant Isolation for AI Training-Data Labeling Pipelines

Every data-labeling and training-data platform I've looked at starts from the same architecture, because it's the obvious one: a worker pool, a job queue, and a `tenant_id` column. Jobs come off the queue, the worker loads the tenant's corpus, runs the transform or the validator or the dedup pass, writes annotations back, and picks up the next job. It works. It scales. It's what I would have built. And it has a failure mode that doesn't look like a security incident at all until somebody trains a model, which is the point at which it becomes permanent.

I'm Ajay; I built PandaStack, which runs untrusted and semi-trusted workloads in Firecracker microVMs. The pitch of this post is narrow and I want to be honest about its edges up front: for the CPU-side stages of a training-data pipeline — the parsing, the chunking, the PII scrubbing, the dedup, the quality scoring, the tenant-supplied validators — a microVM per tenant job is the isolation boundary that matches the risk. For GPU-attached labeling and model training, it isn't, and I'll say why toward the end rather than burying it. What follows is the failure mode, why the usual mitigations don't address it, and what changes when the unit of isolation is a whole machine that gets deleted.

"We filter by tenant_id" is not isolation

A `WHERE tenant_id = ?` clause is a correctness mechanism. It protects you from writing a query that returns the wrong rows. It does not protect you from anything else, and the gap between those two things is where every multi-tenant data incident lives. Isolation is a property of the boundary between two workloads: what one can observe, touch, or influence about the other. A query filter is a property of one statement inside one process. If the process is shared, the filter is a suggestion enforced by the diligence of whoever wrote the code path — including the code path someone adds next quarter under deadline.

The distinction becomes concrete the moment you list what a shared worker process actually shares across tenants. Not hypothetically — structurally, by virtue of being one process on one machine:

• The heap. Tenant A's documents were in memory five milliseconds ago; a buffer-reuse bug, an unpickled object retained by a closure, or a logging library that captures locals on exception can carry them forward.
• The filesystem. `/tmp/extracted/`, a scratch directory a PDF parser wrote to, a downloaded model cache — none of these have a tenant_id column.
• Every in-process cache. This is the big one and it gets its own section, because a dedup cache is a cross-tenant data structure by design.
• The kernel. Shared page cache, shared sysctls, shared everything a container namespace does not actually virtualize. A kernel bug is a cross-tenant bug.
• Environment and credentials. If the worker holds credentials scoped to all tenants because it processes all tenants, then any code that runs inside it holds credentials scoped to all tenants.
• The network. A worker that can reach tenant A's bucket and tenant B's bucket can, from tenant A's job, reach tenant B's bucket. Nothing in the socket knows which job you're on.

And then there's the case that makes all of this urgent rather than theoretical: tenant-supplied code. A lot of labeling platforms let customers ship their own validators, their own normalizers, their own scoring heuristics — because the alternative is a support ticket every time someone's corpus has a weird date format. The moment that's true, you are executing customer-authored code in a process that holds every other customer's data in its address space and every other customer's credentials in its environment. At that point `tenant_id` isn't even a suggestion; it's decoration.

The failure mode: contamination you can't un-train

Ordinary data leaks are bad and recoverable-ish. Someone sees rows they shouldn't; you rotate, you notify, you patch, you write the postmortem, you carry the scar. The training-data version is different in kind, because the leaked data doesn't sit in a log you can purge. It gets baked into weights.

Here's the shape of it. Tenant A is a hospital network; their corpus is clinical notes. Tenant B is a fintech; theirs is transaction narratives and support transcripts. Both run through your pipeline. Your pipeline has a dedup stage, because of course it does — near-duplicate documents wreck a fine-tune, and everybody builds a MinHash or SimHash index to catch them. That index is, by construction, a data structure that holds fingerprints and often the canonical text of documents across everything it has ever seen. If it's process-global, or worse, a shared Redis keyed by content hash rather than by tenant, then a document from A is a legitimate dedup candidate for B's corpus. The canonical copy retained is whichever arrived first. Congratulations: B's cleaned training set now contains clinical notes.

Embedding indexes have the same shape and are, if anything, worse, because the whole point of an embedding index is to retrieve semantically related content and "related" doesn't respect a tenant boundary unless you made it. A dedup or retrieval layer that was designed to find similarity across a corpus will happily find similarity across corpora. It's not malfunctioning. It's doing exactly what you asked, over a data set you didn't mean to give it.

Now walk the incident forward. Six weeks later someone notices. Where is the contaminated data? In B's cleaned corpus, in whatever intermediate artifacts you kept, in the model checkpoint, in the two downstream models distilled from it, and in whatever your customer's customers have already been served. You can delete the rows. You cannot delete the weights' knowledge of them — the model has already eaten it, and "unlearning" is an active research area, not an incident-response runbook. Your recovery plan is retraining from a corpus you now have to prove is clean, which requires an audit trail you probably didn't build because the pipeline was a worker pool with a `tenant_id` column.

A data leak you can revoke is an incident. A data leak that got trained into weights is a product recall — except the product is a probability distribution and you cannot issue a firmware update for it.

Making the boundary match the risk

The fix isn't more careful code. It's making the boundary structural, so carefulness stops being load-bearing. The rule I'd write on the whiteboard: one tenant's job gets one machine, that machine holds exactly one tenant's data for its entire existence, and then the machine is destroyed. Not cleaned. Destroyed.

In practice that means each labeling or transform job runs inside its own Firecracker microVM. That VM has its own guest kernel, so a kernel-level bug in tenant A's job is a bug in tenant A's kernel, not yours. It has its own disk — a copy-on-write clone of a template rootfs, not a bind mount of a shared volume — so `/tmp` is genuinely a private `/tmp` and not a shared surface with a naming convention taped over it. It has its own network namespace with its own veth pair and its own NAT rules, so its idea of "the network" is something you configure per job rather than something it inherits. And it gets credentials scoped to one tenant, because it only ever handles one tenant.

The dedup index question then answers itself in the most boring possible way: the index lives inside the VM, or it's a per-tenant store the VM is credentialed for, and there is no code path by which it can be anything else. You didn't write a rule saying "don't share the dedup cache across tenants." You made sharing require crossing a hypervisor boundary, which is not something a careless import does by accident.

What this used to cost, and why it doesn't anymore

The historical objection is real: a VM per job sounds absurd when a VM means a multi-second boot and a gigabyte reserved up front. That arithmetic is what pushed everyone to shared workers in the first place. Firecracker plus snapshot-restore changes it. PandaStack creates a sandbox by restoring a baked template snapshot rather than cold-booting: the restore step lands around 49ms, end-to-end create is p50 179ms and p99 around 203ms, and only the first-ever cold boot of a template is around 3 seconds. Memory restores copy-on-write and the rootfs is a reflink clone, so the thousandth VM doesn't copy gigabytes off disk. Networking is pre-allocated too — each agent keeps 16,384 /30 subnets with their namespaces and taps already built, so the per-job network setup is a MAC patch rather than a full `ip netns add` dance.

The number that matters for a pipeline is the amortization. If your average labeling job runs for 90 seconds, a couple hundred milliseconds of provisioning is well under 1% overhead, and you bought hardware-level isolation with it. If your average job runs for 400 milliseconds, that math is much less friendly, and I'll come back to that in the tradeoffs section — because that case is real and pretending otherwise would be selling rather than engineering.

Running one tenant's job in one machine

The concrete shape. Create a sandbox, write in exactly one tenant's payload and exactly one tenant's transform code, run it under a timeout, read the result out, kill the machine. The `finally` block is not optional decoration — a job that dies in a way you didn't anticipate must still take its VM with it, or you accumulate a graveyard of machines each holding one customer's confidential corpus, which is a strictly worse outcome than the one you were trying to avoid.

import json
from pandastack import Sandbox

def run_labeling_job(tenant_id: str, docs: list[dict],
                     transform_src: str) -> dict:
    """One tenant, one machine, one lifetime. The VM sees this tenant's
    corpus and nothing else, ever, because it is destroyed afterward."""

    # ttl_seconds is the backstop: if this process is OOM-killed mid-job,
    # the platform reaps the VM anyway. Never rely solely on your own
    # cleanup path to delete a machine holding someone's medical records.
    sbx = Sandbox.create(template="base", ttl_seconds=1800)
    try:
        # Only this tenant's data crosses the boundary. Not a shard of a
        # shared file, not a query against a multi-tenant table --
        # a payload assembled outside and handed in.
        sbx.filesystem.write("/work/input.jsonl",
                             "\n".join(json.dumps(d) for d in docs))

        # Tenant-supplied transform code. It runs here, in a guest kernel
        # that is not your kernel, on a disk that holds one corpus.
        sbx.filesystem.write("/work/transform.py", transform_src)

        r = sbx.exec(
            "cd /work && timeout 900 python transform.py "
            "< input.jsonl > output.jsonl 2> errors.log",
            timeout_seconds=960,
        )
        if r.exit_code != 0:
            # Read the guest's own error file; do NOT echo raw stderr into
            # a shared log sink -- it may contain tenant document content.
            err = sbx.exec("tail -c 2000 /work/errors.log",
                           timeout_seconds=10)
            raise RuntimeError(
                f"transform failed for {tenant_id} "
                f"(exit {r.exit_code}): {redact(err.stdout)}"
            )

        out = sbx.exec("cat /work/output.jsonl", timeout_seconds=60)
        stats = sbx.exec("wc -l < /work/output.jsonl", timeout_seconds=10)
        return {
            "tenant_id": tenant_id,
            "rows": int(stats.stdout.strip() or 0),
            "payload": out.stdout,
        }
    finally:
        # The whole compliance story in one line: the machine that held
        # this corpus no longer exists. Not wiped -- gone.
        sbx.kill()

Two details in there that are easy to skip and expensive to skip. First, `ttl_seconds` on create: your `finally` block runs when your process is healthy, and the scenario you're defending against is your process not being healthy. A TTL means the platform deletes the VM even if your orchestrator got OOM-killed halfway through a batch. Second, the redaction on the error path. Error messages are the single most common way confidential content escapes a well-designed pipeline, because a stack trace from a parser helpfully includes the line it choked on, and that line is a patient record, and your log aggregator is cross-tenant by design. Treat anything coming out of the guest as tainted until you've decided otherwise.

Egress: the annotation store, and nothing else

Isolating the machine solves the accidental-neighbor problem. It doesn't, by itself, solve exfiltration, because a machine with a route to the internet is a machine that can POST a corpus somewhere. If any part of your pipeline runs code you didn't write — tenant validators, a scoring model pulled from a registry, a parsing library with 400 transitive dependencies and a maintainer who got phished — then network egress is the channel that turns a code-execution problem into a data-loss problem.

Because each sandbox has its own network namespace with its own veth pair and its own NAT rules, egress is a per-job policy rather than a host-wide one. The posture I'd default to for a labeling worker: deny outbound by default, allow exactly the annotation store and the object store prefix for this tenant, and log the denials — because a denial is a signal. A transform that has never needed the network and suddenly tries to resolve a hostname is telling you something you want to know before the model training run starts.

#!/usr/bin/env bash
# Applied inside the job's own network namespace, per sandbox.
# Default-deny egress; allowlist the two things a labeling worker
# legitimately needs. Everything else is dropped and logged.
set -euo pipefail

ANNOTATION_STORE="10.42.7.15"      # internal annotation API
OBJECT_STORE="10.42.9.0/24"         # tenant-scoped bucket gateway

iptables -P OUTPUT DROP
iptables -A OUTPUT -o lo -j ACCEPT
iptables -A OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# The annotation store: the one place results are allowed to go.
iptables -A OUTPUT -d "$ANNOTATION_STORE" -p tcp --dport 443 -j ACCEPT

# The tenant's own object prefix, reached through a gateway that
# enforces the tenant scope itself -- belt AND braces.
iptables -A OUTPUT -d "$OBJECT_STORE" -p tcp --dport 443 -j ACCEPT

# Internal DNS only, so a transform cannot resolve pastebin.
iptables -A OUTPUT -d 10.42.0.53 -p udp --dport 53 -j ACCEPT

# Log what got denied. A labeling job that suddenly wants the open
# internet is the most interesting alert in your whole pipeline.
iptables -A OUTPUT -m limit --limit 20/min \
  -j LOG --log-prefix "labeljob-egress-denied: "

# Sanity-check the posture before handing the VM any real data.
if curl -s --max-time 4 https://example.com >/dev/null 2>&1; then
  echo "FATAL: egress policy not applied; refusing to load corpus" >&2
  exit 1
fi
echo "egress locked: annotation store + tenant bucket only"

That last block — actively verifying the policy holds before loading data — is the part I'd insist on. A firewall rule you assume is applied is not a control, it's a belief. Prove it in the same script, fail closed, and make "we couldn't confirm the egress policy" a hard stop rather than a warning nobody reads. Verify the specific mechanics against whatever your host networking actually is; the shape of the argument survives the details, but the rule syntax won't survive a different stack.

Ephemerality is the compliance answer

Here's the part that plays well in a room with lawyers in it, which for this class of product is not a joke — a labeling platform handling medical records or unreleased source code lives or dies on whether it can answer questions from a security review. And the hardest of those questions is the simplest-sounding one: where did that PII go?

With a shared worker pool, the honest answer is a list with a shrug at the end. It was in the worker's heap, and the heap has since been reused by other tenants' jobs. It was in `/tmp`, and it was probably cleaned up, unless the job crashed. It was in the page cache. It might be in a core dump, if one was taken. It's in the log lines a parser emitted on a malformed row. It's in whatever the dedup index retained. You can write mitigations for every item on that list and you will still be reciting a list.

With a VM per job, the answer is one sentence: it was on a machine that no longer exists. The guest's memory, its filesystem, its temp directories, its caches, its logs — all of it was inside a machine whose whole existence was one tenant's job. When the job ends, the VM is deleted, and the copy-on-write disk clone and its memory go with it. Not "we ran a cleanup script." The container for the data is gone.

• Data residency — Shared worker pool: the tenant's data has been in a process that also handled every other tenant, so the residency claim covers a machine, not a workload. MicroVM per job: the workload had its own machine for its whole life, and that machine handled exactly one tenant.
• Retention answer — Shared worker pool: "we delete the records and rotate the scratch directory," plus an asterisk about caches, heaps, and logs. MicroVM per job: "the machine that held it was destroyed," with no asterisk.
• Blast radius of a bad transform — Shared worker pool: every tenant currently resident in that process, plus whatever credentials the worker holds. MicroVM per job: one tenant, one disposable machine, one narrowly-scoped credential.
• Tenant-supplied code — Shared worker pool: you're executing customer code next to other customers' data; the mitigations are language-level sandboxes and hope. MicroVM per job: customer code runs under a separate guest kernel with hardware virtualization between it and everything else.
• Dedup / embedding index scope — Shared worker pool: cross-tenant by default, and correcting it is a code-review discipline you must maintain forever. MicroVM per job: per-tenant by construction; sharing would require crossing a hypervisor boundary on purpose.
• Audit story — Shared worker pool: a log line asserting which tenant a job claimed to be for. MicroVM per job: a VM lifecycle record — created at T, held tenant X's corpus, destroyed at T+n — that's an object, not an assertion.
• Cost per job — Shared worker pool: near-zero marginal, which is genuinely its best argument. MicroVM per job: a snapshot restore of roughly 49ms and p50 179ms end to end, plus one VM's memory for the job's duration.

The audit row is the one that matters more than teams expect. When your isolation boundary is a filter inside a process, your evidence that isolation held is a log line your own code wrote — which is exactly the code whose correctness is in question. When the boundary is a VM lifecycle, the evidence is an independent record of a machine that was created, given one tenant's data, and destroyed. Auditors can distinguish those two things, and increasingly they do.

The honest tradeoffs

I'd rather you deploy this where it fits than everywhere. Three places where it doesn't.

When a container plus process discipline is genuinely enough

If every line of code in your pipeline is code your team wrote and reviewed, if your corpora are all the same sensitivity class, if no tenant ever supplies a transform, and if your dedup and embedding indexes are already keyed per tenant with tests that prove it — then a container with a per-job scratch mount, a scoped credential, and a fresh process per job is a defensible design. The residual risk is a kernel escape and a bug in your own tenant scoping, and those are real but they're not the same category as "we run customer code beside customer data." Don't buy a hypervisor boundary to solve a problem you've already solved organizationally, and be equally honest about whether you actually have solved it or just haven't been bitten yet.

Very short jobs, where the overhead is the job

If a unit of work is scoring a single 2 KB document in 30 milliseconds, wrapping it in a machine that takes p50 179ms to provision is a 6x tax on latency and a comparable one on cost. The mitigation is batching, and it's the right one: make the VM's lifetime a tenant's batch rather than a tenant's row. One machine per tenant per shard, processing ten thousand documents, amortizes provisioning to nothing while keeping the isolation property exactly intact — because the boundary you care about is between tenants, not between rows of the same tenant. If you genuinely cannot batch, because work arrives one row at a time from an interactive annotator, then per-job VMs are the wrong unit and a long-lived per-tenant VM that serves that tenant's interactive session is the better shape.

GPU-attached labeling is a different architecture

This one I want to be unambiguous about, because it's where a vendor would normally get vague. Firecracker is a CPU-side microVM monitor with a deliberately minimal device model — that minimalism is exactly why it boots in milliseconds and why its attack surface is small. It is not the tool for attaching a GPU to a tenant workload. If your labeling stage runs a vision model on GPUs, or your pipeline ends in fine-tuning, that stage needs a different isolation strategy: dedicated GPU nodes per tenant, or whatever passthrough and partitioning story your GPU vendor and orchestrator support, and you should verify what that actually guarantees against their documentation rather than against anyone's blog post, including this one.

The realistic architecture is therefore split. The CPU-side stages — ingestion, parsing, chunking, PII detection and scrubbing, dedup, quality scoring, tenant-supplied validation, format normalization, the whole long tail of pipeline work where untrusted code and confidential text meet — run in microVMs, one tenant per machine, deleted on completion. The GPU stages run somewhere with a GPU-appropriate isolation model. That split is not a compromise; it's just noticing that most of a training-data pipeline is text-mangling, and text-mangling is precisely where the contamination risk lives. The dedup index is not on a GPU. The PII scrubber is not on a GPU. The tenant's weird custom validator is definitely not on a GPU.

Where I'd start on Monday

You don't have to rearchitect a pipeline to get most of the benefit, and you shouldn't try. Find the single stage where untrusted or semi-trusted code touches confidential text — for most teams it's either tenant-supplied validators or a parsing stage with a dependency tree nobody has read. Move that one stage into a per-tenant microVM with a default-deny egress policy and a `finally: sbx.kill()`. Measure the overhead honestly against your actual job durations rather than against a benchmark. Then look at your dedup and embedding indexes and ask whether they are keyed per tenant or merely usually keyed per tenant.

The reason to do it in that order is that the contamination failure is silent. Nothing pages you. No error rate moves. The pipeline reports success, the corpus looks clean, the eval scores are fine, and the problem only surfaces when a customer's model says something it could only have learned from a different customer — at which point the fix is retraining, the conversation is with lawyers, and the honest answer to "how long has this been happening" is "we don't have the audit trail to say." A boundary that is a whole machine, created for one tenant and destroyed with the job, is how you make that question answerable before anyone asks it.