Per-Tenant Isolation for Vector Embedding Jobs

Every multi-tenant RAG or search product ends up with the same background job: take a customer's documents, parse them into text, chunk them, call an embedding model, write the vectors into an index. It's the least glamorous part of the product and the part most likely to be running as a fleet of long-lived Python workers pulling off a queue. Those workers are shared. Tenant A's contracts and tenant B's HR files land in the same process, minutes apart, and the only thing keeping them apart is that your chunker is careful about which `tenant_id` it puts in the metadata column.

That's not isolation. That's a naming convention with good intentions. And the code doing the parsing is the least trustworthy code in your stack, because it is fed files uploaded by strangers: PDFs, DOCX, XLSX, EML, ZIPs, whatever the customer's sales team had on a shared drive in 2011. A PDF parser is a Turing-complete attack surface wearing a business-document costume. You are running it, in a shared process, over adversarial input, next to every other customer's data.

I'm Ajay, I built PandaStack — this post is about the boundary that actually holds for document ingestion: one Firecracker microVM per tenant embedding job, with the parsing step hardware-isolated, egress locked to your model endpoint, and fork used to fan out over large corpora without re-warming anything.

The parser is the attack surface, not the model

Teams reviewing RAG security tend to focus on the fun end: prompt injection in retrieved chunks, jailbreaks, data poisoning the index. Those are real. But the first hostile code in the pipeline runs long before a token is generated, in the extractor. Document formats are not data — they're programs, or close enough. PDF has an embedded scripting model, an object graph with references that can cycle, and a spec thick enough that every implementation has its own dialect of bugs. Office formats are ZIP containers full of XML with entity expansion and external references. Image extraction pulls in codecs written in C in the 1990s. This is a decades-long CVE farm and it is your ingestion hot path.

• Memory-safety bugs in native parsers — the extraction stack under your tidy Python wrapper is usually C or C++: PDF renderers, image codecs, font shapers, XML parsers. A malformed file that trips a heap overflow gives an attacker code execution inside your worker, which is holding every other tenant's documents.
• Zip bombs and decompression amplification — a few kilobytes that expand to terabytes. Office and archive formats are compressed containers, so 'unzip it and read the XML' is an unbounded memory and disk request from an untrusted party. No exploit required; it's a denial of service you invited in.
• XXE and external-entity fetches — XML-backed formats can reference external entities. A naive parser will happily read /etc/passwd or make an HTTP request to an attacker's host from inside your VPC, which is both an exfiltration channel and an SSRF primitive.
• Poisoned content aimed downstream — text crafted to survive chunking and land in another tenant's retrieval results, or to carry instructions into whatever agent consumes the index later. The parser can't tell the difference between a contract clause and an injection payload.
• Pathological structure — 200,000 pages, a cyclic object graph, a font that recurses, a spreadsheet with a million empty columns. Any of these will pin a CPU for hours and starve the queue for every other tenant.

Why a shared embedding worker pool leaks by construction

Even with no exploit at all, a shared worker pool has a data-adjacency problem. The process handles tenant A's documents, then tenant B's, and in between it keeps a heap it didn't zero, a temp directory it cleaned on a best-effort basis, a model client with a connection pool, and any caches your libraries decided to keep. A bug — not an attack, a plain ordinary bug — that reuses a buffer, mixes up a `tenant_id` in a retry path, or writes to the wrong temp filename is enough to put A's text in B's index. You will find out about it from a customer, which is the worst way to find out about it.

Then there's noisy neighbors. Embedding is genuinely heavy: extraction is CPU-bound, chunking is CPU-bound, and a large corpus is a long tail of both. One tenant who uploads a 40 GB document dump doesn't just take a long time — they take the whole pool's headroom, and every other tenant's freshly uploaded files sit in the queue behind a job that was sized by someone else's data. Backpressure in a shared pool is inherently cross-tenant.

• Isolation strength — Shared worker pool: none; every tenant's documents and extracted text pass through one address space with one heap and one temp directory. Container per job: better, but it shares the host kernel, so a parser memory-safety bug plus a kernel bug reaches the host and neighbors. MicroVM per tenant job: hardware-virtualized guest with its own kernel — a compromised parse stays in one disposable VM.
• Blast radius of a malicious file — Shared pool: the process holds other tenants' documents, vector DB credentials, and cloud metadata access. Container: cgroups and seccomp narrow it, but the kernel is a shared surface. MicroVM: the guest holds exactly one tenant's data and is destroyed when the job ends.
• Zip bomb / OOM — Shared pool: takes down the worker and every job on it. Container: a memory limit helps if you set it and if the OOM killer picks the right victim. MicroVM: hits the guest's own fixed RAM ceiling and dies there; the fleet doesn't notice.
• Noisy neighbor — Shared pool: one 40 GB corpus starves everyone's queue. Container: cgroup CPU shares, still competing for the same host kernel and page cache. MicroVM: fixed vCPU/RAM baked per guest plus a ttl_seconds cap; a runaway saturates only itself.
• Egress control — Shared pool: the worker needs broad network reach for everything it might ever do, so every job inherits it. Container: per-container network policy, if your platform actually enforces it. MicroVM: its own network namespace and its own routing — allow the model endpoint, drop the rest.
• Cleanup — Shared pool: best-effort temp deletion and a heap nobody zeroes. Container: docker rm, if nothing escaped first. MicroVM: the guest is deleted; there is no next job for residue to leak into.

One microVM per tenant embedding job

The structural fix is to stop parsing untrusted documents inside your trusted process. Your control plane — the part holding the vector DB credentials, the tenant registry, and the queue — stays on a trusted host and drives disposable sandboxes over an API. The documents are written into the guest, the parser runs in the guest, chunking runs in the guest, and only the finished vectors (plus their metadata) come back. When the job ends you destroy the VM, and everything the parse touched goes with it: temp files, decompressed bombs, whatever a crafted PDF managed to spawn.

The reason this is practical and not just theoretically correct is that creating the VM is cheap. PandaStack doesn't cold-boot per job — every create restores a baked Firecracker snapshot on demand, with the restore step around 49ms and end-to-end create at p50 179ms (p99 ~203ms). A cold boot is ~3s and only happens the first time a template is baked. So a hardware-isolated machine per tenant job costs roughly a fifth of a second, which is noise next to the extraction work itself. Each agent also pre-allocates 16,384 network slots, so running many tenant jobs concurrently is bounded by memory and CPU, not by network plumbing.

from pandastack import Sandbox

def embed_tenant_batch(tenant_id: str, docs: list[tuple[str, bytes]]) -> list[dict]:
    """Parse + chunk + embed one tenant's documents inside a disposable guest.
    The host never opens a customer file."""
    sbx = Sandbox.create(
        template="code-interpreter",
        ttl_seconds=900,               # hard cap: a pathological PDF can't run forever
        metadata={"tenant": tenant_id, "job": "embed"},
    )
    try:
        # Push the untrusted uploads INTO the guest. Nothing is opened host-side.
        for name, blob in docs:
            sbx.filesystem.write(f"/work/in/{name}", blob)

        # Extraction runs where a heap overflow costs you one throwaway VM.
        # --max-bytes is the zip-bomb brake; the guest's own RAM ceiling is the backstop.
        parsed = sbx.exec(
            "cd /work && python3 extract.py --in in/ --out chunks.jsonl --max-bytes 2000000000",
            timeout_seconds=600,
        )
        if parsed.exit_code != 0:
            # A parser crash is now a boring job failure, not an incident.
            return [{"tenant": tenant_id, "error": parsed.stderr[-2000:]}]

        # Embedding call goes out to the model endpoint -- and only there (see egress).
        embedded = sbx.exec(
            "cd /work && python3 embed.py chunks.jsonl vectors.jsonl",
            timeout_seconds=1800,
        )
        print("embed exit:", embedded.exit_code, embedded.stdout[-500:])

        # Only structured output crosses back to the trusted side.
        return read_jsonl(sbx.filesystem.read("/work/vectors.jsonl"))
    finally:
        sbx.kill()   # temp files, decompressed bombs, and any spawned process die here

Note what crosses the boundary in each direction. Going in: bytes the customer gave you. Coming out: a JSONL file of vectors and chunk metadata that your host parses with a strict schema. The host never calls a PDF library. If a crafted document achieves code execution, it achieves it in a guest that contains exactly one tenant's data, has no vector DB credentials, and is deleted in fifteen minutes at the latest.

Egress: the model endpoint and nothing else

Isolation that only covers the CPU is half a boundary. An embedding job legitimately needs the network — it has to reach your model endpoint, whether that's a hosted API or an inference service you run. That single legitimate hole is also the exfiltration channel: a malicious document that gets code execution will try to POST the tenant's text somewhere, or hit the cloud metadata endpoint for instance credentials, or scan your internal network for a vector DB that answers on 6333.

Because each sandbox gets its own network namespace with its own routing and NAT rules, egress policy is per-job rather than per-fleet. The rule you want is boring and strict: allow the model endpoint, allow DNS if you must resolve it, deny link-local (the metadata service), deny RFC1918 (your internal network), deny everything else. Default-deny with a single allow is far easier to reason about than trying to enumerate what's dangerous.

#!/usr/bin/env bash
# Egress policy for an embedding guest: the model endpoint, and nothing else.
# Applied per-sandbox -- each microVM has its own netns, so this is not fleet-wide.
set -euo pipefail

MODEL_IP="${MODEL_ENDPOINT_IP:?set the model endpoint IP}"

# Default deny on the way out. Loopback stays, established replies stay.
iptables -P OUTPUT DROP
iptables -A OUTPUT -o lo -j ACCEPT
iptables -A OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# The cloud metadata service is the first thing an exploited parser reaches for.
iptables -A OUTPUT -d 169.254.0.0/16 -j DROP

# No lateral movement into the VPC: your vector DB, your queue, your neighbours.
iptables -A OUTPUT -d 10.0.0.0/8     -j DROP
iptables -A OUTPUT -d 172.16.0.0/12  -j DROP
iptables -A OUTPUT -d 192.168.0.0/16 -j DROP

# The one legitimate hole: HTTPS to the embedding endpoint.
iptables -A OUTPUT -d "$MODEL_IP" -p tcp --dport 443 -j ACCEPT

# DNS only to the resolver you chose, if you resolve the endpoint by name.
iptables -A OUTPUT -d "${DNS_IP:-1.1.1.1}" -p udp --dport 53 -j ACCEPT

iptables -S OUTPUT

Bake this into the template so it's already in force when the snapshot restores, rather than racing a hostile document to apply it after boot. And prefer pinning the endpoint by IP where you can: if the guest can resolve arbitrary names, DNS itself becomes a low-bandwidth exfiltration channel, and an attacker who can encode a customer's data into subdomain labels does not need your firewall's permission to make a query.

Fan out over a large corpus with fork

One VM per tenant job is the right unit for isolation, but a single tenant with a hundred thousand documents shouldn't be a single-threaded job that runs overnight. This is what fork is for. Prepare one guest with your extraction toolchain, model client, tokenizer, and dependencies installed and warm, snapshot it, then fork that snapshot once per shard of the corpus. Every fork is an independent microVM that shares the baked memory copy-on-write until it writes, so N shards don't cost N copies of your toolchain. A same-host fork lands in 400–750ms (cross-host is 1.2–3.5s), so a 32-way fan-out is a couple of seconds of setup, not a couple of minutes.

The isolation story survives the fan-out because every fork in a job belongs to the same tenant. You are sharding one tenant's corpus across many guests, never packing several tenants into one. The snapshot itself must be tenant-neutral: it holds your code and dependencies, never a previous tenant's documents or extracted text.

from concurrent.futures import ThreadPoolExecutor
from pandastack import Sandbox

SHARDS = 32   # one microVM per shard of THIS tenant's corpus

# 1. Warm one guest with the toolchain, then snapshot it. Tenant-neutral: code only.
base = Sandbox.create(template="code-interpreter", ttl_seconds=1800)
base.exec("pip install pypdf lxml tiktoken httpx", timeout_seconds=600)
base.filesystem.write("/work/extract.py", EXTRACT_SRC)
base.filesystem.write("/work/embed.py", EMBED_SRC)
snap = base.snapshot()
base.kill()

# 2. Fork per shard. ~400-750ms same-host, copy-on-write, no reinstall.
def embed_shard(shard: int) -> int:
    fork = snap.fork(ttl_seconds=1800)
    try:
        for name, blob in tenant_docs_for_shard(shard):
            fork.filesystem.write(f"/work/in/{name}", blob)
        r = fork.exec(
            "cd /work && python3 extract.py --in in/ --out c.jsonl "
            "&& python3 embed.py c.jsonl v.jsonl",
            timeout_seconds=1500,
        )
        if r.exit_code != 0:
            print(f"shard {shard} failed:", r.stderr[-500:])
            return 0
        upsert_vectors(read_jsonl(fork.filesystem.read("/work/v.jsonl")))
        return 1
    finally:
        fork.kill()   # one hostile shard poisons one fork, not the job

with ThreadPoolExecutor(max_workers=SHARDS) as pool:
    ok = sum(pool.map(embed_shard, range(SHARDS)))

print(f"{ok}/{SHARDS} shards embedded")

Sharding also improves your failure semantics. In a shared pool, one document that hangs a parser takes down a worker and whatever else it was holding. Here, a bad shard is a fork that hits its timeout, gets killed, and gets retried — with the specific offending document now identified, because it was the only tenant data in that guest. Debugging ingestion failures stops being archaeology.

The cost and density argument

The usual objection is that a VM per job sounds expensive next to a worker pool that's already running. It's the reverse, and for a specific reason: the shared pool is expensive because it's always on. You size it for peak ingestion, you pay for it at 3am when nobody uploaded anything, and you over-provision headroom precisely because one tenant's 40 GB dump can starve everyone. That idle capacity is pure cost, and it exists to absorb a problem that isolation removes.

Ephemeral microVMs invert that. Nothing runs between jobs; the guest exists for the duration of the work and then it's gone. There is no warm pool to keep fed, because create is a snapshot restore rather than a boot — ~49ms for the restore step, p50 179ms end to end. Density comes from copy-on-write: forks of the same snapshot share memory pages until they diverge, and rootfs clones are reflinks rather than copies, so the 30th shard of a corpus is not the 30th copy of your toolchain. And the practical ceiling is memory and CPU, not networking — each agent pre-allocates 16,384 /30 subnets, so you'd exhaust host RAM long before you ran out of network slots.

The shared pool's cost is idle capacity you keep to survive your worst tenant. Per-job microVMs let you stop paying for that fear.

There is real operational cost here — snapshot hygiene, a queue that hands out jobs, plumbing results back — and if your ingestion only ever touches documents you produced yourself, you may not need any of it. But the moment your pipeline ingests files uploaded by customers, you are running a hostile-input parser in production, and the only question is what's in the room with it when something goes wrong. One microVM per tenant job makes that answer: one tenant's documents, no credentials, and a fifteen-minute lifetime.

For the adjacent problems: running customer-supplied model weights and preprocessing code is covered in /blog/microvm-per-tenant-ml-inference-isolation, the snapshot-and-fork mechanics are in /blog/snapshot-and-fork-explained, and the general model for executing things you don't trust is in /blog/how-to-sandbox-untrusted-code.