How to run a RAG pipeline in an ephemeral sandbox

A user uploads a 900-page scanned planning application. Your ingestion endpoint hands it to an OCR pass and a PDF parser, both running in the same process that serves your API. Forty seconds later the host is out of memory, the health check fails, and the thing that took your service down was a document rather than a traffic spike.

That is the benign version. The other version is that the file was shaped deliberately: a PDF that trips a bug in the image decoder your parser calls, a DOCX whose XML declares an external entity, a ZIP that expands to a few hundred gigabytes. All three are ordinary attacks against ordinary libraries, and RAG tutorials almost never mention them, because the tutorial is always about chunk size and retrieval quality.

Ingestion is the stage of a RAG pipeline where you run a large pile of file-format parsers over bytes that a stranger chose. That is worth a boundary. I build PandaStack, so a Firecracker microVM is the boundary in the examples here — the pipeline shape works with any sandbox provider, or with a VM you spin up yourself.

Why the ingestion stage, specifically

A RAG system has two halves with completely different trust properties, and it helps to be blunt about which is which.

At query time you embed a short string, run a vector search against an index you built, and hand the results to a model. Every byte in that path is either a user's question, which is text you already treat as untrusted content in the prompt, or a chunk out of your own store, which you put there. There is no parser doing format-guessing work. There is no archive extraction. The risks at query time are real — prompt injection through retrieved content, cross-tenant leakage through a filter bug — but they are logic risks, and a microVM does not fix a wrong SQL predicate.

Ingestion is the opposite. It is a machine whose entire job is to accept a file of unknown provenance and run whatever code is needed to turn it into text. PDF, DOCX, XLSX, PPTX, HTML, EML, images, and usually a ZIP or TAR wrapper around some of those. Each of those formats is handled by a library that parses a complicated binary or XML grammar, most of them have a long CVE history, and you are pointing all of them at attacker-chosen input by design. This is the stage that deserves isolation.

What the boundary actually buys

Three specific things, and it is worth being precise about them, because 'more secure' is not a design goal you can check.

The first is a kernel boundary around the parser CVE class. When a malformed font table or a corrupt image stream gives an attacker code execution inside your parser, the question that matters is what that code execution reaches. In-process, it reaches your database credentials, your object storage keys, the other tenants' documents currently in memory, and your service account. In a microVM with its own kernel, it reaches a copy of one document and a process that is about to be destroyed.

The second is a memory ceiling that means something. A container sharing a kernel with your API can be limited with cgroups, and that works until it does not — the OOM killer's choice of victim is not always the process you intended. A microVM is given a fixed amount of RAM at boot. A pathological document exhausts that and the VM dies. Your API host never learns about it. The same applies to CPU: a decompression bomb burning cores burns them somewhere you are not also serving traffic from.

The third is network. Document formats can make a parser fetch a URL. XML external entities are the textbook case, but there are others: remote entity references, a tracking pixel in an HTML email, an SVG that pulls a remote resource. If the parser runs in your VPC, that fetch is issued from inside your network by a process holding your identity, and the classic pivot is the cloud metadata endpoint at 169.254.169.254, where in the worst case sits a set of instance credentials. Running the parser in a sandbox with its own network namespace means that fetch hits an egress policy you wrote rather than your instance metadata service.

The honest framing is that isolation does not stop a malicious document from doing something. It decides what 'something' can be.

The shape of the pipeline

The common design, the one most ingestion code grows into, looks like this: an upload handler receives a file, calls a parse function, chunks the text, calls an embedding API, and writes rows to a vector store, all inside the web process. Everything in that chain shares one blast radius. The parser has the same reach as the code holding your write credentials, because it is the same code.

The sandboxed version splits it at the parse step:

1. The upload lands directly in object storage. Your API touches the bytes as little as possible, ideally not at all beyond a signed URL.
2. An orchestrator provisions a sandbox — one per document for large or high-risk files, one per batch for small ones.
3. The document goes into the sandbox. Parsing, chunking, and embedding all happen inside it.
4. The only thing that comes back out is structured chunks: text, ordinal, page number, and a vector. Nothing else crosses the boundary.
5. The sandbox is destroyed. Not reset, not reused for the next tenant's file — destroyed.

Step four is the part that makes this work rather than just makes it slower. The output is a narrow, boring data shape that you can validate exhaustively before it touches your store. A list of objects with a string, two integers, and a float array of known length. There is no format-guessing on the way back, which means the return path is not a second copy of the problem you just isolated.

# pip install pandastack
import json
from pandastack import Sandbox

DIMS = 1536
MAX_TEXT = 8000

class IngestFailed(Exception):
    pass

def validate(chunk: dict, doc_id: str) -> dict:
    """The whole boundary is this function. Keep it strict and dull."""
    text = chunk.get("text")
    vec = chunk.get("embedding")
    if not isinstance(text, str) or not 0 < len(text) <= MAX_TEXT:
        raise IngestFailed("chunk text failed validation")
    if not isinstance(vec, list) or len(vec) != DIMS:
        raise IngestFailed("embedding wrong shape")
    if not all(isinstance(x, (int, float)) for x in vec):
        raise IngestFailed("embedding not numeric")
    return {
        "doc_id": doc_id,
        "ordinal": int(chunk["ordinal"]),
        "page": int(chunk.get("page", 0)),
        "text": text,
        "embedding": [float(x) for x in vec],
    }

def ingest_document(local_path: str, doc_id: str, ttl: int = 900) -> list[dict]:
    sbx = Sandbox.create(
        template="rag-ingest",          # parsers + model weights baked in
        ttl_seconds=ttl,
        metadata={"stage": "ingest", "doc_id": doc_id},
    )
    try:
        sbx.filesystem.upload(local_path, "/workspace/in/document")
        r = sbx.exec(
            "python /opt/ingest/run.py"
            " --input /workspace/in/document"
            " --output /workspace/out/chunks.jsonl",
            timeout_seconds=600,
        )
        if r.exit_code != 0:
            raise IngestFailed(f"{doc_id}: {r.stderr[-2000:]}")
        raw = sbx.filesystem.read("/workspace/out/chunks.jsonl").decode()
    finally:
        sbx.kill()

    lines = [ln for ln in raw.splitlines() if ln.strip()]
    return [validate(json.loads(ln), doc_id) for ln in lines]

Note where the try block starts. The sandbox is created outside it and killed in the finally, so a parser that hangs, a timeout, or an exception raised by your own validation all end with the VM gone. The TTL is the backstop for the case where your orchestrator process itself dies, not the mechanism you rely on day to day.

Doing it without paying for it twice

The objection to per-document sandboxing is cost, and it is a fair objection if you implement it naively. Building a container image, or pip-installing a parser stack and downloading embedding model weights on every document, turns a thirty-second job into a multi-minute one and makes the isolation cost more than the work.

Bake the environment once instead. Build a template that already contains your parser libraries, your OCR binaries, your tokeniser, and your embedding model weights, then snapshot it. Every job restores that snapshot rather than constructing an environment. On PandaStack a create through the snapshot-restore path has a p50 of 179ms, which is small enough that provisioning stops being a line item you think about. The general point holds on any platform that can restore a prepared image quickly: the fixed cost belongs at build time, not per document.

Then batch by size. Most corpora are mostly small files, and giving a 40KB text document its own VM is wasteful without being safer in any way that matters — the isolation you care about is between your pipeline and the documents, not between two documents from the same customer. Group small files, give large or unusual ones their own sandbox, and set the TTL from the group.

from concurrent.futures import ThreadPoolExecutor, as_completed

SMALL_BYTES = 2 * 1024 * 1024

def plan(docs, batch_size=25):
    """Yield (documents, ttl_seconds) groups."""
    small = [d for d in docs if d.size_bytes <= SMALL_BYTES]
    large = [d for d in docs if d.size_bytes > SMALL_BYTES]
    for i in range(0, len(small), batch_size):
        yield small[i:i + batch_size], 900      # 25 docs, 15 minutes
    for d in large:
        yield [d], 2700                         # its own VM, 45 minutes

def run(docs, on_chunks):
    with ThreadPoolExecutor(max_workers=16) as pool:
        futures = {
            pool.submit(ingest_batch, group, ttl): group
            for group, ttl in plan(docs)
        }
        for fut in as_completed(futures):
            group = futures[fut]
            try:
                on_chunks(fut.result())
            except Exception as exc:
                mark_failed(group, exc)         # one bad file, one dead group

Two details in that snippet earn their place. The TTL is per group rather than global, because a scanned book legitimately needs longer than a batch of memos and a single global timeout is either too short for the book or too generous for a hung OCR process. And failures are recorded per group rather than raised, so one unparseable file marks twenty-five documents for retry instead of killing the whole run.

One caveat on batching: a batch shares a sandbox, so a hostile document in a batch is in the same VM as its twenty-four neighbours. That is fine when the batch comes from one customer and is not fine when you are mixing tenants. Batch within a tenant, never across.

Where the vectors go

Two options, and the choice is mostly about how much you trust the input.

The sandbox can write to your vector store directly, over a credential scoped to insert-only on one collection and one tenant prefix. This is simpler, avoids moving float arrays back through your orchestrator, and is a reasonable choice for a pipeline ingesting your own content or a partner's.

Or the sandbox writes nothing and hands chunks back, and the orchestrator does the insert. Prefer this one when the documents are genuinely untrusted, because then the sandbox never holds a write credential to your index at all. A compromise inside the VM yields an environment with no reachable secret and no path to your store. The scoped-credential version still means an attacker who owns the parser can write whatever they like into the collection that a model will later be asked to read — which is a prompt injection primitive delivered straight into retrieval, not merely a data integrity problem.

Passing vectors back over HTTP costs something, and for large batches it is measurable. It is still the version I would default to for user uploads, because the alternative puts a write credential inside the process that is most likely to be exploited.

When this is overkill

If you are indexing your own documentation out of your own repository, this is too much machinery. The inputs are files your colleagues wrote and your CI already sees, the parser is running over Markdown and maybe some HTML, and an in-process parse in a worker is fine. Adding a sandbox there buys you a slower pipeline and one more thing to operate.

The same goes for a small internal tool where five known people upload files, and for a pipeline over a corpus you obtained once, inspected, and are not adding to. The line is not the size of the corpus or the seriousness of the company. The line is whether someone you have not met chooses the bytes.

Once they do, the shape above is the one I would build: object storage as the front door, a baked template so provisioning is cheap, one sandbox per document or per same-tenant batch, a memory ceiling and a TTL on each, strict validation of the chunk-and-vector output, and the write credential kept outside the VM. It is not much code. It is mostly a decision about which process is allowed to be surprised.