Per-Tenant Search Indexing in Isolated microVMs

Multi-tenant search is one of those features that looks trivial in the demo and turns into an operational minefield in production. You wire up an Elasticsearch (or OpenSearch, or Meilisearch, or a Tantivy/Lucene indexer) with a tenant_id field on every document, filter every query by it, and ship it. Then three things happen. One tenant kicks off a full reindex of two million documents and every other tenant's queries slow to a crawl. A misbehaving analyzer plugin or a sloppy query returns a document with the wrong tenant filter and now Tenant A can read Tenant B's contract. And a tenant uploads a malformed PDF that crashes your document extractor — which is running in the same process that holds everyone's data. Each of these is a different failure, but they all trace back to the same root cause: you put multiple tenants in one process on one machine sharing one kernel.

This post is about the other shape: a Firecracker microVM per tenant for the indexing and query path. I'm Ajay, I built PandaStack, so I'll be upfront about where the model shines (hard CPU/memory isolation, a real data boundary, cheap density) and where it costs you (you now run more moving parts). The claim isn't "never use a shared cluster." It's that when your tenants are untrusted, uneven in size, or feeding you untrusted documents, a per-tenant VM converts three security-and-reliability problems into one boring capacity problem.

The three failure modes of shared-cluster search

Noisy neighbor: one reindex starves everyone

Indexing is bursty and CPU-and-IO-heavy; querying wants low, predictable latency. In a shared cluster those two workloads fight over the same threads, the same page cache, and the same disk. When a big tenant reindexes, segment merges and analysis saturate the node, and every other tenant's p99 query latency spikes even though they did nothing. You can try to fence this with thread pools, per-tenant rate limits, and cluster-level shard allocation awareness — and people do — but it's a soft boundary you're constantly tuning. The kernel scheduler doesn't know that Tenant A's merge should yield to Tenant B's search.

Data leak: the tenant filter is one bug away from failing

"Every query is filtered by tenant_id" is a security control implemented in application code, and application code has bugs. A forgotten filter clause, a query builder that drops the term under a certain boolean nesting, a shared analyzer or scripting plugin that can read across indices, a cache that keys on the wrong field — any one of these turns your logical partition into a leak. The uncomfortable truth is that in a shared index, tenant separation is a convention, not a boundary. Nothing at the OS or hardware level stops Tenant A's data and Tenant B's data from ending up in the same response, because they live in the same process's memory.

Untrusted documents: your parsers are the attack surface

To build an index you have to extract text from whatever the tenant uploads — PDFs, Office docs, HTML, images run through OCR. Those extractors (poppler, LibreOffice headless, Tika's native libs, image codecs) are large C/C++ codebases with a long history of memory-safety CVEs, and you are feeding them adversarial input by design. A crafted PDF that triggers a heap overflow in the extractor is running inside your indexing service, next to every tenant's data. This is the classic "parse untrusted input in a privileged process" anti-pattern, and it's endemic to search pipelines because ingestion is unglamorous and rarely threat-modeled.

What a per-tenant microVM actually gives you

A Firecracker microVM is a real virtual machine: its own guest kernel, its own memory, its own virtual disk, its own network namespace, confined by hardware virtualization (KVM) — the same isolation model AWS Lambda uses to run untrusted code from millions of customers. Put one tenant's indexer in one microVM and the three failure modes above become structurally impossible rather than carefully-avoided:

• Noisy neighbor → gone by construction. Each VM gets its own vCPUs and its own RAM. Tenant A's reindex burns Tenant A's CPU budget; the host scheduler caps it at the VM boundary. Tenant B's query latency doesn't know Tenant A exists.
• Cross-tenant leak → impossible, not merely filtered. Tenant A's index lives on Tenant A's virtual disk inside Tenant A's VM. There is no tenant_id filter to forget, because there is no shared index and no shared address space. Even a total compromise of Tenant A's indexer sees only Tenant A's data.
• Untrusted parser → contained to a disposable VM. A malicious PDF that pops the extractor gets a guest kernel and a throwaway disk, not your ingestion fleet. Blast radius is one tenant's own documents, which they uploaded anyway.
• Per-tenant config → free. Different tenants can run different engines, analyzers, plugin sets, or heap sizes without touching a shared cluster config.

The historical objection to "a VM per tenant" was cost and boot time: full VMs are heavy and slow to start, so you'd never spin one up per tenant. That objection is what snapshot-restore kills. On PandaStack a sandbox is created by restoring a baked snapshot on demand — p50 179ms, p99 ~203ms (the snapshot-restore step itself is ~49ms; the rest is network and disk setup) — versus a first-time cold boot of about 3 seconds. So "give this tenant a fresh indexing VM" is a sub-200ms operation, not a provisioning ticket.

Two shapes: hot index VMs vs. ephemeral reindex jobs

You don't run every tenant the same way. The right split is by whether the tenant needs to serve live queries or just needs an index built.

Persistent sandbox + durable volume for hot tenants

A tenant whose users are actively searching needs a long-lived indexer that answers queries with low latency. Create the sandbox as persistent (exempt from the idle reaper) and attach a durable volume so the index survives restarts and lives on real disk rather than the ephemeral rootfs. This is the same model PandaStack uses for managed databases: a dedicated VM pinned to a host, its state on a durable volume. The tenant's Meilisearch or OpenSearch process stays warm; incremental document updates stream in; queries hit a VM that is doing nothing but serve that one tenant.

Ephemeral VM for batch reindex jobs

A full reindex — reprocess every document, rebuild the whole index — is a batch job, and batch jobs want ephemeral VMs. Spin up a fresh sandbox, feed it the tenant's documents, build the index, snapshot the built index out to object storage (or hand it to the hot VM), then tear the VM down. Because create is sub-200ms and teardown is immediate, running the reindex in its own throwaway VM costs you almost nothing and keeps the CPU-heavy merge work completely off the hot query VM. When a tenant triggers "reindex everything," it never touches anyone else's search — not even their own live index until you flip to the new one.

Sketching the indexer setup

Inside the guest you run whatever engine you like — the VM is just Linux. Here's the shape of a minimal per-tenant Meilisearch-style indexer: a bounded engine, a working directory for the index on the durable volume, and a hard cap on the document extractor so a malicious file can't eat all the RAM. The point isn't the exact tool; it's that this config lives entirely inside one tenant's VM and can't affect anyone else's.

#!/usr/bin/env bash
# Runs INSIDE one tenant's microVM. Index data lives on the durable volume.
set -euo pipefail

INDEX_DIR=/data/index          # durable volume mount, survives restarts
DOCS_DIR=/data/incoming        # tenant's uploaded documents
EXTRACT_MEM_MB=512             # hard cap on the untrusted parser

mkdir -p "$INDEX_DIR" "$DOCS_DIR"

# Extract text from an untrusted upload under tight resource limits.
# ulimit caps address space so a malicious PDF can't OOM the VM;
# even if it crashes, the blast radius is THIS tenant's own VM.
extract() {
  local src="$1" out="$2"
  ( ulimit -v $((EXTRACT_MEM_MB * 1024)); timeout 30s \
      pdftotext -q "$src" "$out" ) || echo "skip: $src (parser failed)"
}

for f in "$DOCS_DIR"/*.pdf; do
  [ -e "$f" ] || continue
  extract "$f" "$INDEX_DIR/$(basename "$f").txt"
done

# Build/refresh the index from the extracted text (engine of your choice).
meilisearch --db-path "$INDEX_DIR" --http-addr 0.0.0.0:7700 \
  --max-indexing-memory ${EXTRACT_MEM_MB}Mb &
echo "indexer up on :7700 for this tenant only"

End to end with the Python SDK

Here's the full loop for a batch reindex: create a per-tenant sandbox, push the tenant's documents into the guest filesystem, run the indexer, verify it built, and either serve queries against it or snapshot the built index and tear the VM down. Note that the untrusted documents only ever touch this one VM — a different tenant would get a different Sandbox instance and never share a byte.

from pandastack import Sandbox

def reindex_tenant(tenant_id: str, documents: dict[str, bytes]) -> None:
    """Build a fresh index for one tenant in its own isolated microVM."""
    # Ephemeral VM for the heavy reindex job. Persistent=False -> reaped after.
    with Sandbox.create(template="base", ttl_seconds=1800,
                        metadata={"tenant": tenant_id}) as sbx:
        # 1. Push this tenant's untrusted documents into ITS guest only.
        for name, blob in documents.items():
            sbx.filesystem.write(f"/data/incoming/{name}", blob)

        # 2. Extract + index inside the VM. A malicious file here can, at
        #    worst, crash this VM -- it cannot reach another tenant.
        build = sbx.exec("bash /opt/index/build.sh", timeout_seconds=1200)
        assert build.exit_code == 0, build.stderr

        # 3. Sanity-check the built index before promoting it.
        stats = sbx.exec("cat /data/index/stats.json", timeout_seconds=30)
        print(f"[{tenant_id}] built index:", stats.stdout.strip())

        # 4. Snapshot the built index out to object storage, then this VM
        #    is torn down on block exit -- zero residual cost.
        packed = sbx.exec("tar czf - -C /data/index . | base64",
                          timeout_seconds=300)
        upload_index(tenant_id, packed.stdout)  # your storage of choice
    # VM destroyed here; the tenant's documents die with it.


def query_tenant(tenant_id: str, q: str) -> str:
    """Hot path: a persistent per-tenant VM answers live queries."""
    sbx = Sandbox.create(template="base", persistent=True,
                         metadata={"tenant": tenant_id})
    try:
        # There is no tenant_id filter to forget -- the index in THIS VM
        # only ever contained THIS tenant's documents.
        res = sbx.exec(
            f"curl -s 'http://localhost:7700/search?q={q}'",
            timeout_seconds=15,
        )
        return res.stdout
    finally:
        # In production you'd keep this warm and reuse it across queries,
        # or hibernate() it between bursts rather than killing it.
        pass

The ergonomics are the same as any other PandaStack workload: `Sandbox.create` returns a handle, `sbx.filesystem.write`/`read` move bytes in and out of the guest, and `sbx.exec` runs a command with a timeout and returns stdout, stderr, and an exit code. What changes for search is the topology — one VM per tenant instead of one index for all — and that topology is what makes the isolation real rather than filtered.

Shared cluster vs. per-tenant container vs. per-tenant microVM

Three ways to carve up multi-tenant search, from softest boundary to hardest. Verify the specifics of any competing engine or runtime against its own docs — behavior varies by version and config.

• Tenant isolation — Shared cluster: logical only (a tenant_id filter in app code; one bug leaks). Per-tenant container: process + namespace isolation, but a shared host kernel (an escape or kernel bug crosses tenants). Per-tenant microVM: hardware-virtualized guest kernel per tenant; separate disk and netns — a real boundary.
• Noisy-neighbor reindex — Shared cluster: constant fight, soft-fenced with thread pools and rate limits. Per-tenant container: cgroups help, but page cache and kernel are shared. Per-tenant microVM: hard CPU/RAM caps at the VM boundary; one tenant's merge can't touch another's query latency.
• Untrusted document parsers — Shared cluster: parser runs next to all tenants' data (worst case). Per-tenant container: contained to a container, but a memory-safety exploit hits the shared kernel. Per-tenant microVM: contained to a disposable VM; escape needs a hypervisor break.
• Per-tenant customization — Shared cluster: hard (one config, one plugin set). Per-tenant container: possible per image. Per-tenant microVM: trivial (each VM is its own machine).
• Density / cost — Shared cluster: highest density, lowest per-tenant overhead. Per-tenant container: moderate. Per-tenant microVM: a few MB of guest overhead per VM, but snapshot-restore + copy-on-write keep it cheap enough to run per tenant.
• Operational complexity — Shared cluster: one thing to run (until it's on fire). Per-tenant container: orchestrate many containers. Per-tenant microVM: orchestrate many VMs — a platform handles the scheduling, networking, and lifecycle for you.

The density math for many small tenants

The instinct is that a VM per tenant can't be affordable if you have thousands of small tenants. Two mechanisms make it work. First, copy-on-write: every sandbox restores the same baked template snapshot with memory mapped MAP_PRIVATE, so identical pages (the guest kernel, the engine binary, shared libraries) are shared across VMs until one writes — a thousand idle indexers don't cost a thousand times one indexer's RAM. Second, scale-to-zero: a tenant nobody is searching right now doesn't need a running VM at all. Hibernate it (snapshot memory + disk, stop the VM) and wake it on the next query; an idle tenant costs storage, not compute.

Capacity-wise, a single PandaStack agent pre-allocates 16,384 /30 subnets, so per-tenant networking isn't the ceiling — host memory and CPU are, and CoW plus scale-to-zero push that ceiling much higher than "one full VM per tenant" would suggest. The economics flip from "absurd" to "boring": you pay for the working set of tenants actively indexing or querying, not for every tenant you've ever onboarded. That's the same reason per-tenant managed databases are viable on this substrate — search indexes are just another stateful per-tenant workload.

When a per-tenant VM is the wrong call

Be honest about the trade. If all your tenants are trusted (internal teams, first-party data), your documents are trusted (you generated them), and your tenants are uniform in size, a shared cluster with tenant filters is simpler and denser, and you should use it — the per-tenant model is buying you isolation you don't need. The per-tenant VM earns its keep exactly when the shared model's soft boundaries are load-bearing security controls: untrusted tenants, untrusted uploaded documents, wildly uneven tenant sizes (a few whales that would starve everyone else), or a compliance requirement that one tenant's data be physically separable from another's. In those cases you're not adding complexity for its own sake — you're replacing three fragile invariants you have to defend forever with one hardware boundary you get for free. And with sub-200ms create and CoW density, the classic reason not to do it (VMs are too slow and too expensive per tenant) no longer holds.