Per-Tenant Log Parsing Isolation on microVMs

Every observability and log-management platform eventually offers the same feature: let the tenant bring their own parsing. Custom Grok patterns. A regex to pull a request ID out of an unstructured line. A little transform script to rename fields, drop PII, or reshape a JSON blob before it hits the index. It's a great feature — and it means you are now running code and patterns written by strangers, over a firehose of ingested logs, inside your infrastructure. That's a much scarier sentence than the product spec made it sound.

This post is about the failure modes that come with per-tenant parse/transform logic, why a shared worker fleet is the wrong place to run it, and the shape that actually contains the damage: one Firecracker microVM per tenant's parse pass, with hard CPU, memory, and wall-clock caps. I'm Ajay, I built PandaStack, so I'll be upfront about the trade — you go from one worker pool to many small VMs — and about why, past a certain trust threshold, that's the trade you want.

The threat model: what tenant-supplied parsing actually lets in

ReDoS: a regex is a program, and it can loop forever

The single most common way a log pipeline falls over is catastrophic backtracking. A regex with nested or overlapping quantifiers — the classic shapes look like `(a+)+$` or `(.*a){20}` — can take exponential time on an input that almost matches. Feed it a log line that's 40 characters of the wrong thing and a single regex evaluation can run for seconds, minutes, or effectively forever, pinning a CPU core at 100% the whole time. The tenant didn't have to be malicious; a well-meaning pattern that looked fine against their sample data melts down on a real log line. Either way, a regex with nested quantifiers is a denial-of-service attack you shipped to yourself. And because a regex is a tiny string, nothing about it looks dangerous when the tenant pastes it into a text box.

Transform code: arbitrary execution by design

The moment you let a tenant supply a transform — a Python snippet, a Lua script, a JavaScript function, even a "safe" expression DSL — you are running arbitrary code, or one CVE away from it. Expression sandboxes leak (people escape them through `__builtins__`, prototype pollution, or a native function you forgot to remove). And the whole point of a transform is side effects on data, so "just don't allow I/O" fights the feature. If that code runs in your worker process, a tenant can read your environment variables, open sockets, exhaust memory, or read another tenant's in-flight logs sitting in the same heap.

Noisy neighbor: one heavy tenant starves the rest

Even with no exploit and no runaway regex, tenants are wildly uneven. One customer ships 200 logs a second; another dumps a 50GB batch backfill at 2am. In a shared worker fleet those two draw from the same CPU, the same memory, and the same queue workers. The whale's backfill saturates the pool and the small tenant's live tail latency spikes, or their alerts fire late, even though they did nothing. This is the same class of problem covered in /blog/microvm-cpu-pinning-noisy-neighbor — the kernel scheduler doesn't know Tenant A's backfill should yield to Tenant B's real-time parse, because to the OS it's all just your process.

Why a shared worker fleet can't fix this with config

The instinct is to fence the shared fleet: add a regex timeout, run a linter over patterns, drop the transform into an in-process sandbox, put per-tenant rate limits on the queue. Each of these helps and none of them is a boundary. A regex timeout in most engines can't actually interrupt a native backtracking loop mid-evaluation without cooperative checkpoints — the standard library `re` in Python has no timeout at all, and killing the thread doesn't reclaim the C-level work already scheduled. A static linter can't decide, in general, whether an arbitrary pattern backtracks catastrophically (you'd need to solve it for every possible input). An in-process language sandbox is a running argument with attackers, not a wall. And rate limits smooth throughput but don't stop a single 30-second regex evaluation from holding a worker hostage.

The deeper problem is that all of these controls live inside the same process, on the same kernel, sharing the same memory as every other tenant's data. They're conventions you have to defend forever, and they fail open: when the regex timeout doesn't fire, the CPU is just gone. What you want instead is a boundary the tenant's code physically cannot cross and a resource cap the host enforces whether or not the guest cooperates.

The fix: one microVM per tenant's parse pass, hard-capped

Run each tenant's parse/transform pass inside its own Firecracker microVM — its own guest kernel, its own memory, its own virtual disk and network namespace, confined by hardware virtualization (KVM), the same isolation model AWS Lambda uses to run untrusted code from millions of customers. Give that VM a fixed vCPU and RAM allocation and a wall-clock deadline. Now the three threats become boring:

• ReDoS → contained and killable. A catastrophic-backtracking regex burns the VM's own vCPU, capped at the VM boundary. When the wall-clock deadline hits, you kill the whole VM — no cooperative interrupt required, no C-level loop to reason about. The tenant's batch fails; nobody else notices.
• Arbitrary transform code → contained to a disposable VM. A tenant's Python/Lua/JS transform runs in a guest kernel on a throwaway disk. An escape needs a hypervisor break, not a Python sandbox bypass. It can see this tenant's own log batch and nothing else — no env vars, no neighbor's heap, no host.
• Noisy neighbor → gone by construction. Each VM gets its own vCPU and RAM budget. The whale's 50GB backfill runs in the whale's VM at the whale's cap; the small tenant's live tail runs in a different VM and never contends for the same core.
• Cleanup → free. When the parse pass finishes (or times out), you destroy the VM. There is no state to scrub, no leaked file descriptor, no half-applied transform lingering in a shared worker — the memory and disk die with the VM.

The historical objection to "a VM per parse job" was latency and cost: full VMs are slow to boot and heavy to run, so you'd never spin one up per tenant batch. That's what snapshot-restore kills. On PandaStack a sandbox is created by restoring a baked snapshot on demand — p50 179ms, p99 ~203ms (the snapshot-restore step itself is ~49ms; the rest is network and disk setup) — versus a first-time cold boot of about 3 seconds. So "give this tenant's batch a fresh, capped VM" is a sub-200ms operation, not a provisioning ticket. This is the same per-tenant isolation argument as /blog/microvm-saas-multi-tenant-isolation, applied to the log-parsing hot path.

What a ReDoS actually looks like

Here's the shape of the problem, so it's concrete. This pattern looks like a reasonable attempt to match a repeated group, but the nested quantifier makes it backtrack exponentially on a non-matching tail. Run it against a benign-looking log line and watch a CPU disappear.

import re, time

# A tenant pastes this into your "custom parser" box. Looks harmless.
tenant_pattern = r"^(\w+\s+)+$"   # nested quantifier -> catastrophic backtracking

# A single real log line that ALMOST matches but ends with '!'
line = "user login from host " + "a " * 30 + "!"

start = time.time()
re.match(tenant_pattern, line)      # this can run for many seconds on one core
print(f"took {time.time() - start:.1f}s to (not) match one line")

# In a shared worker: that core is now pinned, that worker is not
# processing anyone else's logs, and re.match has no timeout argument.
# Multiply by a batch of lines and the whole pool stalls.

Running a tenant's parser in its own microVM

Here's the full loop for one tenant's batch: create a capped, ephemeral sandbox, push the tenant's parser and their log batch into the guest, run the parse pass with a hard timeout, and read the structured output back. The tenant's untrusted pattern and code only ever touch this one VM — a different tenant gets a different `Sandbox` and never shares a byte or a CPU cycle. The `timeout_seconds` on `exec` is your wall-clock cap; if the tenant's regex melts down, the call returns and you tear the VM down.

from pandastack import Sandbox


def parse_batch(tenant_id: str, parser_py: str, log_batch: bytes) -> str:
    """Run ONE tenant's untrusted parser over ONE log batch, contained."""
    # Ephemeral, capped VM. Baked template governs vCPU/RAM; ttl reaps it.
    with Sandbox.create(
        template="code-interpreter",
        ttl_seconds=300,
        metadata={"tenant": tenant_id, "job": "log-parse"},
    ) as sbx:
        # 1. Push the tenant's parser + their log batch into THIS guest only.
        sbx.filesystem.write("/work/parser.py", parser_py.encode())
        sbx.filesystem.write("/work/logs.ndjson", log_batch)

        # 2. Run the parse pass with a HARD wall-clock cap. A ReDoS regex
        #    or a fork bomb burns only this VM's budget; when the timeout
        #    trips, we abandon the VM -- no cooperative interrupt needed.
        run = sbx.exec(
            "cd /work && python parser.py logs.ndjson > out.ndjson",
            timeout_seconds=20,
        )
        if run.exit_code != 0:
            # Timed out or the tenant's code raised. Their batch fails;
            # every other tenant's pipeline is completely unaffected.
            raise RuntimeError(f"[{tenant_id}] parse failed: {run.stderr[:400]}")

        # 3. Read the structured output back out of the guest.
        result = sbx.filesystem.read("/work/out.ndjson")
        return result.decode()
    # VM destroyed on block exit. The tenant's parser, their logs, and any
    # runaway process die with it -- nothing to scrub, nothing left running.

The ergonomics are the same as any PandaStack workload: `Sandbox.create` returns a handle, `sbx.filesystem.write`/`read` move bytes in and out of the guest, and `sbx.exec` runs a command with a timeout and returns stdout, stderr, and an exit code. What changes for log processing is the topology — one capped VM per tenant's parse pass instead of one shared worker pool for everyone — and that topology is what turns a runaway regex from a fleet-wide incident into a single failed job.

Shared workers vs. per-tenant microVM

Two ways to run tenant-supplied parse/transform logic, from softest boundary to hardest. Verify the specifics of any regex engine, language sandbox, or container runtime against its own docs — behavior varies by version and config.

• Isolation — Shared workers: everything runs in one process on one kernel; a transform escape or a peek at shared heap reaches every tenant's in-flight data. Per-tenant microVM: hardware-virtualized guest kernel, separate memory, disk, and netns per tenant — an escape needs a hypervisor break.
• ReDoS blast radius — Shared workers: one catastrophic regex pins a shared worker's core; the timeout may not fire and killing the thread doesn't reclaim the native work — throughput drops for everyone. Per-tenant microVM: the backtracking loop is confined to that VM's vCPU and killed at the wall-clock cap; other tenants never share the core.
• Resource caps — Shared workers: cgroups and rate limits are soft and shared; a whale's backfill starves small tenants. Per-tenant microVM: fixed vCPU/RAM per VM enforced by the host, whether or not the guest cooperates — hard caps at the VM boundary.
• Cleanup — Shared workers: state, file descriptors, and half-run transforms linger in a long-lived process you have to scrub. Per-tenant microVM: destroy the VM and everything — memory, disk, runaway processes — dies with it; zero residual state.

The density math for many small tenants

The worry is that a VM per parse pass can't be affordable at thousands of tenants ingesting continuously. Two mechanisms make it work. First, copy-on-write: every sandbox restores the same baked template snapshot with memory mapped MAP_PRIVATE, so identical pages — the guest kernel, the Python runtime, shared libraries — are shared across VMs until one writes. A thousand parse VMs don't cost a thousand times one VM's RAM. Second, ephemerality: a parse pass is a short job, so the VM exists only for the seconds it runs and is reaped immediately after, rather than sitting idle. You pay for the working set of batches actively parsing right now, not for every tenant you've onboarded.

Capacity-wise, a single PandaStack agent pre-allocates 16,384 /30 subnets, so per-VM networking isn't the ceiling — host memory and CPU are, and CoW plus short-lived jobs push that ceiling far past what "one full VM per tenant batch" naively implies. The economics flip from "absurd" to "boring": the cost of isolating a parse pass is a sub-200ms create and a few MB of guest overhead, and in exchange a tenant's worst regex becomes their problem instead of your outage.

When a per-tenant VM is the wrong call

Be honest about the trade. If your parsing is all first-party — you write every Grok pattern and transform, tenants only pick from a vetted menu, and no tenant supplies raw regex or code — then a shared worker pool is simpler and denser, and you should use it. The per-tenant VM earns its keep exactly when the shared model's soft boundaries become load-bearing security-and-reliability controls: tenants who paste arbitrary regexes (ReDoS you can't statically rule out), tenants who supply real transform code (arbitrary execution you can't sandbox in-process with confidence), or wildly uneven tenants where a few whales would starve everyone on a shared pool. In those cases you're not adding complexity for its own sake — you're replacing three fragile invariants you'd defend forever with one hardware boundary and a wall-clock cap you get for free. And with sub-200ms create plus CoW density, the classic reason not to do it — VMs are too slow and too expensive per job — no longer holds.