Your Tenant Wrote a Regex. The Regex Is the Malware.

A customer opens your rule editor, types a grok pattern to pull a request ID out of their application logs, and clicks Save. Ten seconds later your ingest pipeline is running that pattern a hundred thousand times a second, on your workers, against a stream that also carries every other customer's data. Nobody in this story has done anything wrong. There is no attacker and no leaked credential. There is a text box, a regex engine, and a shared worker fleet, and that turns out to be enough.

I'm Ajay; I built PandaStack, a Firecracker microVM platform, and observability vendors are a recurring shape in my inbox. Their value proposition — "send us your messy logs and write rules to make sense of them" — is also an invitation for every customer to submit code into the hot path. This post covers what actually breaks, why the standard mitigations only get you partway, the honest middle ground most teams should ship this week, and the one place a per-tenant sandbox pays for itself immediately.

Your customer didn't write malware, they wrote a regex

The highest damage-to-effort-ratio failure in this category is catastrophic backtracking. Most engines you'd reach for — PCRE, Java's `java.util.regex`, JavaScript's `RegExp`, Python's `re`, Ruby's `Regexp` — backtrack: when a match can proceed more than one way they pick a branch, run with it, and rewind to try the next if it dead-ends. For almost every pattern anyone writes, that is fast and fine.

It stops being fine when the number of branches grows exponentially with input length. The two classic shapes are a quantified group nested inside another quantifier — `(a+)+`, `(\w+\s*)+` — and alternation whose branches can match the same text, like `(a|ab)*`. Both give the engine exponentially many ways to divide the input among the repetitions, and it must try all of them before it can honestly report "no match." Failure, not success, is the expensive case — which is why these patterns behave perfectly in testing and detonate in production.

Go is the notable exception, and it's worth being precise about why. Go's `regexp` package is built on RE2, which compiles to an automaton and is guaranteed linear in the length of the input — it does not backtrack, so there is no exponential case to trigger. Rust's `regex` crate makes the same guarantee. The cost is real: RE2-style engines drop backreferences and lookaround, because those are precisely the features that require backtracking. If your DSL exposes lookahead to customers, you have chosen a backtracking engine, whether or not you framed it that way.

# redos_demo.py -- illustrative. Shows WHY a "fine" pattern is a liability.
import re, time

# This is not an exotic pattern. It is the shape of a hundred grok
# patterns in production: "one or more word-ish groups, then end of line."
# A nested quantifier -- (...)+ wrapping \w+ -- is all it takes.
PATTERN = re.compile(r"^(\w+\s*)+$")

for n in range(18, 30):
    # The trailing "!" makes the match FAIL. Failure is the expensive case:
    # the engine must exhaust every way of splitting the input before it
    # can conclude "no". That set of ways grows exponentially in n.
    line = ("word " * n) + "!"

    t0 = time.perf_counter()
    PATTERN.match(line)
    print(f"n={n:>3}  len={len(line):>5}  {time.perf_counter() - t0:8.4f}s")

# Each +1 to n roughly doubles the time. You will not reach n=40.
# Nothing here is malicious: the pattern is correct for the lines the
# customer tested it on. Then a Java stack trace lands in the message
# body -- one long line, no newline until the end -- and the worker that
# picked it up stops answering for the rest of its natural life.
#
# Go's regexp package does NOT have this failure mode: it uses RE2,
# which is guaranteed linear in the length of the input and does not
# backtrack at all. Rust's regex crate makes the same guarantee. The
# price is real -- RE2-style engines drop backreferences and lookaround,
# because those are exactly the features that require backtracking.

What makes this an availability problem rather than a bug report is what happens next in a shared fleet. Python's `re` has no timeout parameter, and the match is one long-running C call — a signal handler or watchdog thread generally can't preempt it, because the handler only runs between bytecodes and the interpreter isn't executing bytecodes. A runaway `RegExp` stops a JavaScript worker answering anything, including its own health check. Some runtimes have added escape hatches — .NET has a non-backtracking option and a timeout, recent Ruby versions grew one — so check what your version actually offers against its current docs.

The other unbounded thing: memory

Time is the failure everyone can name; memory is the one that pages you at 3am, because it kills the process outright and takes every other tenant's in-flight batch with it. Multiline join rules are the usual culprit. The standard configuration is "a new event starts at a line matching this pattern; everything else is a continuation" — which is a correct description of a stack trace and an unbounded buffer if the start pattern never matches again. One malformed emitter, one log format change, and a worker is accumulating a single "event" until the OOM killer arrives. Any join rule needs a maximum line count and a maximum byte size, and it should flush what it has rather than discard the buffer — otherwise your customer loses the exact data they were trying to capture.

Greedy captures are the same story at smaller scale: `(.*)` against a two-megabyte line materializes a two-megabyte capture group, and a rule extracting six fields has made six copies before writing anything. And then there's the case that sounds like a joke until it's in your incident channel — somebody `cat`s a core dump into the stdout of a containerized process whose logs you are dutifully collecting. Your pipeline has no opinion about whether that is a log line. It just buffers it.

Transform snippets are a scripting runtime in your ingest path

Beyond patterns, most ingest products eventually ship user-supplied transforms: a Lua filter, a JavaScript snippet, a purpose-built DSL like VRL. A real distinction is worth drawing. Restricted DSLs are deliberately not general-purpose — limited or absent looping, no filesystem, no network, a bounded value model — and that restriction is genuinely load-bearing. An embedded Lua or JS runtime is a different proposition: a general-purpose language interpreter, in your process, evaluating text a customer typed.

The usual hardening is a restricted environment — strip `os`, `io`, and `require` from the globals table, expose only the host functions the transform needs. Worth doing, and a mitigation rather than a boundary, for three reasons. Removing globals doesn't stop resource exhaustion: `while true do end` is not an escape, it's a loop, and it pins a core as effectively as a bad regex. The host functions you exposed are themselves attack surface, and they have no track record because you wrote them last quarter. And the interpreter is a large body of native code parsing untrusted input — memory-safety bugs in language runtimes are a normal, recurring class of finding. The argument doesn't rest on one advisory; it rests on the base rate.

The leak nobody models: logs are the worst data you hold

Everything above is availability. This section is the one that ends careers, and it gets a fraction of the attention because it produces no alert.

Consider what is actually in a log stream. Session cookies, because someone logged request headers during a debugging push and never took it out. Bearer tokens in URLs. Names, emails, and phone numbers in application errors. SQL with parameter values inlined. Logs are, in practice, the least-curated and most sensitive dataset a company owns, and the access controls on them are looser than on the database they came from.

Now run tenant A's transform in a process that has been handling tenant B's lines for four hours. None of the channels below need a clever attacker; three of the four are ordinary engineering decisions that were correct in a single-tenant design and quietly became a data-sharing mechanism the day the product grew a tenant column.

• Shared address space. A transform with any read primitive — an interpreter bug, an exposed host function more general than intended, an out-of-bounds read in a native extension — is reading a heap holding other tenants' recently parsed lines. Freed buffers are not zeroed; the allocator is under no obligation to be discreet.
• The error path. This one bites teams with otherwise excellent isolation, because it's so reasonable: when parsing fails, log the offending line so an engineer can debug the rule. Now tenant B's weirdest line is sitting in your platform's own log stream, which your support team reads and which ships to a vendor.
• The shared dead-letter queue. A single `dlq.unparsed` topic accumulates exactly the lines your parsers couldn't handle, from every tenant, under one set of permissions, usually with a long retention because nobody wants to delete data they haven't looked at yet.
• Metrics labels. Somebody adds the unparsed message as a label to debug a counter. Now raw log content sits in your metrics backend — different retention, different access control, much wider internal audience — and you've invented a cardinality explosion that will page you separately about a week later.

The ladder of mitigations, and where each one stops

There's a well-worn sequence of fixes here. Each rung is better than the one below it, and each has a ceiling worth naming, because the failure mode I see most often is a team that implemented rung two and believes it's done.

1. Regex timeouts. Where the runtime supports them, use them. The ceiling: several major runtimes don't offer one, and where the match is a single native call an in-process watchdog can't preempt it. A timeout you can't enforce is a comment.
2. Pattern linting and complexity limits. Reject nested quantifiers and overlapping alternation at rule-save time. Catches the common shapes; well worth shipping. The ceiling: it's a heuristic over a hard problem, producing false positives that annoy legitimate users and false negatives that reach production. A pass means "no obvious footgun," not "safe."
3. Rate limits and quotas. Excellent for noisy-neighbor throughput, and they keep one tenant's backfill from starving everyone. The ceiling: entirely orthogonal to confidentiality. A rate limit has never stopped a transform from reading a buffer.
4. A separate process per parse, with rlimits. `RLIMIT_AS`, `RLIMIT_CPU`, and a hard kill. A real step change, because a process can be killed when a thread cannot, and the honest recommendation for most teams here — it addresses both the time and memory failures with tools already in your standard library. The ceiling: same kernel, same host, same credentials in the environment, and the separation is only as good as what you didn't hand the child.
5. A microVM per tenant workload. Its own guest kernel, its own memory, hardware-enforced. The ceiling: it's the most infrastructure of the five, and the sections below are about where that's justified rather than reflexive.

Before any of that, though, cap the line length. The blowup is exponential in input size, so a 64KB ceiling on a single line converts an unbounded problem into a bounded one for a one-line diff. It is the best ratio of safety to effort available anywhere in this post.

# The unglamorous controls. Ship these whether or not you sandbox.
import re, resource, subprocess, json

MAX_LINE = 64 * 1024          # bytes; longer lines are suspect by definition
RULE_TIMEOUT_S = 0.25         # per rule, per line -- generous is still bounded

# 1. Cap line length BEFORE the regex engine ever sees the bytes. The
#    exponential is in the input length, so this is the single highest
#    leverage line of code in the whole pipeline.
def admit(line: bytes, tenant: str):
    if len(line) > MAX_LINE:
        return dead_letter(tenant, reason="line_too_long", size=len(line))
    return line

# 2. Reject the known-dangerous SHAPES at rule-save time. This is a lint,
#    not a proof -- it catches the common nested-quantifier and
#    overlapping-alternation forms and misses cleverer ones. Treat a pass
#    as "no obvious footgun", never as "safe".
NESTED_QUANT = re.compile(r"\([^)]*[+*][^)]*\)\s*[+*]")   # (a+)+ , (\w*x)*
OVERLAP_ALT  = re.compile(r"\((\w+)\|\1\w*\)\s*[+*]")     # (a|ab)* , (a|a)*

def lint(pattern: str) -> list[str]:
    warnings = []
    if NESTED_QUANT.search(pattern):
        warnings.append("nested quantifier: a quantified group inside a quantifier")
    if OVERLAP_ALT.search(pattern):
        warnings.append("alternation branches can match the same text")
    if len(pattern) > 1000:
        warnings.append("pattern is very long; split it into named fields")
    return warnings

# 3. Run the match in a CHILD PROCESS with rlimits, not in the ingest
#    worker. A thread cannot be preempted mid-match: the regex is one long
#    C call, so your timeout callback simply does not get scheduled until
#    the engine returns. A process can be killed. This is the difference.
def match_isolated(pattern: str, line: bytes) -> dict:
    def limits():
        resource.setrlimit(resource.RLIMIT_AS, (512 << 20, 512 << 20))  # 512 MiB
        resource.setrlimit(resource.RLIMIT_CPU, (1, 1))                  # 1s CPU
    try:
        out = subprocess.run(
            ["python3", "match_one.py", pattern],
            input=line, capture_output=True,
            timeout=RULE_TIMEOUT_S, preexec_fn=limits,
        )
        return json.loads(out.stdout or b"{}")
    except subprocess.TimeoutExpired:
        return {"error": "rule_timeout"}

# 4. Dead-letter PER TENANT. A shared DLQ is a bucket where every
#    customer's unparsed lines -- the ones most likely to be weird and
#    full of PII -- pile up under one set of permissions.
def dead_letter(tenant: str, **fields):
    topic = f"dlq.{tenant}"        # never "dlq.unparsed"
    # And do NOT put the raw line in a metric label or a shared error log.
    # Emit a hash and a length; keep the bytes in the tenant's own store.
    publish(topic, {"tenant": tenant, **fields})

The strongest move: validate the rule before it goes live

Here's the recommendation I'd actually lead with, and it's a product decision more than a security one. In most log products, the first time a customer's new pattern meets real data at real throughput is in production, on shared workers, at whatever hour they clicked Save. The rule editor already asks for a sample of their logs to show a live preview — so you have the corpus, the pattern, and a natural moment to test the combination before it can hurt anyone.

So run it. Create a throwaway sandbox, write the pattern and sample corpus into it, execute the matcher with a hard timeout and memory ceiling, and turn the bad outcome into a form validation error: "line 4,812 of your sample took 8.2 seconds to evaluate; the ingest budget is 100ms per line." That is not an incident, it's a feature — it tells the customer their pattern is quadratic before it tells your on-call that ingest is behind.

This is where a per-job VM is easy to justify, because the workload is bursty and short: you create a VM when someone edits a rule and destroy it seconds later, rather than paying for a standing fleet. Snapshot-restore is what makes it practical — on PandaStack a sandbox isn't cold-booted but restored from a pre-baked snapshot on demand, p50 179ms and p99 203ms, with only the first-ever boot of a template taking around 3 seconds. Bake the parser toolchain into the template once. A pathological pattern then gets its own guest kernel, a vCPU and RAM ceiling enforced by the VM rather than by hoping the engine returns, and deny-by-default egress, so a transform can't ship the corpus it's parsing anywhere.

from pandastack import Sandbox

# The guest-side probe. It runs ONE pattern against a sample corpus with a
# hard per-line CPU + address-space limit, and reports which line was slow.
PROBE = r"""
import json, re, resource, sys, time

resource.setrlimit(resource.RLIMIT_AS, (512 << 20, 512 << 20))  # 512 MiB
resource.setrlimit(resource.RLIMIT_CPU, (5, 5))                  # 5s CPU, hard

pattern = open("/work/pattern.txt").read().strip()
budget  = 0.10          # seconds per line before we call it too slow

try:
    rx = re.compile(pattern)
except re.error as e:
    print(json.dumps({"verdict": "invalid", "detail": str(e)}))
    sys.exit(0)

worst, worst_line, matched = 0.0, None, 0
for i, line in enumerate(open("/work/corpus.log", errors="replace")):
    t0 = time.perf_counter()
    m = rx.search(line)
    dt = time.perf_counter() - t0
    matched += 1 if m else 0
    if dt > worst:
        worst, worst_line = dt, i

print(json.dumps({
    "verdict": "slow" if worst > budget else "ok",
    "worst_seconds": round(worst, 4),
    "worst_line": worst_line,
    "matched": matched,
}))
"""


def validate_rule(tenant_id: str, pattern: str, corpus: bytes) -> dict:
    """Try a tenant's parser rule in a throwaway VM before it reaches ingest.

    A pathological pattern burns one disposable microVM for a few seconds
    and comes back as a 422. It does not touch the ingest fleet.
    """
    with Sandbox.create(
        template="base",
        ttl_seconds=300,                       # backstop if we drop the handle
        metadata={"tenant": tenant_id, "purpose": "rule-validation"},
    ) as sbx:
        sbx.filesystem.write("/work/probe.py", PROBE)
        sbx.filesystem.write("/work/pattern.txt", pattern)
        sbx.filesystem.write("/work/corpus.log", corpus)

        # Two nested caps on purpose: `timeout` bounds the process inside the
        # guest, and timeout_seconds bounds the whole exec from our side. The
        # VM's own vCPU/RAM ceiling bounds it whether or not either fires.
        run = sbx.exec(
            "timeout -s KILL 30 python3 /work/probe.py",
            timeout_seconds=45,
        )

        # GNU timeout reports 124 on expiry; a KILLed child can surface as 137.
        if run.exit_code in (124, 137):
            return {
                "accepted": False,
                "reason": "pattern_too_slow",
                "message": (
                    "This pattern did not finish within 30s on your sample "
                    "logs. Patterns with a quantified group inside another "
                    "quantifier -- like (\\w+\\s*)+ -- can take exponential "
                    "time on long lines. Try anchoring it or matching a "
                    "narrower character class."
                ),
            }
        if run.exit_code != 0:
            return {"accepted": False, "reason": "probe_failed",
                    "message": run.stderr[-2000:]}

        report = json.loads(run.stdout)
        if report["verdict"] == "invalid":
            return {"accepted": False, "reason": "invalid_regex",
                    "message": report["detail"]}
        if report["verdict"] == "slow":
            return {
                "accepted": False,
                "reason": "pattern_too_slow",
                "message": (
                    f"Line {report['worst_line']} of your sample took "
                    f"{report['worst_seconds']}s to evaluate. The ingest "
                    f"budget is 0.10s per line."
                ),
            }
        return {"accepted": True, "matched_lines": report["matched"]}
    # VM destroyed here -- along with the tenant's sample logs, which were
    # never on a host shared with anyone else's data in the first place.

Four topologies for running tenant-supplied parse logic

Same workload, four boundaries, softest to hardest. Verify any regex engine, embedded interpreter, or container runtime against its own current documentation — behavior varies by version and configuration, and it changes.

• ReDoS blast radius — Shared ingest worker: one catastrophic pattern pins a core in a process serving every tenant, and an in-process timeout often can't preempt a native match, so throughput degrades fleet-wide. Process-per-tenant with rlimits: `RLIMIT_CPU` and an external kill actually land, so the damage is one child and one tenant's lag. Container per tenant: cgroup CPU limits throttle it, though the work still contends for the host scheduler. MicroVM per tenant: the loop burns that guest's own vCPU allotment and is reclaimed at the TTL.
• Memory bounding — Shared ingest worker: an unbounded multiline join triggers the OOM killer, which reaps whichever process looks tastiest and takes unrelated tenants' in-flight batches with it. Process-per-tenant with rlimits: `RLIMIT_AS` gives a hard per-parse ceiling; the allocation fails cleanly inside the child. Container per tenant: cgroup memory limits are a genuine improvement, on a process tree sharing the host's page cache and reclaim. MicroVM per tenant: RAM is fixed by the baked snapshot and the guest's own OOM killer reaps the guest's own parser.
• Cross-tenant data leakage — Shared ingest worker: tenant A's transform runs in an address space that held tenant B's lines minutes ago, and the shared error path, dead-letter queue, and metrics labels each carry raw content across the boundary. Process-per-tenant with rlimits: a fresh address space per parse is a large improvement, though the child inherits the parent's environment, credentials, and filesystem view unless you strip them. Container per tenant: separate filesystem and process namespaces, shared kernel. MicroVM per tenant: separate guest kernel and memory, so cross-tenant reads require breaking the hypervisor.
• Egress control — Shared ingest worker: it needs your queues, object store, and databases, so it has broad network access a transform inherits wholesale. Process-per-tenant with rlimits: shares the host's network namespace by default; restricting it is extra plumbing. Container per tenant: per-container policy is achievable and in practice often left permissive, because the pipeline needs egress to function. MicroVM per tenant: each guest gets its own network namespace and tap device, so deny-by-default is the natural posture.
• Cost and ops complexity — Shared ingest worker: cheapest and simplest, one fleet to run — exactly why it's the default and why the failures above are common. Process-per-tenant with rlimits: modest fork overhead and a little supervision code, on primitives already in your standard library — the best value here for most teams. Container per tenant: image builds, a registry, a scheduler, and per-job pull latency unless you keep things warm. MicroVM per tenant: the most infrastructure, offset by snapshot-restore creates at p50 179ms.

When this is overkill

If every parser rule in your system was written by your own engineers and went through review, you do not have an isolation problem. You have a process problem with a much cheaper fix: use a linear-time engine where your language offers one, cap line length, put a timeout on the parse, and go home. A VM boundary between your team and its own reviewed code buys nothing a lint rule wouldn't. The same goes for single-tenant deployments, where "cross-tenant leakage" isn't a category and the worst case is a customer degrading their own pipeline — annoying, but correctly their problem.

It's also fair to say that for a lot of multi-tenant products, rung four is the right stopping point. A separate process with `RLIMIT_AS` and `RLIMIT_CPU`, a per-tenant dead-letter queue, hashed rather than raw values in error paths, and a length cap will eliminate the availability failures entirely and most of the realistic leakage ones. If you implement nothing else from this post, implement that — it's a couple of days of work and it addresses the incidents you're actually going to have.

The per-tenant microVM earns its place at a narrower intersection: you're multi-tenant, your customers write transform code rather than only patterns, and the log data is sensitive enough that "tenant A's snippet ran in a process that recently held tenant B's session cookies" is a sentence you cannot afford to say to an auditor. Be honest about the cost — scheduling, capacity, and a debugging story that's a step removed, since you can't attach a profiler to a VM that's already destroyed. What I'd push back on is treating it as all-or-nothing. The rule-validation path is a small, bounded place to start: bursty, off the hot path, fails safe, and it turns your scariest recurring incident into a form field that goes red.