Per-Tenant Feature-Flag Evaluation in microVMs

At some point every feature-flag product ships the power-user feature: let tenants write their own targeting rules. Not just "enable for 10% of users," but real conditions — segment expressions, custom attributes, a little JSONLogic tree, sometimes an outright script the tenant authored to decide who gets the new checkout flow. It's a fantastic feature, and it quietly changes what your evaluation service is: you now run logic written by strangers, on the hot path of every request, deciding behavior for all of your customers at once. That's a much scarier sentence than the roadmap ticket implied.

This post is about what goes wrong when tenant-authored flag rules all run in one shared evaluator, why config-level guardrails don't actually contain it, and the shape that does: one Firecracker microVM per tenant (or per evaluation batch), with hard CPU, memory, and wall-clock caps. I'm Ajay, I built PandaStack, so I'll be honest about the trade — you go from one evaluator process to many small VMs — and about why, past a certain trust threshold, that's the trade you want.

What tenant-authored flag rules actually let in

Data bleed: one tenant reading another's context

Flag evaluation is a function of context — the user, their attributes, the org, sometimes a whole request payload. When every tenant's rules execute in the same process, that process is holding many tenants' evaluation context in the same heap at the same time. A tenant-authored rule that can reach a global, walk an object graph, or exploit a leaky expression sandbox can read a context object that isn't theirs. Even without malice, a sloppy shared cache keyed on the wrong field hands Tenant A's user attributes to Tenant B's rule. The scary part is that flag rules look inert — they're "just conditions" — so nobody treats them like the untrusted code they are.

Runaway CPU: the rule that is secretly an infinite loop

The moment a tenant can express real logic — a loop over their custom segments, a regex against a user attribute, a recursive rule tree — they can express logic that never terminates. Sometimes it's a genuinely malicious `while(true)`. More often it's an honest rule that looked fine against the tenant's sample data and melts down on a real request: a regex with nested quantifiers that backtracks for seconds, a segment expansion that's accidentally quadratic in the number of users. In a shared evaluator, that single evaluation pins a CPU core, and because flag evaluation sits in front of the request, the latency spike lands on everyone's traffic, not just the tenant who wrote the rule.

Crashing the shared evaluator

A tenant rule that throws an unhandled exception, allocates 40GB, or trips a bug in your expression engine can take the evaluator process down. In a shared fleet that isn't one tenant's outage — it's a global one, because the same process was answering every tenant's flag lookups. "Fail open to the default value" helps, but a process that's crash-looping or OOM-killed doesn't reliably fail anywhere; it just adds latency and errors across the board while it restarts.

Why a shared evaluator can't fix this with config

The instinct is to fence the shared process: run rules through an in-process expression sandbox, add an evaluation timeout, cap recursion depth, rate-limit per tenant. Each helps and none is a boundary. In-process language sandboxes are a running argument with attackers — people escape them through builtins, native functions, or prototype pollution — not a wall. A rule timeout often can't interrupt a native regex-backtracking loop mid-evaluation without cooperative checkpoints, and killing the thread doesn't reclaim the work already scheduled. Recursion caps and rate limits smooth the common case but don't stop one 30-second evaluation from holding a worker hostage or one bad allocation from OOMing the box.

The deeper problem is that all of these controls live inside the same process, on the same kernel, sharing memory with every other tenant's context. They are conventions you have to defend forever, and they fail open: when the timeout doesn't fire, the CPU is just gone; when the sandbox leaks, the neighbor's data is just readable. What you want instead is a boundary the tenant's rule physically cannot cross and a resource cap the host enforces whether or not the guest cooperates.

The fix: one microVM per tenant's evaluation, hard-capped

Run each tenant's rule evaluation inside its own Firecracker microVM — its own guest kernel, its own memory, its own virtual disk and network namespace, confined by hardware virtualization (KVM), the same isolation model AWS Lambda uses to run untrusted code from millions of customers. Give that VM a fixed vCPU and RAM allocation and a wall-clock deadline. Now the three failure modes become boring:

• Data bleed → structurally impossible. The tenant's rule runs in a VM that only ever holds that tenant's context. There is no shared heap to walk, no neighbor's cache to read, no global to reach — a different tenant is a different VM that never shared a byte.
• Runaway CPU → contained and killable. A rule that loops forever or backtracks for seconds burns that VM's own vCPU, capped at the VM boundary. When the wall-clock deadline hits you kill the whole VM — no cooperative interrupt required, no native loop to reason about. That tenant's evaluation fails to its default; nobody else notices.
• Crash → contained to one VM. A rule that OOMs or throws takes down its own throwaway VM, not the shared evaluator. Every other tenant's flag lookups keep answering, because they were never in the same process to begin with.
• Cleanup → free. When the evaluation finishes (or times out), you destroy the VM. No state to scrub, no leaked descriptor, no half-run rule lingering — the memory and disk die with the VM.

The historical objection to "a VM per evaluation" was latency and cost: full VMs are slow to boot and heavy to run, so you'd never spin one up on the flag hot path. That's what snapshot-restore kills. On PandaStack a sandbox is created by restoring a baked snapshot on demand — p50 179ms, p99 ~203ms (the snapshot-restore step itself is ~49ms; the rest is network and disk setup) — versus a first-time cold boot of about 3 seconds. So "give this tenant's rules a fresh, capped VM" is a sub-200ms operation, not a provisioning ticket. And for the highest-traffic tenants you don't create per request at all — you keep one warm VM per tenant and reset it from a snapshot, or fork a clean copy (same-host fork is 400–750ms, cross-host 1.2–3.5s).

What a runaway rule actually looks like

Here's the shape of the problem, so it's concrete. A tenant defines a targeting rule as a small expression, and it ships to production looking completely reasonable. The bug is that it never terminates on certain inputs — and in a shared evaluator that single evaluation pins a core for as long as your timeout (if any) allows.

// A tenant pastes this into your "custom targeting rule" box.
// It's supposed to bucket a user by a hashed attribute. Looks fine.
function shouldEnable(ctx) {
  let bucket = 0;
  // Off-by-one from the tenant: `n` is never decremented on the
  // branch a real request hits, so this loops forever.
  let n = ctx.user.segments.length;
  while (n >= 0) {
    bucket = (bucket * 31 + ctx.user.id.charCodeAt(bucket % ctx.user.id.length)) % 100;
    if (bucket === 42) n--;          // only decrements sometimes
  }
  return bucket < ctx.flag.rolloutPercent;
}

// In a SHARED evaluator: one core is now pinned at 100%, every
// tenant's flag lookups queued behind it slow down, and killing the
// worker thread mid-loop may not reclaim the native work cleanly.
// In a PER-TENANT VM: the loop burns only that tenant's VM budget,
// the wall-clock cap trips, you kill the VM, and everyone else is fine.

Running a tenant's rules in its own microVM

Here's the full loop for one tenant's evaluation: create a capped, ephemeral sandbox, push the tenant's rule and the evaluation context into the guest, run the evaluation with a hard timeout, and read the decisions back. The tenant's untrusted rule only ever touches this one VM — a different tenant gets a different `Sandbox` and never shares a byte or a CPU cycle. The `timeout_seconds` on `exec` is your wall-clock cap; if the rule loops forever, the call returns and you tear the VM down and fall back to default values.

import json
from pandastack import Sandbox


def evaluate_flags(tenant_id: str, rule_src: str, contexts: list[dict]) -> list[dict]:
    """Evaluate ONE tenant's untrusted flag rules over a batch, contained."""
    # Ephemeral, capped VM. The baked template governs vCPU/RAM; ttl reaps it.
    with Sandbox.create(
        template="code-interpreter",
        ttl_seconds=120,
        metadata={"tenant": tenant_id, "job": "flag-eval"},
    ) as sbx:
        # 1. Push the tenant's rule + the evaluation contexts into THIS guest.
        sbx.filesystem.write("/work/rule.py", rule_src.encode())
        sbx.filesystem.write("/work/contexts.json", json.dumps(contexts).encode())

        # 2. Evaluate with a HARD wall-clock cap. An infinite loop or a
        #    catastrophic regex burns only this VM's budget; when the
        #    timeout trips we abandon the VM -- no cooperative interrupt.
        run = sbx.exec(
            "cd /work && python eval_runner.py rule.py contexts.json > out.json",
            timeout_seconds=5,
        )
        if run.exit_code != 0:
            # Timed out or the tenant's rule raised. Fail SAFE: this tenant's
            # flags fall back to defaults; every other tenant is unaffected.
            raise RuntimeError(f"[{tenant_id}] eval failed: {run.stderr[:400]}")

        # 3. Read the decisions back out of the guest.
        decisions = sbx.filesystem.read("/work/out.json")
        return json.loads(decisions)
    # VM destroyed on block exit. The tenant's rule, their contexts, and any
    # runaway process die with it -- nothing to scrub, nothing left running.

The ergonomics are the same as any PandaStack workload: `Sandbox.create` returns a handle, `sbx.filesystem.write`/`read` move bytes in and out of the guest, and `sbx.exec` runs a command with a timeout and returns `stdout`, `stderr`, and an `exit_code`. What changes for flag evaluation is the topology — one capped VM per tenant instead of one shared evaluator for everyone — and that topology is what turns a runaway targeting rule from a fleet-wide incident into a single tenant's flags quietly falling back to their defaults.

Shared evaluator vs. per-tenant microVM

Two ways to run tenant-authored flag rules, from softest boundary to hardest. Verify the specifics of any expression engine, language sandbox, or container runtime against its own docs — behavior varies by version and config.

• Isolation — Shared evaluator: every tenant's rules run in one process on one kernel with everyone's context in the same heap; a sandbox escape or a leaky global reaches another tenant's data. Per-tenant microVM: hardware-virtualized guest kernel, separate memory, disk, and netns per tenant — an escape needs a hypervisor break, and the VM only ever holds that tenant's context.
• Runaway CPU — Shared evaluator: one looping or backtracking rule pins a shared core; the timeout may not fire and killing the thread doesn't reclaim the native work — latency spikes for every tenant's lookups. Per-tenant microVM: the loop is confined to that VM's vCPU and killed at the wall-clock cap; other tenants never share the core.
• Crash blast radius — Shared evaluator: a rule that OOMs or throws can take down the process answering everyone's flags — a global outage. Per-tenant microVM: it takes down one throwaway VM; every other tenant keeps evaluating.
• Cleanup — Shared evaluator: state and half-run rules linger in a long-lived process you have to scrub. Per-tenant microVM: destroy the VM and everything — memory, disk, runaway processes — dies with it; zero residual state.

The density math for many small tenants

The worry is that a VM per tenant can't be affordable at thousands of tenants. Two mechanisms make it work. First, copy-on-write: every sandbox restores the same baked template snapshot with memory mapped MAP_PRIVATE, so identical pages — the guest kernel, the language runtime, shared libraries — are shared across VMs until one writes. A thousand evaluation VMs don't cost a thousand times one VM's RAM. Second, ephemerality: an evaluation is a short job, so a per-batch VM exists only for the milliseconds-to-seconds it runs and is reaped immediately, rather than sitting idle. For warm per-tenant VMs you keep only your active tenants resident, and reset the rest from snapshot on demand.

Capacity-wise, a single PandaStack agent pre-allocates 16,384 /30 subnets, so per-VM networking isn't the ceiling — host memory and CPU are, and CoW plus short-lived jobs push that ceiling far past what "one full VM per tenant" naively implies. The economics flip from "absurd" to "boring": the cost of isolating an evaluation is a sub-200ms create and a few MB of guest overhead, and in exchange a tenant's worst targeting rule becomes their problem instead of your outage.

When a per-tenant VM is the wrong call

Be honest about the trade. If your flag rules are all first-party — you write every targeting condition and tenants only pick from a vetted menu of pre-built segments and percentages, with no tenant-authored expressions or scripts — then a shared evaluator is simpler and denser, and you should use it. The per-tenant VM earns its keep exactly when the shared model's soft boundaries become load-bearing security-and-reliability controls: tenants who write arbitrary expressions (runaway CPU you can't statically rule out), tenants who supply real rule code (arbitrary execution you can't sandbox in-process with confidence), or highly sensitive context where one tenant reading another's attributes is a breach you can't risk. In those cases you're not adding complexity for its own sake — you're replacing three fragile invariants you'd defend forever with one hardware boundary and a wall-clock cap you get for free. And with sub-200ms create plus CoW density, the classic reason not to do it — VMs are too slow and too expensive per job — no longer holds.