Running Per-Tenant Billing and Usage Metering in MicroVMs

Most isolation failures are privacy incidents. You disclose something you shouldn't have, you write the notification email, you tighten the query, and life continues. A billing pipeline is the one workload where a cross-tenant read isn't a disclosure at all — it's a wrong number that gets rendered into a PDF, charged to a card, booked as revenue, and reported to your board. By the time anyone notices, the bug has propagated into your accounts receivable, your revenue recognition, and possibly a filing. It is the only bug class where the incident review includes your CFO.

I'm Ajay; I built PandaStack. This post is about why per-tenant billing and usage metering deserves stricter isolation than almost anything else you run, and about a specific architecture: one ephemeral Firecracker microVM per rating run, per tenant, restored from a pinned snapshot, with only that tenant's events in scope and no way to reach anyone else's. Along the way: customer-defined pricing formulas as the untrusted-code problem they actually are, determinism for audits, idempotency and the double-charge failure mode, and why a crashed rating job must fail closed rather than leave half an invoice behind.

Why billing is not just another batch job

The mechanics look mundane. Raw usage events land somewhere — API calls, GB-hours, seats, tokens, rows scanned. A rating engine reads a tenant's events for a period, applies that tenant's plan (tiers, included allowances, overage rates, minimum commitments, discounts, credits, proration for mid-cycle changes), and produces line items. Those become an invoice, that invoice hits a payment processor, and the resulting numbers become your financial record. It's a `GROUP BY` with feelings.

What makes it different from every other batch job is the consequence gradient. Consider the failure modes side by side, because they are not the same severity at all.

• A search-indexing job with a tenant-filter bug shows the wrong results, someone reports it, you fix the filter and reindex. Recoverable in an afternoon.
• A billing job with a tenant-filter bug charges a customer for consumption they didn't have — and, worse, the amount they were charged is a function of a competitor's usage. You now have a wrong invoice, a refund, a restatement, and a customer who can reasonably ask what else you got wrong.
• A log-processing job that OOMs halfway drops some log lines. Annoying. A rating job that OOMs halfway may have already written half the line items, so the invoice is a partial truth that looks complete.
• A recommendations job that runs twice produces the same recommendations. A rating job that runs twice, without idempotency, produces two charges — and the customer notices that one immediately.
• An analytics query that isn't reproducible is a mild embarrassment. A billing run that isn't reproducible means you cannot answer an auditor's question about why a customer was charged what they were charged, six months ago, under pricing rules that have since changed.

Every other pipeline fails into an incident. A billing pipeline fails into a financial restatement.

That gradient should drive your architecture, and usually doesn't. The typical shape is a shared worker fleet — Airflow, a Celery pool, a Spark cluster, a Kubernetes CronJob — where the same long-lived process rates tenant after tenant in a loop, holding warehouse credentials that reach every tenant's events, with a `WHERE tenant_id = ?` as the only thing standing between customers. That predicate is doing an enormous amount of load-bearing work for a fragment of SQL.

Correctness isolation: make the wrong number impossible to compute

The standard defense is care: code review the queries, add row-level security, write tests that assert tenant scoping. All good practices, and all of them are probabilistic. They reduce the odds that some future engineer, at 4pm on a Friday, adds a join that drops the predicate or a cache that keys on the wrong thing. They don't make it impossible.

Structural isolation is a different claim. If the process computing tenant B's invoice has, in its address space and on its network path, only tenant B's events and only tenant B's rating rules, then a cross-tenant read isn't caught by a filter — it has nothing to read. A dropped `WHERE` clause returns tenant B's rows because tenant B's rows are the only rows present. The bug becomes a no-op instead of an incident.

That's the argument for one microVM per rating run. The orchestrator — the trusted part that knows which tenants need rating, holds the master warehouse credentials, and decides what runs — stays on a host that never executes tenant-specific logic. For each run, it provisions a guest, hands it exactly one tenant's event data (or a short-lived, tenant-scoped credential that can only fetch that data), runs the rating logic inside, collects the line items, verifies them, and destroys the guest. The blast radius of a bug in the rating engine is one tenant's invoice, which is exactly the blast radius you want it to have.

Customer-defined pricing rules are untrusted code with a friendly name

If you sell usage-based billing to other companies — a metering platform, a billing-as-a-service product, an infrastructure vendor with enterprise pricing — you will eventually ship customer-defined rating rules. It starts small and reasonable: a formula field so a customer can express "charge per GB, but round up to the nearest 10 GB, and waive the first 100." Then it grows conditionals. Then it grows a small expression language. Then someone asks for a Python hook because their contract has a clause your DSL can't express.

At that point, you are running arbitrary code supplied by one customer inside a process that computes other customers' money. "Rating expression" is a friendly name for remote code execution against your most financially sensitive service. And the standard mitigations for expression languages are exactly as leaky here as everywhere else.

• Expression languages grow escape hatches — attribute access, function registries, and template filters are all reachable surface. A sandboxed evaluator is a security boundary maintained by a library author who did not know your threat model.
• The interesting attack isn't exfiltration, it's influence — a rule that can read a shared cache, a global config object, or another tenant's aggregate doesn't need to steal anything. It just needs to make its own bill smaller, which is a fraud vector that looks like a rounding difference.
• Resource exhaustion is a denial-of-invoicing — an accidental (or deliberate) unbounded loop in one customer's formula stalls the shared worker, and now nobody's invoices go out on the first of the month.
• Formula bugs are indistinguishable from platform bugs — when a customer's own rule produces a nonsense number, your support team gets a ticket claiming your billing is broken. Running each rule in its own guest gives you a clean per-run log and per-run resource accounting to point at.
• Custom rules need real dependencies — customers will want a date library, a currency library, a tax table lookup. The moment you allow imports, the evaluator sandbox is over and you're relying on the process boundary you didn't build.

A microVM handles this the same way it handles any user-defined function in a SaaS product: give the untrusted code a real machine with a real kernel, give it nothing else, and throw the machine away. The customer's formula can `import os` all it likes; there is no other tenant's data on that filesystem, no shared cache in that address space, and no credential in that environment beyond what this one run needs.

from pandastack import Sandbox
import json

def rate_tenant_period(tenant_id: str, period: str, events_ndjson: str,
                       plan: dict, custom_rule: str | None) -> dict:
    """Rate ONE tenant for ONE period in a guest that has nothing else in it.

    The orchestrator's warehouse credentials never enter this VM. The only
    usage data present is this tenant's -- so a dropped tenant filter inside
    the rating engine can't read anyone else's rows. There are none.
    """
    sbx = Sandbox.create(
        template="code-interpreter",
        ttl_seconds=900,            # a runaway formula cannot bill all night
        metadata={
            "tenant": tenant_id,
            "period": period,
            "kind": "rating-run",
            "idempotency_key": f"{tenant_id}:{period}",
        },
    )
    try:
        # ONE tenant's events. Nothing to leak, nothing to cross-contaminate.
        sbx.filesystem.write("/work/events.ndjson", events_ndjson)
        sbx.filesystem.write("/work/plan.json", json.dumps(plan, sort_keys=True))

        # The customer's own rating expression: untrusted code, isolated guest.
        if custom_rule:
            sbx.filesystem.write("/work/rule.py", custom_rule)

        out = sbx.exec(
            "cd /work && python3 -m rating.run "
            "--events events.ndjson --plan plan.json "
            f"--period {period} --out lines.json",
            timeout_seconds=600,
        )

        # FAIL CLOSED. A non-zero exit, a timeout, or an OOM means we emit
        # nothing at all -- never a partial set of line items that looks whole.
        if out.exit_code != 0:
            raise RatingFailed(tenant_id, period, out.stderr)

        lines = json.loads(sbx.filesystem.read("/work/lines.json"))
        return {"tenant": tenant_id, "period": period, "lines": lines}
    finally:
        sbx.kill()   # the guest, the events, and the customer's rule all vanish

Note what the orchestrator keeps: the decision about which tenants to rate, the master credentials, and the write path into your ledger. Note what it gives away: one tenant's events and one tenant's rules, into a machine that exists for the duration of one computation. That split is the entire design.

Determinism: re-run last quarter and get the same answer

Sooner or later someone will ask you to justify a number. It might be an enterprise customer disputing an overage line, an auditor sampling transactions, a finance team reconciling a variance, or your own engineers trying to understand why March looked strange. The question is always some version of: re-run this billing period and show me how you got there.

On a shared worker fleet, that question is surprisingly hard to answer honestly. The rating code has been deployed forty times since March. The dependency tree has drifted — a decimal library patched its rounding behaviour, a date library changed a timezone table, a pip install picked up a newer transitive package. The worker had a warm cache of exchange rates that no longer exists. Re-running gets you a number, but it's a number produced by today's environment against yesterday's data, and if it differs from the original you cannot tell whether you found a bug or just found drift.

A snapshot-based sandbox turns the environment into an artifact you can pin. A PandaStack template's first spawn is a real cold boot — around 3 seconds — during which the environment initializes; the resulting snapshot is then restored on every subsequent create, at about 49ms for the restore step and 179ms p50 (~203ms p99) end to end. Every restore of that snapshot starts from a byte-identical machine: same interpreter, same library versions, same locale and timezone data, same everything. Version the snapshot alongside your rating code, record which snapshot generation produced which invoice, and "re-run March" means restoring March's environment rather than approximating it.

• Pin the environment — record the template/snapshot generation used for each billing run in the run record itself. The environment becomes an input you can reproduce, not a background condition you hope was stable.
• Pin the inputs — the event set for a period must be immutable once rated. Late-arriving events belong in a subsequent period or an explicit adjustment, never a silent mutation of a period you already invoiced.
• Pin the rules — plan configuration and any custom formula must be versioned and snapshotted per run, because a customer's plan will change and you will need to rate old periods under old terms.
• Eliminate ambient nondeterminism — no wall-clock reads inside rating (pass the period boundaries in explicitly), no unordered map iteration feeding aggregation order, no floating point where decimal arithmetic belongs, no random tie-breaking.
• Hash the output — compute a digest over the sorted line items and store it with the run. A re-run that produces a different digest is a signal you can alert on rather than a discrepancy someone stumbles into during an audit.

import hashlib, json

# Everything that can change the answer becomes part of the run record.
# If any of these differ, a different number is EXPECTED rather than alarming.
run_record = {
    "tenant": tenant_id,
    "period": "2026-06",
    "idempotency_key": f"{tenant_id}:2026-06:v1",
    "snapshot_generation": "rating-engine@2026-06-01",  # the pinned env
    "rating_code_sha": "9f3c1e2",
    "plan_version": 7,
    "custom_rule_sha": "a41b8dd" if custom_rule else None,
    "event_set_sha": hashlib.sha256(events_ndjson.encode()).hexdigest(),
}

# Deterministic digest over the OUTPUT: sorted keys, stable ordering,
# decimal strings rather than floats. Re-running must reproduce this exactly.
def digest(lines: list[dict]) -> str:
    canonical = json.dumps(
        sorted(lines, key=lambda l: (l["sku"], l["starts_at"])),
        sort_keys=True, separators=(",", ":"),
    )
    return hashlib.sha256(canonical.encode()).hexdigest()

run_record["result_digest"] = digest(result["lines"])

# Six months later, an auditor asks about this invoice. Restore the SAME
# snapshot generation, replay the SAME event set, and compare digests.
# Match -> the number is explained. Mismatch -> you have a real finding,
# and you know it's a finding rather than environment drift.

Idempotency, retries, and the double-charge failure mode

Distributed systems retry. Your queue redelivers on visibility timeout, your scheduler fires twice across a leader election, an operator re-runs a job that appeared stuck, a network blip makes a successful write look failed. In most pipelines that's fine — recomputing an aggregate twice yields the same aggregate. In billing, an unguarded retry is a second charge on a real customer's real card, and unlike most bugs, the customer finds it before you do.

Per-run microVMs help here in a specific and slightly unintuitive way: they make the unit of work sharply bounded. A run is one tenant, one period, one guest, one result. That's a natural key — `(tenant, period, rules_version)` — and a natural place to enforce exactly-once semantics at the boundary where results are committed.

1. Derive an idempotency key from the inputs, not from the attempt. `(tenant_id, period, plan_version, event_set_sha)` identifies the computation. Two retries of the same work produce the same key; a genuine re-rate after a plan correction produces a different one, deliberately.
2. Make the sandbox pure. The guest computes line items and returns them. It does not write to your ledger, does not call the payment processor, does not send email. Anything with a side effect stays in the trusted orchestrator, where you control it.
3. Commit once, transactionally, on the key. Insert line items and the run record together, with a uniqueness constraint on the idempotency key. A duplicate run loses the race and becomes a no-op instead of a second invoice.
4. Treat the payment step as a separate idempotent action with its own key. Every serious payment processor supports idempotency keys; use one derived from the invoice identity, not from a request identity.
5. Alert on digest mismatch, don't silently overwrite. If a retry produces a different result digest for the same key, something changed that shouldn't have. That's an incident to investigate, not a value to update in place.

Fail closed, never partial

The worst billing outcome isn't a job that fails. It's a job that half-succeeds. A rating process that OOMs after emitting sixty percent of a tenant's line items leaves behind an invoice that looks complete, sums to a plausible number, and undercharges by forty percent — or, if the failure hit a discount stage, overcharges. Nothing is marked broken. There's no error page. The invoice just goes out, wrong, and the discrepancy surfaces months later during reconciliation, when the events that would have explained it have rolled off retention.

Shared workers make this failure mode more likely, because a shared worker's failures are noisy and correlated. One tenant's enormous event set triggers the host OOM killer, and the kernel reaps whichever process looks tastiest — which may well be a completely unrelated tenant's rating job, mid-write. The blast radius of one customer's pathological month is every customer whose job happened to be co-resident.

A microVM per run bounds this cleanly. The guest has a fixed vCPU and RAM allocation baked into its snapshot, so an oversized run exhausts its own guest and nobody else's. A TTL guarantees a wedged run is reclaimed rather than lingering. And because the guest is pure — computing line items with no ability to write to your ledger — a crash simply means no result was returned. Failure is total by construction, which for a financial pipeline is precisely what you want.

# Fail closed: results are only committed if the run produced a COMPLETE,
# self-consistent set of line items. A partial invoice is worse than none.
def commit_if_whole(run_record: dict, result: dict, expected_event_count: int):
    lines = result["lines"]

    # 1. Did the guest actually finish? (exit code was already checked)
    # 2. Does the rating engine's own accounting agree with what we sent in?
    rated = sum(l["event_count"] for l in lines)
    if rated != expected_event_count:
        raise IncompleteRun(
            f"rated {rated} of {expected_event_count} events -- refusing to commit"
        )

    # 3. Sanity bounds. A 40x jump month-over-month is either a real spike
    #    or a broken formula; either way a human should look before we charge.
    total = sum_decimal(l["amount"] for l in lines)
    if exceeds_variance_threshold(run_record["tenant"], total):
        return quarantine_for_review(run_record, lines, total)

    # 4. Single transactional commit, keyed on idempotency. A duplicate run
    #    loses the uniqueness race and becomes a no-op, not a second charge.
    with ledger.transaction() as tx:
        tx.insert_run(run_record, on_conflict="do_nothing")   # unique key
        if tx.run_was_inserted:
            tx.insert_line_items(lines)

# Everything above runs in the TRUSTED orchestrator. The sandbox never had
# the ability to write a line item, so it could never write half of one.

Shared worker vs container-per-run vs microVM-per-run

• Cross-tenant correctness — Shared worker: a `WHERE tenant_id = ?` and a module-level cache are the only barriers; a dropped predicate or a mis-keyed memo produces a wrong invoice, not an error. Container per run: process-level separation helps, but a shared kernel and shared credentials still make the whole event store reachable. MicroVM per run: only one tenant's events are present in the guest, so a missing filter has nothing else to return.
• Custom pricing rules — Shared worker: a customer's formula evaluates inside the process that rates every other customer; the evaluator's sandbox is your security boundary. Container: better blast containment, but kernel-shared and typically credential-rich. MicroVM: hardware-virtualized guest with its own kernel; the rule runs against one tenant's data with no shared cache to influence.
• Determinism for audits — Shared worker: the environment drifts between deploys, so re-running a period compares yesterday's data against today's dependency tree. Container: image tags help, but layer caches and base-image drift still bite. MicroVM: every run restores a pinned snapshot generation — byte-identical interpreter, libraries, and locale data — so a re-run is a genuine replay.
• Partial-failure behaviour — Shared worker: a co-resident tenant's memory blowup can get an unrelated job OOM-killed mid-write, leaving a partial invoice that looks complete. Container: cgroup limits bound memory, but shared-kernel pressure remains. MicroVM: fixed vCPU/RAM per guest plus a hard TTL; a run either returns a complete result or returns nothing.
• Idempotency surface — Shared worker: side effects are scattered through long-lived code, so "did this already commit?" is genuinely hard to answer. Container: same problem, smaller box. MicroVM: the guest is pure — compute only — so exactly-once lives in one transactional commit in the orchestrator.
• Cost of isolation — Shared worker: near zero per job, which is why everyone starts here. Container: image pull and cold process start per run. MicroVM: on PandaStack a create is 179ms p50 / ~203ms p99 because every create restores a baked snapshot (~49ms for the restore step) rather than cold-booting (~3s, only at bake time). A fifth of a second per invoice is not the expensive part of billing.

The capacity question comes up immediately, because billing is bursty by nature — most of your runs happen in a narrow window at the start of a cycle. Each PandaStack agent pre-allocates 16,384 /30 network subnets, so concurrency is bounded by host CPU and memory rather than by network slot exhaustion. Rating ten thousand tenants means ten thousand short-lived guests, scheduled across your fleet, each existing for the seconds its computation takes.

Ephemeral VMs as audit artifacts

There's a bonus that isn't obvious until you've lived through a billing dispute. When each rating run is its own machine, the run becomes a discrete, describable object rather than a smear of log lines interleaved with a hundred other tenants' output.

• Attributable logs — every line of stdout from a run belongs to exactly one tenant and one period. No grep-by-tenant-id across a shared worker's log stream, no risk that the log line you're quoting to a customer came from someone else's job.
• Honest resource accounting — CPU seconds and peak memory for a run describe that run. When a customer's formula is the thing that's slow, you can demonstrate it rather than assert it.
• A reproducible environment reference — the run record names a snapshot generation, and restoring it gives you the exact machine that produced the number under dispute.
• Clean failure semantics — a run either completed and returned a result or it didn't. There's no ambiguous middle state where a long-lived worker might have partially applied something.
• Sharp boundaries for retention — the guest is destroyed with the tenant's event data inside it. What persists is what you deliberately extracted: line items, the run record, the digest.

None of this is exotic infrastructure. It's the same pattern as running per-tenant ETL transforms or per-tenant analytics queries in isolated guests — see /blog/microvm-per-tenant-etl-pipeline-isolation and /blog/microvm-per-tenant-analytics-query — applied to the pipeline where being wrong costs the most. If you're also generating the invoice documents themselves, /blog/microvm-per-tenant-pdf-invoice-generation covers that half, and /blog/per-tenant-database-isolation covers keeping the underlying event stores separate in the first place.

The summary

Billing is the workload where isolation and correctness are the same property. A cross-tenant read in a rating engine doesn't leak data that you later contain — it produces an invoice built partly from someone else's numbers, charges it, and books it as revenue. The defense that scales isn't more careful `WHERE` clauses; it's making the wrong number impossible to compute by ensuring the process doing the computing has only one tenant's data in scope.

One ephemeral microVM per rating run gets you four things at once. Structural tenant isolation, so a dropped filter is a no-op rather than an incident. A real boundary around customer-defined pricing formulas, which are arbitrary code no matter how friendly the field label is. A pinned, snapshot-backed environment, so "re-run last quarter" is a genuine replay rather than an approximation — and a digest mismatch becomes an alert rather than an audit finding. And clean failure semantics: a pure guest with a fixed memory budget and a hard TTL either returns a complete result or returns nothing, while the trusted orchestrator commits once, transactionally, keyed on idempotency, so a retry can never become a second charge.

The cost is a fifth of a second per run. The alternative is an incident review where the person asking the hardest questions works in finance.