Knowing Which Customer Costs You Money

The invoice from your cloud provider is one number. Your customers are a list. Everything interesting about running a platform business lives in the mapping between those two things, and for the first year or so most teams do not have one — they have a total, a headcount, and a feeling about which accounts are heavy.

I'm Ajay; I build PandaStack, a Firecracker microVM platform where every sandbox create is a snapshot restore. We meter our own fleet, our customers meter their customers on top of us, and I have personally shipped at least three bugs in this area that cost real money in one direction or the other. So this is a post about cost attribution written from the inside of a meter: which dimensions genuinely exist, which unit almost everybody picks first and why it is wrong, and the two specific classes of bug that make a metering pipeline quietly lie to you.

What the number is actually for

Before picking dimensions it is worth being honest about why you want them, because the answer changes what precision you need. There are four uses and they have very different bars.

• Pricing. You cannot know whether your price covers your cost until you know your cost per unit of the thing you sell. This needs to be right in aggregate, not per tenant.
• Gross margin by account. The question is which customers are underwater. This needs per-tenant accuracy but tolerates a wide error bar — you are looking for outliers, not cents.
• Abuse and capacity. One tenant hammering a fleet shows up as a shape in the attribution data long before it shows up as an outage. This needs freshness more than accuracy.
• Chargeback. You are going to charge a card. This needs to be defensible line by line, reproducible after the fact, and wrong in the customer's favour when it is wrong at all.

Most teams start at the second bullet and only later discover they built something the fourth bullet cannot use. The difference is not query complexity. It is whether every billable resource carries a tenant tag from the instant it is created, and whether every resource that stops emits an event saying so. Those two properties are cheap on day one and archaeological on day four hundred.

Wall-clock vCPU-hours is the wrong unit

Here is the default that nearly everyone reaches for. A sandbox is provisioned with N vCPUs. It lived for T seconds. Bill N × T vCPU-hours. It is easy to compute, easy to explain, and it maps to how you buy instances from a cloud provider, which makes it feel like the honest unit.

It is not, and the reason is that vCPU count on a modern microVM platform is a burst ceiling, not a reservation. Every first-party template we bake gets 8 vCPUs. That is not eight cores set aside for you; it is the number of cores the guest is allowed to see, with cgroup weight sharing the physical cores fairly when several guests want them at once. A sandbox sitting at an interactive prompt waiting for an agent to send it the next command is consuming approximately zero CPU while presenting eight of them.

Bill wall-clock vCPU on that and you have built a systematic bias with two ends. The tenant who spawns a hundred sandboxes, burns four seconds of real compute in each, and tears them down gets charged as if they held eight hundred cores for the duration. The tenant who leaves one sandbox up for eleven days doing nothing but holding state gets charged for eight idle cores that cost you nearly nothing to provide. You have overcharged the workload you want and undercharged the workload that actually consumes your scarce resource.

Idle CPU is nearly free to you. Idle memory is not free at all. Any unit that treats those two the same is going to misprice somebody, and it will be the same somebody every month.

So we bill active CPU: the CPU seconds a sandbox actually burned, read from the cgroup's cpu.stat, clamped so the billed value can never exceed what committed vCPU × wall-clock would have charged. The clamp is doing something specific — it makes the change discount-only. No customer's bill can go up as a result of switching basis, and the reconciler's over-wall invariant stays true by construction rather than by hoping the accounting is right.

The dimensions that actually exist

Strip away the pricing-page poetry and a sandbox platform consumes four physically distinct things. Each behaves differently under idleness, which is the property that matters.

Memory

Memory is the one that binds. A guest that is merely alive — booted, resumed from a snapshot, doing nothing — still holds real pages on your host. You cannot give those pages to another tenant without either the guest noticing or the kernel doing something drastic. Unlike CPU, memory does not go idle in a way you can resell; it goes idle in a way that sits there. This is why memory, not vCPU, sets density, and why it is the first dimension you should be able to attribute.

CPU

CPU is the opposite. Unused cycles evaporate; nobody is worse off because a sandbox did not use its share. Overcommitting CPU is close to free, which is exactly why the vCPU count on a burst platform is a marketing number and the cpu.stat delta is the real one.

Storage at rest

Rootfs clones are copy-on-write, so an ephemeral sandbox's disk footprint is metadata plus whatever it wrote. Durable volumes and managed-database volumes are different — those are provisioned bytes that exist whether or not anyone touches them, which is why they meter on a monthly rate (ours is $0.15 per GiB-month past the tier's included quota) rather than on a per-second compute basis.

Egress

Bytes leaving your network are the one dimension where your provider's bill is genuinely linear and genuinely large. We measure egress per sandbox off the host-side veth in the sandbox's network namespace, keep the counts in the same usage-event table as everything else, and bill $0 for them. That is a deliberate product decision, not an oversight — but the measurement stays, because a workspace pushing hundreds of gigabytes is worth seeing even when it costs the customer nothing.

Committed, resident, and the gap between them

Once you accept that CPU should bill on actual use, the obvious next question is whether memory should too. The intuitive answer is no — memory is committed, you cannot reclaim it, so charge for the commitment. That is where we started, and it is a defensible place to stop.

We did not stop there, and the reason is worth spelling out because it cuts against the intuition. Our rate card says working-set GiB per hour, with no exceptions. Billing one class of workload on committed memory and another on resident memory would mean running a second, unpublished basis behind a card that claims one. So resident GiB-seconds — integrated from a residency scrape, clamped to the committed ceiling exactly like the CPU half — is what the meter charges, across sandboxes, apps and managed databases alike. Managed databases keep their capacity guarantee separately: the db class is never overcommitted on the host and is skipped by the memory-pressure ladder. What changed is that a customer pays for the memory their database actually holds, never more than the tier they provisioned.

The practical upshot is that you want both numbers in your event rows. Store the actual (active CPU seconds, resident GiB-seconds) because that is what you bill, and store the committed (vCPU × duration, provisioned GiB × duration) because that is what a reservation-priced competitor would have charged for the same window. The gap between them is not an accounting artifact. It is the entire value proposition of an idle-cheap platform, expressed as a number the customer can look at.

Overcommitting memory is a bet, and it has a failure mode

Attribution tells you what a tenant costs. Density decides what that cost is. On a microVM fleet the lever is memory overcommit, and it is the single biggest input to your unit economics — bigger than instance type, bigger than region, bigger than anything on your negotiated rate sheet.

Our first admission gate was committed arithmetic: refuse a create when the sum of baked memory across live sandboxes plus the new one exceeds the host budget. Correct in the narrow sense, and it did stop the guest-OOM over-packing it was written for. It also capped a 32 GiB host at roughly seven sandboxes while the host sat more than 90% empty, because every first-party template bakes 4 GiB, Firecracker faults pages in lazily, and a VM that is merely alive holds a few hundred megabytes of real RAM. On one memorable day the fleet refused every create and every wake with a capacity error while 14 booked VMs were using 2.4 GiB of 62 GiB physical.

The fix is to admit against measured residency instead of baked commitments, which turns admission into roughly this shape:

budget      = (MemTotal - reserve) x overcommit
guaranteed  = SUM(memory_mb)          over live db-class VMs   # committed, never overcommitted
resident    = SUM(Rss + hugetlb)      over live other VMs      # measured, 15s scrape
inflight    = SUM(reserve(r))         over admitted-but-unmeasured creates
headroom    = the pressure ladder's first water line

used   = guaranteed + resident + inflight + headroom
admit  <=>  used + reserve(request) <= budget

reserve(r)  = db ? r.memory_mb : max(512 MiB, ceil(r.memory_mb x 0.25))

Three details in there are the difference between a working gate and an outage. The in-flight term exists because a create that has been admitted but has not yet produced measurable residency is invisible to the scrape, and a burst of them would all pass the same stale check — reservations expire on a TTL so a failed create cannot shrink the host forever. The headroom term means admission stops where the pressure ladder starts, so the gate and the reclaim machinery do not fight. And when the residency measurement goes stale, the gate degrades to committed accounting for that tick: over-refusing is the safe direction, over-admitting is the OOM direction the gate exists to prevent.

Be clear-eyed that this is a bet. Twelve days of shadow data across two hosts — 27,200 samples — put resident-over-committed at a median of 0.06 to 0.08, a p95 of 0.20 to 0.27, and a worst observation of 58% of the host with the ladder holding. A 0.25 reserve factor sits around the 94th percentile of that distribution. That is a good bet and it is still a bet: the tail exists, the failure mode is real memory pressure on a host with other people's workloads on it, and the only honest mitigation is a ladder of increasingly aggressive responses plus the discipline to keep one class (databases) out of the wager entirely.

Tag at creation, not in the log pipeline

Everything above is arithmetic. This part is the operational discipline that makes the arithmetic possible at all, and it is where retrofits go to die. A resource must carry its tenant identity from the moment it is created. Not derived later from a request log, not joined at query time against an audit trail, not inferred from a naming convention someone changed in March.

Concretely, on our side: the tenant tag is a key in the sandbox's metadata map, stamped by the control plane at create time and carried on the row for the resource's whole life. The meter reads it back off the row, and a row with no tenant tag is skipped — it bills nobody. That is the correct behaviour for a warm-pool or platform-internal VM and a silent revenue hole for anything else, which is why the reconciler treats untagged live resources as a finding rather than a curiosity.

If you are building on top of a sandbox platform, the same rule applies one level up. Your customers are not our customers; we see one workspace where you see four hundred accounts. The metadata map is the hook for your own attribution:

import os
from pandastack import Sandbox   # pip install pandastack; PANDASTACK_API_KEY in env

def run_for_tenant(tenant_id: str, plan: str, source: str):
    # The tag goes on at creation. There is no later moment at which you can
    # recover "which customer was this for" without guessing, and guessing is
    # how you end up with a spreadsheet nobody trusts.
    sbx = Sandbox.create(
        template="code-interpreter",
        ttl_seconds=900,          # an END event that fires even if you crash
        metadata={
            "tenant": tenant_id,  # who pays
            "plan": plan,         # what they pay under
            "job": "notebook",    # what kind of work this is
            "req": os.environ.get("REQUEST_ID", ""),  # join key back to your logs
        },
    )
    try:
        return sbx.exec(f"python3 -c {source!r}")
    finally:
        sbx.kill()                # the END event you control

# The cheapest attribution audit there is: anything alive and untagged is a
# cost you cannot assign to anyone. Run it on a schedule, alert on non-zero.
for sbx in Sandbox.list():
    if not sbx.metadata.get("tenant"):
        print("UNATTRIBUTED", sbx.id, sbx.template, sbx.created_at)

One design note on that metadata map. Tags that decide money should be set by your platform code from an authenticated identity, never accepted verbatim from the caller. We learned this on the workload-class side: the signals that classify a sandbox as an app are platform-set — stamped by the deploy pipeline, or implied by a reserved template prefix — precisely so a user cannot relabel their own workload into a different bucket by passing a metadata key.

Start events are easy. End events are where the money leaks

Every metering pipeline records creation correctly, because creation is a synchronous request that somebody is waiting on. The interesting failures are all on the other end, and they come in two flavours that fail in opposite directions.

Hazard one: the end event that never fires

A resource goes away — killed, reaped by TTL, lost with its host, deleted by a path that forgot to call the meter — and nothing closes the window. If your meter computes cost as now minus start, it now bills forever, against a resource that does not exist, to a customer who will eventually notice. The variant that is harder to catch is partial: a resource that hibernates and whose sleep window is billed at running rates, which is exactly the over-charge that scale-to-zero is supposed to remove.

The structural fix is to stop computing cost from a start timestamp at all. Keep a per-resource watermark of the last time it was billed, emit closed slices between the watermark and now, and advance the watermark under a compare-and-swap so two writers cannot bill the same window twice. Every lifecycle transition that ends a running period — hibernate, pause, delete — finalizes the open window up to that instant and pins the watermark there; wake resets it to wake-time. The sleep gap is then billed by neither side, not because someone remembered to subtract it, but because no writer owns it.

Advance the watermark before writing the usage row, not after. A crash between the two then loses at most one interval of billing, which is an undercount. The other ordering double-bills, which is a refund and an apology.

Hazard two: the row the meter silently skips

This one is nastier because it looks like nothing. If the anchor timestamp a meter computes its window from can be corrupted, the window collapses and the resource stops billing — permanently, with no error, no alert, and a perfectly healthy-looking VM serving traffic.

We hit this in production. An update path rewrote a sandbox's created_at column from a struct whose creation time had never been populated, stamping the Go zero value — year one — onto live rows. The window function returned a zero duration, the code bailed before reaching the watermark swap, and the watermark never advanced. An always-on 4 GiB managed Postgres billed zero seconds for its entire life. The mirror-image corruption is worse in the other direction: a row map with no created_at key yields the Unix epoch, and a meter that trusts it would bill a fifty-six-year window at full rate on the very next tick.

The defence is a floor. Define an epoch before which no resource of yours can plausibly have been created, treat any anchor below it as corrupt rather than as history, and when a tenant-owned row has no usable anchor at all, repair it — stamp the watermark at now, forfeit the unreconstructable past, log loudly. Forfeiting is the right call: the past window cannot be reconstructed, so guessing at it means retro-billing a customer for a period you cannot evidence. Skipping the row forever, which is what we were doing, is a permanent revenue leak dressed up as caution.

The aggregation query

The data model that survives all of the above is boring on purpose: an append-only usage-event table where each row is one resource, one closed window, one tenant tag, and both bases. Slices are duration-additive, so the sum of a resource's slices plus its final stopped event equals its whole-lifetime total — which is a property you can write a test for.

-- Row shape: ts, workspace, sandbox_id, template, event,
--            cpu_count, mem_mb, duration_sec,
--            cpu_seconds, gb_seconds, cost_micros, net_tx_bytes
-- 'event' is one of created | metered | stopped | egress | storage.

-- Per-tenant rollup: what you bill, next to what a reservation-priced
-- provider would have billed for the same windows.
SELECT workspace                                        AS tenant,
       SUM(cpu_seconds)                                 AS active_cpu_seconds,
       SUM(gb_seconds) / 3600.0                         AS gib_hours,
       SUM(cpu_count * duration_sec)                    AS committed_cpu_seconds,
       SUM((mem_mb / 1024.0) * duration_sec) / 3600.0   AS committed_gib_hours,
       SUM(cost_micros) / 1000000.0                     AS usd
FROM usage_events
WHERE event IN ('metered', 'stopped')
  AND ts >= :from AND ts < :to
GROUP BY workspace
ORDER BY SUM(cost_micros) DESC;

-- Same table, one GROUP BY away from "which resource inside this tenant is
-- expensive" -- the question a customer actually asks when the bill surprises
-- them. Cap the LIMIT server-side; nobody needs an unbounded scan here.
SELECT sandbox_id, MAX(template) AS template,
       SUM(cost_micros) / 1000000.0 AS usd, MAX(ts) AS last_seen
FROM usage_events
WHERE workspace = :tenant AND ts >= :from AND ts < :to
GROUP BY sandbox_id
ORDER BY SUM(cost_micros) DESC
LIMIT 20;

One detail worth stealing: decide early whether a failed aggregate is allowed to blank the page. On a billing view, an empty chart sitting next to a non-zero month-to-date total reads as a confident false statement about the customer's usage, so surface the error explicitly rather than rendering a zero that looks like an answer.

Reconciliation: assume the meter is lying

Every metering bug I have described was found late. Not one of them announced itself. That is the nature of the domain: a meter that is wrong produces numbers, the numbers look like numbers, and the pipeline downstream of it is perfectly happy. The only defence that scales is a separate job whose entire purpose is to disbelieve the meter on a schedule.

Ours runs a few minutes after boot and then every six hours over a trailing window, checking five invariants against lifecycle ground truth. It writes its findings to a table so the history is queryable, and it is deliberately read-only — the smoke detector, not the sprinkler.

1. PHANTOM: meter events timestamped after the resource's deletion, past a slack window. You are billing the dead.
2. OVER-WALL: a live resource whose summed CPU or GiB seconds exceed what its provisioned shape could physically have produced over the overlap. You are billing more than wall-clock.
3. SILENT-ZERO: a resource running right now, older than the window, with zero meter events in it. The reverse leak, and the one nobody alerts on.
4. QUEUE-STUCK: decided-but-unpushed events older than a couple of hours. The payment-processor push has stalled and revenue is sitting in a queue.
5. RATE-DRIFT: cost values that match no rate table within tolerance. Something is mis-rating, which usually means two copies of the rate card have drifted apart.

The slack window matters more than it sounds. A meter that runs on an interval will legitimately emit an event a few minutes after a deletion — that is the final flush — and totals can legitimately exceed exact wall-clock by up to one metering period. Set the slack to absorb that and no more, or your invariant net becomes an alert nobody reads.

Two more habits pay for themselves. Decide a row's billed amount once and persist that decision before calling the payment processor, so every retry reuses the same payload and idempotency key. And bound retroactivity: when we first wired up metered overage, every undecided row an org had ever accumulated became chargeable the moment it became billable, which would have retro-charged brand-new customers for their entire free trial on their first invoice. The bound is a subscription-start timestamp with no fallback — when it is missing we apply no bound rather than guessing one, because forfeiting revenue beats inventing a charge.

One rate card, or you will be explaining yourself

A closing point about what you do with the number, because attribution has a way of leaking into pricing. We used to run three rate cards: sandboxes anchored to one competitor, apps to another, managed databases to a third. Every card was individually defensible. Collectively they meant the same Firecracker VM on the same host billed 13.5 times differently for CPU depending on which product name it had been created under, and the whole scheme needed a paragraph of explanation on the pricing page.

That got collapsed to one card — $0.054 per vCPU-hour of active CPU and $0.0162 per GiB-hour, the same for sandboxes, apps and databases — while it was still cheap to do, before any of it was load-bearing. The workload classes still exist for reporting and capacity policy; they no longer touch price. There is a test that pins every class to identical rates so a future edit cannot quietly reintroduce a per-class price. If your cost model needs a diagram, your customers will assume the diagram is hiding something, and often they will be right.

The summary

Cost attribution is not a reporting problem you can solve at query time. It is a set of properties your platform either has or does not: a tenant tag applied at creation from an authenticated identity, an event for the end of every billable window as well as the start, a watermark instead of a start-timestamp subtraction, and a reconciler that assumes all of the above is broken.

Pick the dimensions from physics rather than from your instance bill. Memory is committed and cannot be resold while idle, which makes it the binding constraint on density and the number that decides whether your unit economics work. CPU evaporates when unused, so bill what was burned and clamp it below the committed ceiling so the change can only ever be a discount. Storage at rest and egress are separate meters with their own units, and if you do not bill for one of them, make sure your meter agrees with your pricing page in that direction.

And expect to find leaks. Not because your team is careless, but because a metering pipeline's failure mode is producing a plausible number instead of an error. The bug that billed a live database zero for its entire life looked exactly like a database that was not being used. The only reason we found it is that something else was watching.