Designing Quotas That Don't Wreck Your Users

There are two ways a quota system fails, and they look nothing alike. In the first, an agent loop with a retry bug creates sandboxes in a while-true until something falls over, and the invoice arrives three weeks later with a number on it nobody at either company wants to discuss. In the second, a paying customer's production traffic starts returning 403 at 00:00 UTC on the first of the month, because a counter reset raced a subscription lookup and for eleven minutes the platform believed a real customer was on the free tier.

I have shipped the second one. It is much worse than it sounds, because the customer cannot fix it, cannot route around it, and has no way to know it is a metering artefact rather than something they did. They just see their thing be down while your status page is green.

I'm Ajay, I build PandaStack — a platform where people run untrusted code and long-lived apps in Firecracker microVMs, and where one misbehaving tenant can make the host miserable for everyone else. I have got quotas wrong in production more than once. This is the design I would give someone starting from scratch: what to count, where the check lives, and — the part almost nobody writes down — what should happen when someone hits the limit.

You need three counters, and they do different jobs

The most common quota mistake is picking one number and calling it the limit, which protects one thing and quietly fails at the other two. There are three distinct quantities, and each defends against a different disaster:

• What it measures — Concurrency: how much is alive right now (running sandboxes, committed vCPU, committed GiB). Consumption: how much has been burned this period (CPU-seconds, GiB-hours, egress bytes). Rate: how often an operation happens (creates per hour, deploys per day).
• What it protects — Concurrency: the fleet. It is the only limit that stops one tenant filling a host. Consumption: the invoice. It is the only limit that bounds spend. Rate: the control plane, and the user from themselves. It is the only limit that catches a loop before either of the other two notices.
• How it fails — Concurrency: a leaked counter permanently locks the tenant out even though nothing is running. Consumption: it is eventually consistent, so it is always a little wrong and always a little late. Rate: it punishes legitimate bursts if the window is too short, and misses real loops if it is too long.
• How you reset it — Concurrency: you don't; it is derived from live state and must be reconciled against reality. Consumption: on a billing period boundary, which is exactly where the midnight-403 bug lives. Rate: on a rolling or fixed window measured in minutes to hours.
• What a user does about it — Concurrency: shut something down. Consumption: upgrade, or wait for the period. Rate: back off and retry, which is the only one of the three where retry is the correct response.

Concurrency without consumption means one tenant runs three sandboxes for thirty days and hands you a bill you never approved. Consumption without concurrency means they start four hundred sandboxes in a minute and hurt every other tenant on the host long before the meter catches up. And neither stops an agent that creates and immediately deletes a sandbox ten times a second, which burns almost nothing of either while happily melting your control plane's database.

The unit I would not use: requests

Per-request quotas and per-request pricing are everywhere, and they are the wrong unit for anything that runs your code rather than dispatching it. We looked at shipping request tiers and turned them down, for two structural reasons.

First, a request is not a cost. A 2ms request returning a cached string and a 2s request rendering a PDF bill identically under request pricing while costing orders of magnitude apart. Request counting is a proxy that platforms adopt because they cannot see inside the workload; if you can meter real CPU-seconds and resident memory, using it anyway throws away the good signal in favour of the bad one.

Second, it double-charges. Serving that request already burned metered CPU and memory, so a request fee bills the same physical resource twice — and it punishes exactly the workload shape you want on your platform: chatty, cheap, well-optimised services. The team that spent a sprint getting p50 from 40ms to 4ms should see their bill fall.

Requests remain a fine abuse signal — a million a day from an account that signed up an hour ago tells you something. Just do not put a price on it.

Where the check lives: admission, and only admission

There is exactly one place you can say no cheaply: the control plane's admission path, the moment a create request arrives and before you have allocated a network slot, reflinked a disk, or forked a hypervisor. Every later refusal costs you the work you already did and hands the user a resource that exists for a moment and then does not.

The hard part is being correct while dozens of creates arrive concurrently. The naive implementation — read usage, compare to the cap, insert — is a textbook race: two requests read 9 of 10, both pass, both insert, you are at 11. Under an agent that fans out twenty creates in parallel you do not overshoot by one, you overshoot by twenty.

The fix is reserve-then-commit: take the reservation and the check in one atomic statement, let the database serialise them, and release only on a terminal outcome.

-- One statement: reserve capacity only if it fits. Zero rows back
-- means "over quota" -- there is no window between the check and
-- the write for a concurrent create to slip through.
WITH quota AS (
  SELECT max_vcpu, max_memory_mb FROM org_limits WHERE org_slug = $1
)
INSERT INTO resource_reservations (id, org_slug, vcpu, memory_mb, state, expires_at)
SELECT $2, $1, $3, $4, 'reserved', now() + interval '5 minutes'
FROM quota q
WHERE (
  SELECT COALESCE(SUM(vcpu), 0) FROM resource_reservations
  WHERE org_slug = $1 AND state IN ('reserved', 'active')
) + $3 <= q.max_vcpu
AND (
  SELECT COALESCE(SUM(memory_mb), 0) FROM resource_reservations
  WHERE org_slug = $1 AND state IN ('reserved', 'active')
) + $4 <= q.max_memory_mb
RETURNING id;

Note that the incoming request is added into each sum. Too obvious to state, until you meet the version that does not — a bug I shipped. A check written as "if current usage is at or above the cap, refuse" never counts the thing being created, so a cap of 10 GiB really means ten plus the largest single guest minus one. Where the standard template bakes 4 GiB, an advertised 10 GiB cap cheerfully admitted 12: at the third create usage is 8, 8 is less than 10, welcome aboard. We found it the way everyone does, in the logs of an account that had used considerably more than it was allowed.

The reservation needs an expiry and a sweep. Anything reserved that never became active — a create that failed at the hypervisor, a process SIGKILLed between the INSERT and the launch — is a phantom that eats a user's quota forever. A reconcile loop that expires stale reservations and re-derives the concurrency counter from the actual fleet is not optional. Concurrency is not a number you own; it is a fact about the world that your number is trying to describe.

-- Runs every 60s. Two jobs: drop reservations that never became
-- real, and heal 'active' rows whose resource no longer exists.
UPDATE resource_reservations
   SET state = 'expired'
 WHERE state = 'reserved'
   AND expires_at < now();

-- The important half: trust the fleet, not the ledger.
UPDATE resource_reservations r
   SET state = 'released'
 WHERE r.state = 'active'
   AND NOT EXISTS (
     SELECT 1 FROM sandboxes s
      WHERE s.id = r.id AND s.status IN ('running', 'paused')
   );

Bias that sweep toward releasing. A quota system that leaks reservations upward locks paying customers out of a platform that has capacity sitting idle, and they will not file a bug — they will file a churn.

What happens at the limit: a ladder, not a wall

This is the part that matters and the part that gets three lines in most design docs. A limit is a policy about how you treat people at their worst moment — mid-incident, at 2am, because something they wrote is misbehaving. The response should be graduated:

1. At 80% of the period budget: tell them. One email, one webhook event, one banner. Not three of each — a quota system that cries wolf trains people to filter it. Include what is consuming the budget, not just the percentage.
2. At 100%: stop new work, let in-flight work finish. Refuse creates. Do not kill running things. The sandbox that is halfway through a build did not do anything wrong, and killing it converts a billing event into a data-loss event.
3. Shortly after 100%: pause or hibernate idle resources rather than deleting them. Snapshot state, stop the meter, keep the disk. The user's project still exists; it is just asleep. This is the step that makes the whole ladder recoverable.
4. After a long, loudly-announced grace period: delete. We use 72 hours, with a second email at the start of the clock that says plainly what will be deleted and when. Anything shorter is a trap for someone on holiday.
5. At any point, on upgrade or a payment method: resume automatically, cancel the clock, clear the state. The recovery path must be one action, and it must not require a support ticket.

The gap between step 3 and step 4 is the entire ethical content of a quota system. Hibernating costs you disk and nothing else; deleting is irreversible and, if your metering was wrong, irreversibly wrong. Make the window generous, the warnings impossible to miss, and the deletion boring — a scheduled job that re-confirms every precondition immediately before it acts, not an inline branch firing on a stale read.

One rule with no exceptions: never hard-block a paying customer's live traffic on a metering artefact. Serving requests for an account whose meter says they are over is a business decision you can reverse on an invoice. Returning 503 to their users is a decision you cannot reverse at all. If you have a payment method on file and a live subscription, the answer to overage is a bill, an email, and a conversation — not a wall.

Soft limits, burst, and not making launch day outage day

Real workloads are not flat. They are quiet for six days and then someone posts on Hacker News. A hard ceiling set at the 95th percentile of normal usage guarantees that the most important hour of the year is the hour your platform says no.

Two ceilings fixes it: a sustained limit the tenant can hold indefinitely, and a burst allowance well above it they can draw on for a bounded window — N concurrent resources continuously, 3N for up to an hour, refilling over the following day. A token bucket does this in about forty lines, and users already have the mental model from CPU credits, so you get the explanation for free.

The same logic applies to rate. A limit of 100 creates per hour enforced as "reject the 101st in any 60-minute window" is fine for a human and terrible for a CI matrix that legitimately fans out 80 jobs in ninety seconds. Enforce over the window you actually care about — minutes, for loop detection — and let the hour-scale number be the sustained one.

Quota state the user can see, before they hit it

A limit the user cannot query is a trap. Three things need to exist, and they are all cheap:

• Quota state in the API. Current usage, the limit, the period, and the reset time — as a first-class endpoint, not a number that only appears in the error body when you have already failed.
• Events, pushed. Warning, exhausted, suspended, resumed. Polling a usage endpoint every thirty seconds so you can find out you are at 80% is a habit worth designing away.
• Per-resource attribution. Percentages tell a user they have a problem. Attribution tells them which of their forty sandboxes is the problem, which is the only form of the information they can act on at 2am.

# What is my current state, and what ate the budget?
curl -s https://api.pandastack.ai/v1/quota \
  -H "Authorization: Bearer $PANDASTACK_API_KEY"
# -> { "period": "2026-08", "resets_at": "...",
#      "concurrency": { "vcpu": { "used": 24, "limit": 32 } },
#      "consumption": { "cpu_seconds": {...}, "gib_hours": {...} },
#      "top_consumers": [ { "id": "sbx_...", "cpu_seconds": 41200 } ] }

# Stop polling. Subscribe once.
curl -s -X POST https://api.pandastack.ai/v1/webhooks/endpoints \
  -H "Authorization: Bearer $PANDASTACK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://hooks.example.com/pandastack",
    "events": ["quota.warning", "quota.exhausted", "quota.resumed"],
    "description": "page oncall before the ladder starts"
  }'

Then route quota.warning somewhere a human reads within the hour, and treat quota.exhausted as a page. Verify the signature and dedupe on the delivery id — there is a full walkthrough of the handler side in the webhooks guide, and the failure modes there apply exactly as much to quota events as to deploys.

The free tier's quota is abuse control wearing a cost-control hat

Free-tier limits look like a spend cap and mostly are not one — the dollars are trivial. What the limit really does is bound the value of a stolen or fraudulently created account, because the people most enthusiastic about your free tier include a category whose business model is running a crypto miner on someone else's electricity.

That reframing changes the design. If the limit is abuse control, the interesting signals are shape, not magnitude: account age, whether a payment method was ever attached, and the duration of sustained full-core CPU. A legitimate build saturates eight cores for four minutes; a miner saturates eight cores for four days. Those separate trivially on duration and not at all on instantaneous magnitude — which is why a memory cap alone cannot stop CPU abuse when every template bakes the same burstable core count.

It also means the free tier is the one place where a slightly aggressive limit is correct: a false positive costs you a signup who has invested nothing yet, and a false negative costs you a host full of xmrig and an email from your cloud provider's abuse team. More on the specific signals in the free-tier abuse post.

Every quota system lies a little

Metering is eventually consistent. Usage rows land seconds or minutes after the work happened, aggregations are cached, and a host that loses network for ninety seconds reports its CPU-seconds late and then all at once. Your quota number is a stale estimate wearing the costume of a fact.

Design the grace so that the lie is never the reason someone's production went down.

That sentence generates most of the advice above. If the number is approximate, the response to crossing it must tolerate the approximation. Warnings can be wrong — send another. Refusing a create can be wrong — the user retries in a minute. Hibernating can be wrong — you wake it. Deleting cannot be wrong, so it goes behind seventy-two hours, two emails, and four re-confirmed preconditions. And 403ing live production traffic cannot be wrong either, which is why the answer to a paying customer over budget is an invoice, not an error page.

The short version

Count three things — concurrency for the fleet, consumption for the invoice, rate for the loops — and do not count requests. Enforce at the admission path with a reserve-then-commit counter that includes the incoming request in the sum, cover every create-shaped endpoint including fork and clone, and reconcile against reality on a loop biased toward releasing. At the limit, climb a ladder: warn, refuse new work, hibernate idle things, delete only after a long clock with receipts. Give the user a queryable quota state, a webhook, and per-resource attribution. And build all of it assuming your meter is a little bit wrong, because it is.