SLOs and error budgets for a code-execution platform

We had an incident where the API's availability graph never moved. Every request returned 200; every dashboard was green. And for forty minutes, roughly one in six sandboxes customers created came up with no working network, so their code sat there failing to reach anything until the TTL reaped it. The control plane had done its job perfectly — accepted the request, picked a host, wrote a row, returned an ID. The thing the customer paid for never happened.

I'm Ajay; I run PandaStack, a Firecracker microVM platform where creating a sandbox means restoring a real machine from a snapshot in a couple hundred milliseconds. That incident is why I think most SRE writing on SLOs is quietly wrong for this category. The canonical examples assume a long-lived service where the response is the product. On an execution platform the response is a receipt. What you promised is a working machine, and whether you delivered it is something your load balancer cannot answer.

This is the layer above monitoring — which metrics to scrape and alert on is a separate post, linked below. Here: turning those signals into commitments, budgets, and a policy with teeth.

Why "successful responses over total responses" is nearly useless here

The default availability SLI — the fraction of requests that did not return a 5xx — works when the response body is the deliverable. The response and the product are the same object.

An execution platform has a gap between those two, and every interesting failure lives inside it. `POST /v1/sandboxes` returns 201 the moment the control plane has accepted and placed the work. Everything the customer cares about happens after that: a network slot is configured, a rootfs cloned, the VMM started, the snapshot restored, and the guest comes up and answers. Any of those can fail while the 201 you already sent sits in your success counter looking excellent.

The same gap runs through the whole product. A deploy returns 202 and the build fails eleven minutes later. A managed Postgres create returns immediately and the database is ready somewhere in the next 30 to 90 seconds, or isn't. The HTTP status describes acceptance of an intent; the SLI has to describe fulfilment of it.

Pick SLIs that match what you actually promised

Start from the sentences you would say to a customer about the product, and turn each into something countable. For us there are four.

1. Create success rate — of the sandboxes a customer asked for, what fraction reached a usable state: the guest answered, not that we wrote a database row.
2. Time-to-usable — from the customer's call to the moment their first command could actually run.
3. Exec success rate — of commands submitted to a live sandbox, what fraction were delivered, run, and faithfully reported. Note this says nothing about the exit code.
4. Durability indicators for stateful things — for managed databases and volumes: did the data survive, was a backup taken in the window we promised, could it be restored. Not a rate of requests at all.

The word doing the work is "usable", and each has to be measured at the boundary the customer touches. Here is the same set, sorted into the version that works and the version that flatters you.

• Sandbox create — Good SLI because: it counts requested sandboxes whose guest answered a probe, which is the promise. Bad SLI because: 2xx on the create endpoint measures your control plane's willingness to accept work, and a job that never boots is invisible.
• Time-to-usable — Good SLI because: measured client-side, it covers the whole pipeline including the parts your API does not wait for. Bad SLI because: handler duration ends when you return an ID, a fraction of the real wait.
• Exec — Good SLI because: it counts whether the command was delivered and its result faithfully reported. Bad SLI because: treating non-zero exits as failures burns budget every time a customer's test suite legitimately fails.
• Managed database — Good SLI because: readiness measured at the first successful client connection, plus backup-taken-in-window as a durability indicator. Bad SLI because: "the provisioning job returned" says the orchestration ran, not that Postgres accepts connections.
• Deploys — Good SLI because: platform-attributable build failures over total builds, with user build errors labelled out. Bad SLI because: raw build success rate is dominated by broken code in the repo.
• Host health — Good SLI because: it isn't one. Bad SLI because: heartbeat freshness is a component metric customers cannot perceive, and a dead host is fine if the scheduler routed around it.

With the counters labelled properly the ratio is boring, which is the point. An SLI query should be short enough to read during an incident.

# Recording rules. `outcome` is set where the create pipeline ends,
# not where the HTTP response is written.
groups:
  - name: slo-sli
    interval: 30s
    rules:
      - record: slo:create_success_ratio:rate5m
        expr: |
          sum(rate(pandastack_sandbox_creates_total{outcome="usable"}[5m]))
          /
          sum(rate(pandastack_sandbox_creates_total{fault_domain!="user"}[5m]))

      # Latency SLI as a threshold-and-percentage, not a percentile:
      # creates whose time-to-usable landed in a bucket <= 1s.
      - record: slo:create_fast_ratio:rate5m
        expr: |
          sum(rate(pandastack_time_to_usable_seconds_bucket{le="1",path="restore"}[5m]))
          /
          sum(rate(pandastack_time_to_usable_seconds_count{path="restore"}[5m]))

Measure it from outside, at least once

Internal instrumentation measures the pipeline you believe you have. A prober using the public SDK exactly as a customer would measures the one you shipped — load balancer, auth path, the DNS you forgot about. Run it against production from outside your network and treat its numbers as the tiebreaker.

import time
from pandastack import Sandbox

# Client-side time-to-usable: the number the customer experiences.
# t0 is before the API call; t1 is after a command has actually run
# inside the guest. Everything between is ours to answer for.
t0 = time.perf_counter()
sbx = Sandbox.create(template="base", ttl_seconds=900)

try:
    result = sbx.exec("echo ready")
    t1 = time.perf_counter()

    usable = result.exit_code == 0 and "ready" in result.stdout
    print(f"time_to_usable_s={t1 - t0:.3f} usable={usable}")
finally:
    sbx.destroy()

Emit that pair — a boolean and a duration — and you have both SLIs from the only vantage point that is not self-graded. Ours has caught things internal metrics could not, because much of what breaks sits between the customer and your instrumentation.

The attribution problem, which is specific to this category

Here is what makes execution platforms harder to measure than a normal service. A large share of the events that look like failures are the customer's own code doing what it was written to do. A test suite exits 1. A script asks for more memory than the guest has and the kernel kills it. A build fails on a dependency pinned to a version that no longer exists. Count those against your SLO and you will burn the month's budget by lunchtime for doing your job correctly — and you will learn to ignore the burn, which is worse than having none.

The opposite failure is more dangerous and more tempting. Once you have an "it's user error" label, it becomes somewhere to sweep things. A guest that OOMs because the customer's process asked for too much is user error. A guest that OOMs because we placed it on a host under memory pressure is not, and both look identical from inside. If the classification is made after the fact by whoever owns the SLO, the SLO is fiction.

So write the rule down once, and make it structural rather than editorial.

The platform is responsible for delivering a working execution environment and for faithfully reporting what happened inside it. The platform is not responsible for the exit code.

That resolves nearly every case. Sandbox never became usable: ours. Sandbox became usable, the command ran, exit 137: theirs. Command submitted, connection lost, and we cannot say whether it ran: ours — faithful reporting is half the promise, so an ambiguous result is a platform failure even when the work succeeded. Build failed on a missing module: theirs. Build failed because the package registry was unreachable from the guest: ours, since egress is something we deliver.

Then put the classification in the metric labels, decided in code where the outcome is known, so the split is auditable by anyone rather than assertable by you.

# Every outcome carries fault_domain, set once, at the site that knows.
#   fault_domain="platform"  -> counts against the SLO
#   fault_domain="user"      -> excluded from numerator AND denominator
#   fault_domain="unknown"   -> counts against the SLO, on purpose
#
# "unknown" defaulting to platform is deliberate: an unclassifiable
# failure is a gap in our instrumentation, and it should hurt.

- record: slo:exec_user_fault_share:rate1h
  expr: |
    sum(rate(pandastack_exec_total{fault_domain="user"}[1h]))
    / sum(rate(pandastack_exec_total[1h]))

- alert: FaultAttributionDrift
  expr: |
    sum(rate(pandastack_exec_total{fault_domain="unknown"}[30m]))
    / sum(rate(pandastack_exec_total[30m])) > 0.01
  for: 30m
  labels: { severity: ticket }
  annotations:
    summary: "Over 1% of exec outcomes unclassified — the SLO is going blind"

Picking targets honestly

The target is the number where people start lying, because it wants to be round and impressive and to appear on a marketing page. Two inputs should decide it, and neither is aesthetic.

First, measured historical performance. Compute the SLI over the last quarter and set the target slightly below what it actually was. Not above — you are describing reliability you have demonstrated, not reliability you would like. A target you have never met produces a permanently exhausted budget, which the organisation learns to route around in about three weeks.

Second, what you are willing to be woken up for. An SLO is a standing agreement that some rate of failure is acceptable and everything past it is an emergency. If you are not prepared to get out of bed when the budget burns fast, the target is too tight. That is the honest version of the "how many nines" conversation: nines are not a quality setting, they are a staffing and architecture commitment.

And the cost is superlinear. Each nine removes ninety percent of the remaining allowed failure, and what is left after the easy fixes are the expensive ones — a single-region deployment, a control-plane database that fails over in seconds rather than instantly, one object store, a deploy that involves a human. Bad to decent is retries and health checks. Decent to excellent means deleting single points of failure you rely on for your sanity. Five nines is a promise you make with someone else's weekend.

State latency SLOs as a threshold and a percentage

Do not write "p99 time-to-usable under one second". Write "99% of creates are usable within one second". These sound identical and are not.

The percentile version makes the target a moving statistic of your traffic. It cannot be aggregated across windows or regions without lying — you cannot average percentiles — and "how much p99 did we spend today" has no answer. The threshold version gives latency the same shape as availability: good events over valid events. One budget arithmetic, one burn-rate template, and a number that composes over any window.

The cold-start honesty problem

This one is specific to snapshot-restore platforms, and it is where I see the most quiet dishonesty in the category — including some I have been tempted by.

Our latency distribution is bimodal by design. A create that restores a baked snapshot lands around 179ms at p50 and 203ms at p99, the restore step itself about 49ms. A genuine first cold boot — a template never baked on that host, so the machine boots a kernel from scratch — takes roughly 3 seconds. Not one population: two operations sharing an endpoint.

You have two honest options and one dishonest one.

• Exclude the cold path with an explicit label — a `path="restore"` selector on the SLI, and a separate, looser objective for `path="cold"`. My preference: the two paths have different causes, fixes and acceptable durations, and merging them destroys information.
• Set one threshold above the slow path. "99% of creates usable within 5 seconds" is defensible and covers both. It is also far weaker than what you could truthfully say about the fast path, and it will not notice restores degrading from milliseconds to a second.
• The dishonest option: one bucket, one aggregate, and enough fast restores to bury the slow ones under a percentile that never moves. Not a rounding choice — marketing with a Prometheus query attached.

The same logic covers every operation whose cost is genuinely bimodal. A same-host fork runs 400–750ms; cross-host is 1.2–3.5s because memory comes over the network. A managed Postgres create takes 30–90 seconds. One SLO per operation class, with a threshold a customer would recognise as the promise for that thing.

Error budgets and burn-rate alerting

The budget is the target restated as a permission: a 99.9% objective over 30 days is permission to fail 0.1% of valid events. Framed that way, the useful question stops being "are we above the line" and becomes "how fast are we spending, and will we run out before the window resets".

That rate is the burn rate: how much faster than budget-neutral you are consuming. A burn rate of 1 exhausts the budget exactly at the end of the window; 14.4 exhausts it in about two days. Alerting on burn rate rather than the SLI itself separates a short severe outage from a long mild degradation without needing two metrics.

The practical implementation is multi-window, multi-burn-rate: pair a fast, sensitive window with a long, confirming one, so a one-minute blip does not page and a real outage does within minutes. Two tiers is enough. More than that is a hobby.

# Illustrative: a 99.9% create-success objective over 30 days.
# Derive the target from YOUR measured history; this is a worked example.
#
# error_ratio = 1 - slo:create_success_ratio
# burn_rate   = error_ratio / (1 - 0.999)

groups:
  - name: slo-burn
    rules:
      - alert: CreateSLOFastBurn
        # 14.4x sustained -> ~2% of the 30d budget gone in one hour.
        # The short window keeps it from firing on an already-resolved spike.
        expr: |
          (
            (1 - slo:create_success_ratio:rate1h) > (14.4 * 0.001)
            and
            (1 - slo:create_success_ratio:rate5m) > (14.4 * 0.001)
          )
        for: 2m
        labels: { severity: page }
        annotations:
          summary: "Create SLO burning 14x — 2% of the monthly budget per hour"

      - alert: CreateSLOSlowBurn
        # 3x over 6h -> a real but non-urgent regression. Ticket, not page.
        expr: |
          (
            (1 - slo:create_success_ratio:rate6h) > (3 * 0.001)
            and
            (1 - slo:create_success_ratio:rate30m) > (3 * 0.001)
          )
        for: 15m
        labels: { severity: ticket }
        annotations:
          summary: "Create SLO burning 3x — something regressed, look this week"

The short second clause is not decoration. Without it, a five-minute outage keeps the one-hour average elevated for the next fifty-five, and you get paged repeatedly for something already over. Requiring both windows to be hot means the alert clears when the problem does.

The half that makes it real: what happens when the budget is gone

Arithmetic is the easy half. An SLO with no consequence attached is a dashboard with self-esteem. What makes a budget mean anything is a written, pre-agreed answer to "what do we do differently when it is exhausted" — decided while everyone is calm and nobody is arguing about a specific feature.

Ours is four lines:

1. Budget healthy: ship normally, risky changes included. The budget exists to be spent on velocity, not hoarded.
2. Below 25%: no changes to the create path, scheduler or storage path without a second reviewer and a staged rollout. Everything else proceeds.
3. Exhausted: feature work on the affected surface stops, and the next item is reliability work chosen from the incidents that spent the budget. A resource allocation rule everyone already agreed to, not a punishment.
4. Exhausted twice in consecutive windows: the target is wrong or the architecture is. Re-derive it from measured performance, or commit to the structural change. Do not simply declare an emergency every month.

The failure mode to watch for is the silent exception — the quarter where the budget is blown, everyone agrees it is serious, and the roadmap does not change. After that happens twice the SLO is decoration, and your on-call engineers know it before you do.

What not to make an SLO

The strongest constraint on a good SLO set is how small it is. Every addition dilutes the ones that matter, and there is a pull toward promoting any internal metric because the graph already exists and looks important.

• Component health. Heartbeat freshness, lease-cache hit ratio, snapshot-store latency, page-fault ratios — diagnostic signals, and they belong in alerting, not objectives. An unhealthy host is not customer-visible if the scheduler placed the work elsewhere.
• Anything the customer cannot perceive. A failure absorbed by a retry that stays inside your latency threshold already shows up in the latency SLO; counting it again burns budget for a system working as designed.
• Resource utilisation. Memory pressure and host density are capacity questions with their own process. As an SLO they conflate "we are short on headroom" with "we failed a customer", which deserve different responses.
• Things you do not control end to end. A customer's build pulling from a package registry shows up in your latency numbers but should not be in your objective — you cannot fix it and should not pretend to.
• Every endpoint you have. Pick the operations carrying the product's promise — create, exec, deploy, database ready — and leave the rest to ordinary alerting. Ten SLOs means no SLOs.

Where to start

Build it in this order. Instrument one SLI end to end — create success, measured to the guest answering — with a `fault_domain` label decided in code where the outcome is known. Run an external prober through your public SDK for an unbiased second opinion. Measure for a month before setting a target. Then set it just under what you demonstrably achieved, convert it to a budget, write the two burn-rate alerts, and write the policy for what happens when the budget is gone.

That last step is the one everybody skips, and the only one that turns a nice graph into something that changes what your team builds next quarter.