How to monitor a sandbox fleet: the metrics and alerts that actually catch problems

The first monitoring dashboard I built for the sandbox fleet was beautiful and almost entirely useless. Twelve panels. CPU per host, memory per host, a request-rate graph, a big number for sandboxes running. When we had our first real incident — creates timing out on one host while the rest of the fleet sat idle — not one of those panels moved in a way that told me anything. Host CPU looked fine because the stuck process was blocked on IO. The sandbox count looked fine because the stuck sandboxes were still counted as alive.

I'm Ajay; I run PandaStack, a Firecracker microVM platform. This is the monitoring setup I ended up with after a couple of years of deleting panels that never fired and adding ones that did. It is short on purpose. If you operate any fleet where users create and destroy compute quickly, most of it transfers directly.

Why host metrics lie to you here

Classic host monitoring assumes long-lived processes. You watch a service's CPU and memory and latency, and when something goes wrong those move. A sandbox fleet breaks that assumption in two ways.

First, the interesting unit is the create, not the host. A create is a short pipeline — allocate a network slot, clone a rootfs, fork the VMM, restore a snapshot, wait for the guest to answer — and it can fail or stall at any of those steps while the host looks completely healthy. Aggregate host CPU tells you nothing about the p99 of that pipeline.

Second, memory is committed, not used. A microVM with 4 GiB of guest RAM has 4 GiB reserved from the host's point of view long before the guest touches most of it. So "free memory" on the host is a number that can look terrible while everything is fine, and can look acceptable right up until the moment a burst of page faults makes it real. You need a pressure signal, not a free-bytes signal.

Where the metrics come from

Each PandaStack host agent serves a Prometheus endpoint, and the control-plane API serves its own. They are separate listeners on separate ports so you can scrape the data plane and the control plane with different intervals and different alert routing.

# on a host agent
PANDASTACK_METRICS_LISTEN=:9100 pandastack-agent

# on the control-plane API
PANDASTACK_METRICS_LISTEN=:9101 pandastack-api

# what a host is actually publishing right now
curl -s http://localhost:9100/metrics | grep -E '^pandastack_(sandbox|uffd|host)'

Everything below uses the real metric names from that endpoint. If you are running the open-source agent, they will match. If you are on the hosted product you get the same signals through the dashboard, but the reasoning is the same and I would rather show you the raw thing.

Signal 1: boot latency, as a histogram, per host

This is the one metric I would keep if I could only keep one. A create either completes in a couple hundred milliseconds or something is wrong, and the failure modes show up as latency long before they show up as errors.

- alert: SandboxBootLatencyHigh
  expr: |
    histogram_quantile(0.99,
      sum by (agent_id, le) (
        rate(pandastack_sandbox_boot_duration_seconds_bucket[5m])
      )
    ) > 1.5
  for: 10m
  labels: { severity: page }
  annotations:
    summary: "p99 sandbox boot > 1.5s on {{ $labels.agent_id }}"

Two details matter. Group by host, not fleet-wide — a single sick host gets averaged into invisibility across twenty healthy ones, and "one host is bad" is the most common real incident. And use a long-ish `for` window. Boot latency is genuinely spiky when a host takes a burst of creates at once, and a five-minute p99 excursion is normal traffic, not an incident.

Signal 2: create failure ratio, not create failure count

Counting failures is a trap, because failure volume tracks traffic volume. What you want is the ratio, and you want it to survive low traffic without producing a divide-by-nothing alert storm at 3am.

- alert: SandboxCreateFailureRatio
  expr: |
    (
      sum(rate(pandastack_sandbox_creates_total{result="error"}[10m]))
      /
      sum(rate(pandastack_sandbox_creates_total[10m]))
    ) > 0.05
    and
    sum(rate(pandastack_sandbox_creates_total[10m])) > 0.05
  for: 10m

That second clause is the whole trick. It says "only evaluate the ratio if we are actually doing more than about three creates a minute". Without it, one failed create during an idle hour is a 100% failure rate and your pager goes off for nothing. I have written this alert wrong at least twice.

Signal 3: host memory pressure, not free memory

The agent publishes a pressure level derived from the kernel's own stall accounting rather than from a free-bytes reading, and a counter of the actions it took in response — evicting, hibernating, refusing to place new work.

- alert: HostMemoryPressureSustained
  expr: pandastack_host_mem_pressure_level > 1
  for: 15m

- alert: PressureActionsFiring
  expr: increase(pandastack_pressure_actions_total[30m]) > 0
  labels: { severity: ticket }

Treat these as two different severities on purpose. Sustained pressure is a capacity conversation you can have during business hours. Pressure actions firing means the host is already shedding work, which is a thing you want a ticket for even when it resolves itself, because it is how you find the tenant whose workload changed shape.

Signal 4: the page-fault path, if you stream memory

This one is specific to the way our restore path works, but the class of problem is general. When a sandbox restores, the guest's memory is not copied in up front; pages are served on demand as the guest touches them, and the ones that are all zeroes are filled locally without fetching anything. So the health of a restore is visible as a ratio between three counters.

# fraction of faults served from a remote chunk fetch (the expensive path)
rate(pandastack_uffd_chunk_fetches_total[5m])
  / rate(pandastack_uffd_page_faults_total[5m])

# fraction served as a local zero fill (free)
rate(pandastack_uffd_zero_fill_total[5m])
  / rate(pandastack_uffd_page_faults_total[5m])

On a healthy host the zero-fill share is large — the overwhelming majority of a fresh guest's memory is untouched — and the fetch share is small and stable. When the fetch share climbs, either the local chunk cache got evicted or a template was re-baked and every host is paying first-fetch cost again. Both are real events worth knowing about.

The two counters I actually alert on here are the ones that mean something is broken rather than merely slow: retries and fatals.

- alert: MemoryFaultHandlerFatal
  expr: increase(pandastack_uffd_handler_fatal_total[10m]) > 0
  labels: { severity: page }

- alert: MemoryFaultRetriesElevated
  expr: rate(pandastack_uffd_fault_retries_total[10m]) > 1
  for: 15m

A fatal in the fault handler kills a running guest. There is no acceptable rate for that, so the alert has no threshold — any occurrence pages. Retries are the leading indicator: they mean fetches are failing and being retried, which is what a slow object-store blip looks like from the inside, minutes before it turns into a fatal.

Signal 5: control-plane view of host liveness

The last signal lives on the API side and it is the one people forget. The scheduler decides where to place a sandbox using a cached view of which hosts are alive. If that cache is answering "this host is gone" for a host that is fine, you reject work you had capacity for.

sum by (result) (rate(pandastack_lease_cache_total[10m]))

Graph it split by result. A steady low miss rate is normal. A step change in the negative result is a symptom that has bitten us for real — a cached negative is not the same as a dead host, and trusting one costs you availability on hardware that is up and idle.

One trap: two different things are called logs

# your application's stdout/stderr, streamed
curl -N -H "Authorization: Bearer $PANDASTACK_API_KEY" \
  "https://api.pandastack.ai/v1/apps/$APP_ID/runtime-logs?follow=1"

# the hypervisor log for a raw sandbox
curl -N -H "Authorization: Bearer $PANDASTACK_API_KEY" \
  "https://api.pandastack.ai/v1/sandboxes/$SANDBOX_ID/logs?follow=1"

What I deleted

• Per-host CPU utilisation as an alert. It is useful context on a dashboard and it has never once been the thing that told me something was wrong. Boot latency got there first, every time.
• A running-sandbox count alert. It tracks customer behaviour, not system health. A quiet Sunday and an outage look identical.
• Disk-free percentage on the host root. We kept alerting on a filesystem that was not where the sandbox images live. Measure the mount that actually holds the data.
• Anything averaged across the fleet. Averages are where single-host incidents go to hide. Group by host or do not bother.

If you are starting from nothing

Scrape both endpoints at fifteen seconds. Build one dashboard with four panels: boot-latency p50 and p99 by host, create rate split by result, memory pressure by host, and the fault-path ratios. Write the three page-severity alerts — boot latency, create failure ratio, fault handler fatal — and nothing else for a month. Add an alert only after an incident where you wished you had it. That rule alone will keep your alert list shorter and more trusted than any monitoring guide, including this one.