Best Firecracker Monitoring & Observability Tools (2026)

Firecracker will happily boot you a hardware-isolated guest in tens of milliseconds and then tell you almost nothing about how it's doing unless you go and ask. That's by design — it's a minimal virtual machine monitor, one process per guest, and observability is a thing you bolt on around it rather than a dashboard it ships with. Which is fine right up until you're running a fleet, a fraction of creates start failing, and you realize you have no idea which host, which template, or which layer is at fault. This post is a 2026 field guide to monitoring Firecracker: what signals actually exist (from the VMM's own metrics down through the host and up through your control plane), the cardinality trap that eats naive setups alive, and the tooling landscape you'd assemble to turn all of it into an SLO dashboard you can trust at 3am.

The uncomfortable truth of microVM operations is that 'the VM booted in 49ms' is a great line for a launch post and completely useless as an operational signal — it's only impressive until the one that didn't boot is invisible to you. A p50 you can screenshot is not the same as a p99 you can alert on, and neither tells you why the tail is fat. Good monitoring is the difference between a platform that's fast on the happy path and one you can actually run.

What you're actually monitoring: four layers

Before reaching for tools, name the signals — because 'monitor Firecracker' spans four distinct layers, and most gaps come from watching one and assuming it covers the others. Each layer answers a different question, and a healthy fleet needs all four wired up.

• The VMM itself — Firecracker's own metrics: per-device counters (block, net, vsock queues and throughput), API request counts and latencies, and crucially the seccomp and fault counters that tell you when a guest tripped a filter or the VMM took a signal. This is the layer only Firecracker can see, and it's emitted as a JSON metrics stream (see below).
• The host / hypervisor layer — KVM exits (the vmexit rate is a direct read on how hard the guest is making the host work), cgroup CPU and memory accounting per VM, major/minor page faults, OOM-kill events, and the network plumbing: tap-device and veth counters, iptables/nftables packet and drop counts. This is where 'the fleet is unhealthy' usually shows up first.
• The boot / restore path — the number that defines a sandbox platform: time-to-create, broken down. Snapshot-restore latency, UFFD page-fault service rate on streamed memory, netns/pool allocation time, and the boot-failure rate. A create is a pipeline; you want each stage timed, not just the total.
• The control plane — scheduling decisions, per-host capacity and lease health, queue depth, and the business-shaped signals (creates per tenant, failures per template). This is the layer your API emits and where SLOs ultimately live.

Firecracker's built-in metrics (the JSON stream)

Start at the source. Firecracker emits its own metrics as a stream of JSON objects written to a file or FIFO you configure at boot (via the machine config's metrics path, or the equivalent API call). Each flush is a JSON document with nested counters: per-block-device and per-net-device queue stats and byte/packet counts, vsock activity, API server request counts and durations, and the security-relevant counters — seccomp filter hits and various fault/signal counters that spike when something in the guest or VMM misbehaves. Because it's a FIFO/append stream rather than a scrape endpoint, the standard pattern is a small sidecar that tails it, parses the JSON, and re-exposes it in a form your metrics stack understands (a Prometheus textfile or an exporter). Verify the exact metric field names and the metrics-configuration mechanism against the Firecracker docs for your version — the schema has grown over releases and the names are not stable across the whole history.

// Point Firecracker at a metrics FIFO/file at boot. Shape is illustrative —
// verify the exact path/field against the Firecracker metrics docs for your
// version (configured via the machine config or the PUT /metrics API call).
{
  "metrics": {
    "metrics_path": "/run/firecracker/metrics.fifo"
  }
}

// A single flushed record then looks roughly like this (fields are
// version-dependent — treat as a pointer, not a schema):
// {
//   "utc_timestamp_ms": 1721203200000,
//   "block":   { "read_count": 128, "write_count": 44, "queue_event_fails": 0 },
//   "net":     { "rx_packets_count": 903, "tx_packets_count": 811 },
//   "vsock":   { "rx_bytes_count": 40960, "tx_bytes_count": 8192 },
//   "seccomp": { "num_faults": 0 },
//   "api_server": { "process_startup_time_us": 1204 }
// }

The host layer: KVM, cgroups, page faults, and the network

Firecracker can't see the host it runs on, and the host is where fleet-wide trouble surfaces first. The signals worth wiring up here are the ones that turn 'creates feel slow' into a diagnosis. KVM exit rate (how often guests trap out to the host) tells you whether a guest is thrashing the hypervisor. Per-VM cgroup CPU and memory accounting — Firecracker is typically run under a cgroup (the jailer sets this up) — gives you real resource attribution and, critically, catches the VM that's about to be OOM-killed. Major page-fault rate matters doubly on a snapshot-restore platform, because restore is fundamentally a controlled storm of page faults. And the network plumbing each microVM gets — its tap device, veth pair, and iptables/nftables rules — carries counters (packets, bytes, drops) that are your only view into per-sandbox connectivity problems and egress abuse.

The good news is that most of this is standard Linux telemetry that existing host-monitoring tools already collect — you're reading /proc, /sys, cgroup files, and netfilter counters, not inventing instrumentation. The work is less 'how do I measure a KVM exit' and more 'how do I attribute it to the right sandbox and roll it up without exploding cardinality.' That attribution problem is the recurring theme of Firecracker observability.

Boot-time and snapshot-restore latency: the number that matters

For a sandbox platform, time-to-create is the SLO, and it is a pipeline, not a scalar. Instrument each stage: network/netns allocation, rootfs clone, VMM process spawn, snapshot load, resume, and the readiness probe that confirms the guest is actually serving. Measure the whole thing as a latency histogram so you can watch p50 and p99 independently, and count failures separately from timing successes — a fast p50 hiding a rising boot-failure count is the classic way a platform looks healthy on a graph while users are getting errors. On a streaming-restore path (guest memory paged on demand rather than fully loaded up front), add a UFFD page-fault service-rate metric and a fault-latency measurement, because that's where a slow object store or a cold cache turns into a slow create. For calibration on why the pipeline breakdown matters, PandaStack's restore step lands around 49ms with an end-to-end create of roughly 179ms p50 and 203ms p99, and the only slow path — a first-ever cold boot of a brand-new template — is around 3s; but those are our numbers, and the point here is that you can only defend numbers like that if every stage is timed.

The tooling landscape, tool by tool

With the signals named, here's the field of tools you'd assemble a Firecracker observability stack from. Everything below is described qualitatively from each project's own documentation — verify current capabilities, exporters, and integrations against their docs, because this space moves and the exact feature matrix is version-dependent.

• Prometheus + a Firecracker metrics exporter/textfile scraper — the default backbone. Prometheus scrapes numeric time series and stores them for querying and alerting. Firecracker doesn't expose a scrape endpoint natively, so the common pattern is a small sidecar/exporter that tails the JSON metrics FIFO and re-publishes it (often via the node_exporter textfile collector, or a purpose-built exporter). Strong at aggregated fleet metrics and alerting; the thing to design around is cardinality — aggregate per-host, don't label per-VM. Verify exporter maintenance and the FC metric fields it maps against the project's repo.
• node_exporter + cAdvisor-style host metrics — for the host layer. node_exporter surfaces standard Linux host metrics (CPU, memory, disk, network, filesystem, and via the textfile collector your custom Firecracker metrics), and cAdvisor-style collectors surface per-cgroup/per-container resource usage, which maps onto per-VM cgroups. Together they cover the KVM-host and resource-attribution signals. Confirm how each attributes cgroup metrics to your specific jailer layout.
• Grafana — the visualization and alerting layer most of this stack renders into. It queries Prometheus (and Loki, ClickHouse, and others) to build the dashboards and alert rules that make the signals legible. Its value is turning scattered metrics into a coherent SLO view; it stores nothing itself. Community Firecracker/KVM dashboards exist as starting points — treat them as templates to adapt, and verify they match your metric names.
• OpenTelemetry (tracing the create/restore path) — where metrics tell you a create was slow, distributed tracing tells you which stage. Instrumenting the create pipeline with OTel spans (allocate netns → clone rootfs → spawn VMM → load snapshot → resume → probe) gives you per-stage latency and a trail across the control plane and agent. It's the right tool for the pipeline-breakdown problem above. Verify the current OTel SDK/collector story for your language and backend.
• eBPF tooling (bpftrace, Parca, Pixie-style) — for the deep visibility metrics can't give you. eBPF lets you observe KVM exits, syscalls, and scheduler behavior at the kernel level with low overhead — bpftrace for ad-hoc one-liners when you're diagnosing a specific host, continuous-profiling tools (Parca-style) for CPU flame graphs across the fleet, and auto-instrumentation tools (Pixie-style) for zero-code visibility. This is your scalpel for the 'why is this host's vmexit rate weird' questions. Verify kernel-version support and overhead characteristics for your hosts.
• Log & metrics sinks (ClickHouse, Loki) — for the high-cardinality, per-event data that does NOT belong in Prometheus labels. A columnar store like ClickHouse is well suited to per-sandbox event rows (create, boot, fail, destroy) that you want to query by tenant/template/host after the fact; Loki does the same shape for logs, indexed by labels and queried with LogQL. This is where the per-VM detail lives so your time-series backend stays lean. Verify ingestion throughput and retention design against your volume.

What scraping the fleet looks like in practice

Concretely, most of this stack comes down to a Prometheus exposition endpoint per host that Prometheus scrapes on an interval, plus a trace pipeline for the create path. Here's the shape of poking at an agent's metrics endpoint by hand while debugging — the kind of thing you'd do before wiring it into Prometheus, or when a host looks unhealthy and you want the raw numbers now.

# Hit a host/agent's Prometheus endpoint directly to eyeball the signals
# before (or instead of) going through Grafana. On a PandaStack agent the
# boot + UFFD-streaming counters are namespaced pandastack_*:
curl -s http://localhost:9100/metrics | grep pandastack_

# Boot latency histogram + UFFD page-fault counters specifically:
curl -s http://localhost:9100/metrics \
  | grep -E 'pandastack_(sandbox_boot|uffd)'

# Standard host layer from node_exporter (KVM/cgroup/network live here):
curl -s http://localhost:9100/metrics \
  | grep -E 'node_(cpu|memory|network|vmstat)'

# And the custom Firecracker metrics re-published via the node_exporter
# textfile collector (your sidecar tails the FC JSON FIFO into a .prom file):
cat /var/lib/node_exporter/textfile/firecracker.prom

What a good sandbox-platform SLO dashboard tracks

Pulling it together, here's the short list of what actually belongs on the dashboard you'd page on — the signals that, if any one goes red, mean users are being hurt. Everything else is supporting detail you drill into from here.

• Create latency, p50 and p99 separately — the headline SLO. Watch the tail, not the median; a healthy p50 with a climbing p99 is a real regression that a single 'average' number hides.
• Boot / restore failure rate — counted independently of latency. Fast failures don't show up in a latency histogram at all, so a create-success ratio is a distinct, must-have signal.
• UFFD page-fault service rate (and fault latency) — on a streaming-restore platform, this is where a slow object store or cold cache becomes a slow create. It's the leading indicator for restore-path degradation.
• netns / network-pool exhaustion — how much of the pre-allocated subnet pool is in use, and how often allocation falls back to the slow cold path. Running the pool dry silently converts fast creates into slow ones.
• OOM kills and per-VM memory pressure — an OOM-killed guest is a create that lied about succeeding. Alert on the kill events, not just on aggregate host memory.
• Per-host capacity, lease health, and scheduler decisions — so you can see a host going stale or a scheduler skewing load before it becomes a capacity incident.
• Seccomp / fault counters from the VMM — a rising seccomp-fault count often means a guest workload hit a filter (a bug, or an abuse attempt); worth surfacing even if it's rarely red.

How PandaStack instruments this (the vendor part)

Here's the flagged vendor section. PandaStack is an open-source (Apache-2.0) Firecracker platform, and because snapshot-restore is the boot path rather than an optimization — every create restores a baked snapshot on demand, there's no warm pool — observability of that path had to be first-class from the start. It emits OpenTelemetry traces of the create/restore pipeline, so the stage breakdown (allocate netns → clone rootfs → spawn VMM → load/stream snapshot → resume → probe) is a real trace you can inspect, not a guess. It exposes a Prometheus /metrics endpoint on each agent with the boot and streaming counters namespaced under pandastack_sandbox_boot* (the create-latency histogram) and pandastack_uffd* (page-fault service rate on streamed memory), which is exactly the set the SLO dashboard above wants. And it runs a ClickHouse event sink for the high-cardinality per-sandbox lifecycle events (create, boot, fail, destroy, keyed by tenant/template/host), keeping the per-VM detail out of the Prometheus label set and in a store built to query it — the cardinality discipline this post keeps harping on, applied.

The signals map straight onto the four layers: the OTel trace covers the boot/restore pipeline, pandastack_uffd* covers streaming-restore health, the ClickHouse events give per-tenant/per-template attribution without cardinality blowup, and standard node_exporter-style host metrics cover the KVM/cgroup/network layer underneath. The design context is the usual PandaStack shape — per-microVM networking drawn from 16,384 pre-allocated /30 subnets per agent (so pool-exhaustion is a real metric to watch), restore around 49ms, ~179ms p50 / ~203ms p99 end-to-end, ~3s only on a template's first cold boot. Because it's self-hostable, you point it at your own Prometheus, your own Grafana, and your own ClickHouse — the observability is yours to own, not a black box you get a dashboard screenshot of.

The bottom line

Firecracker gives you a fast, isolated microVM and a terse JSON metrics stream, and everything else — host signals, boot-latency breakdown, per-tenant attribution, the SLO dashboard — is a stack you assemble. Prometheus and Grafana are the backbone, node_exporter and cAdvisor-style collectors cover the host, OpenTelemetry traces the create pipeline, eBPF tooling is your scalpel for kernel-level questions, and ClickHouse or Loki hold the high-cardinality per-event detail that must not live in Prometheus labels. Watch four layers, not one; alert on p99 and failure rate, not just p50; and design for per-VM cardinality before it detonates your TSDB. Because a microVM that boots in 49ms is a nice number — but the one you actually need to see is the one that didn't boot at all, and that only shows up if you built the observability to catch it. Start from the layer that's blind, pick the tool that lights it up, and verify every metric name against its own docs before you page on it.