Firecracker Metrics and Logger FIFOs, Explained

Firecracker does not have a `/metrics` endpoint. There is no scrape target, no exporter baked in, no Prometheus handler quietly listening on a port inside the jail. What it has instead is a pair of sinks you configure at boot — one for structured JSON metrics, one for the log stream — and a firm expectation that whoever launched the VMM is standing at the other end of them with a bucket. This is a deliberate design choice, and it moves an entire load-bearing component of your observability stack from "a thing Prometheus does" to "a thing you have to write, correctly, before the guest boots."

I'm Ajay, I built PandaStack (a Firecracker microVM platform), and I have personally watched a metrics collector take down the microVMs it was collecting metrics from — which, I maintain, is the purest form of irony available to an SRE. This post covers how the metrics and logger sinks actually work, why they're push-to-a-file-descriptor instead of pull-from-HTTP, the named-pipe footgun that makes your hypervisor block on `write()` when your collector dies, what's inside the JSON that matters operationally, and how to turn the whole thing into a pipeline that doesn't detonate your metrics backend at fleet scale.

The design: two sinks, configured over the API socket

Firecracker exposes a small REST API over a Unix domain socket, and two of its resources concern observability. One configures the metrics sink — where the VMM periodically flushes a structured JSON document of its internal counters. The other configures the logger — where the VMM writes its own human-readable log lines (API calls it served, device setup, snapshot load and restore, errors), and optionally where the guest's emulated serial console output lands. Both are configured by `PUT`ing a small JSON body naming a path on the host filesystem, and both take effect only for what happens after the call lands.

Qualitatively, the metrics body is essentially "here is a path, write metrics to it," and the logger body is "here is a path, write logs to it, at this level, with or without the level and origin prefixes." I'm describing the shapes rather than quoting a schema on purpose: the exact field names and the set of accepted options have moved across Firecracker releases, and a blog post is a terrible source of truth for an API surface. Check the upstream Firecracker documentation and the OpenAPI spec bundled with the version you actually run, then treat the snippet below as the ordering lesson rather than the field reference.

#!/usr/bin/env bash
set -euo pipefail

SB=sbx-4f2c9a
RUN=/run/pandastack/$SB
API=$RUN/firecracker.sock
mkdir -p "$RUN"

# Two sinks, two pipes: structured counters and the VMM's own log stream.
mkfifo -m 0600 "$RUN/metrics.fifo"
mkfifo -m 0600 "$RUN/logger.fifo"

# CRITICAL ORDERING. The supervisor opens the READ ends first and holds them
# open for the whole life of the VM. Do this before firecracker exists.
# Everything else in this post is a consequence of getting this line wrong.
./pandastack-agent tail-fifo \
  --sandbox "$SB" \
  --metrics "$RUN/metrics.fifo" \
  --logs    "$RUN/logger.fifo" &
COLLECTOR=$!

firecracker --api-sock "$API" --id "$SB" &
FC=$!

# Now point the VMM at the sinks. Field names are version-dependent --
# verify against the Firecracker docs/OpenAPI spec for YOUR build before
# copying this verbatim.
curl -s --unix-socket "$API" -X PUT 'http://localhost/metrics' \
  -H 'Content-Type: application/json' \
  -d "{\"metrics_path\": \"$RUN/metrics.fifo\"}"

curl -s --unix-socket "$API" -X PUT 'http://localhost/logger' \
  -H 'Content-Type: application/json' \
  -d "{\"log_path\": \"$RUN/logger.fifo\", \"level\": \"Info\", \"show_level\": true}"

# Both PUTs must land before you boot or resume the guest. Anything the VMM
# would have emitted before the sinks were configured is simply gone --
# which is exactly the window where the interesting failures live.
trap 'kill $FC $COLLECTOR 2>/dev/null || true' EXIT

Why a FIFO and not a scrape endpoint

The obvious question is why a VMM in 2026 doesn't just serve `/metrics` on a port like everything else. The answer is that Firecracker's entire product is a small attack surface, and an HTTP metrics server is the opposite of that. Serving a scrape endpoint means a listening socket reachable from somewhere, a server loop running inside a process that's deliberately confined by seccomp and a jailer, request parsing on untrusted input, and more code in the thing whose whole selling point is that there isn't much code in it. Every one of those is a line item on a threat model that Firecracker's authors clearly did not want to write.

Pushing to a file descriptor sidesteps all of it. The VMM opens a path you gave it, writes bytes, and never listens for anything. The jailer can hand it exactly that descriptor and nothing else. There's no port, no auth question, no "can the guest reach the metrics server" conversation, and no parsing of anything a hostile party controls. It's the same instinct behind having one Firecracker process per guest and no shared daemon: reduce the number of things an attacker can talk to, down to approximately zero.

Firecracker didn't ship a metrics server. It shipped a file descriptor and a strong implication that you are the metrics server now.

The consequence is architectural, not cosmetic. In a normal service, the metrics system is a passive observer: if Prometheus goes down, your app keeps serving and you lose graphs. Here, the collector is an active participant in the runtime. Your supervisor — the agent process that spawns and babysits each microVM — is the collector, and it is now on the critical path of a running guest. That is a genuinely different reliability posture, and it's why the next section exists.

The footgun: a pipe with no reader blocks the writer

A named pipe is not a file. It is a fixed-size kernel buffer with a reader end and a writer end, and it has two behaviors that matter enormously here. First, opening the write end blocks until someone opens the read end (unless the opener passes `O_NONBLOCK`). Second, once the buffer fills — because the reader is slow, or gone, or never showed up — a `write()` into it blocks until space is freed. That's not an error you can catch and log. That's the calling thread parked in the kernel, indefinitely, waiting for a consumer that may never come.

Now recall who the writer is. It is Firecracker. Which means a metrics flush into a pipe nobody is draining is your virtual machine monitor sitting in an uninterruptible-looking `write()` while a customer's workload waits for it. Your observability stack has achieved the SRE hat trick: it is now an availability dependency, a latency source, and a single point of failure for the thing it was installed to watch. If the collector crashes and nothing reopens the read end, the failure doesn't show up as a missing graph — it shows up as a microVM that stopped doing work, and a graph that is missing because of it. The two symptoms have the same root cause and look nothing alike in a postmortem.

The mitigations are all boring, which is the good news. Pick some combination of these and the footgun stops being one:

• Open the read end before you exec Firecracker, and keep it open — Not "start the collector around the same time." Open the file descriptor first, in the supervisor, then spawn the VMM. There is then never a moment where the pipe exists with a writer and no reader.
• Open the read end non-blocking and drain in a loop — `O_RDONLY | O_NONBLOCK` returns immediately even with no writer attached yet, which lets the supervisor set up the reader before anything else exists. Then read continuously; never let downstream work happen on the same goroutine/thread as the read.
• Hold a keeper write end open too — A reader on a pipe with zero writers sees EOF. If Firecracker closes and reopens, or you race a flush boundary, a naive reader exits and you're back to an unattended pipe. Holding your own `O_WRONLY` descriptor keeps the stream continuous for the life of the VM.
• Never block the drain on downstream — If your parse/enrich/ship pipeline stalls, drop samples and count the drops. Losing metrics is an incident; wedging a paying customer's VM to avoid losing metrics is a bigger one. This trade should be made once, explicitly, in code, not implicitly at 3am.
• Or just use a regular file plus rotation — Writes to a regular file go to the page cache and never wait on a consumer. You pay disk, you pay for rotation and cleanup, and you accept that a full disk becomes the new failure mode. For many fleets this is the correct, boring answer.
• Make the supervisor own the reader's lifetime — Whatever you choose, the process whose death should kill the VM is the process that should hold the read end. Coupling them means the two can't get out of sync; decoupling them means they will.

Named pipe vs regular file: the actual trade

Both sinks accept either. It's worth being deliberate about which you pick per sink — I've ended up with a pipe for metrics (high frequency, want it streamed and gone) and a file for the logger (low frequency, want it on disk when a VM dies mysteriously), and the reasoning goes like this:

• Backpressure — Named pipe (FIFO): a full buffer blocks the writer, and the writer is your hypervisor; a slow or dead reader becomes a guest-visible stall. Regular file: writes land in the page cache and never wait on a consumer, so the VMM is decoupled from your collector entirely.
• Failure mode — Named pipe (FIFO): collector dies, VM wedges. Loud, immediate, and terrifying. Regular file: nobody notices for six weeks and then the disk fills, which wedges every VM on the host at once. Loud, delayed, and also terrifying, but at least it's on your monitoring's turf.
• Data durability — Named pipe (FIFO): nothing is retained; if you weren't reading, the sample never existed. Post-mortem forensics on a crashed VM are impossible unless your collector already shipped the bytes somewhere. Regular file: the last flushes before a crash are still sitting on disk, which is frequently the only evidence you get.
• Disk cost — Named pipe (FIFO): effectively zero, a kernel buffer per VM. This matters at fleet density. Regular file: real bytes per VM per flush interval, plus rotation machinery, plus cleanup on VM teardown that you will forget to write and then discover during a capacity incident.
• Operational complexity — Named pipe (FIFO): requires a correct reader with lifetime coupled to the VM; the code is short but the ordering is unforgiving. Regular file: requires logrotate-style plumbing and a retention policy, which is more moving parts but all of them are parts your ops team already understands.
• Multi-consumer — Named pipe (FIFO): two readers on one pipe split the byte stream between them and corrupt both. Do not casually `tail -f` a live metrics FIFO. Regular file: any number of readers can open it independently, which makes ad-hoc debugging trivially safe.

What's actually in the metrics document

Each flush is a JSON object with a timestamp and a set of nested subsystem objects, roughly one per thing Firecracker emulates or exposes: the vCPU layer (exit counts by type), block devices (read and write counts, queue events, rate-limiter activity), net devices (rx/tx packet and byte counts, tap read/write failures, drops, rate-limiter activity), vsock, MMDS, the API server, seccomp, and the balloon device if you use it. The counters are cumulative, but exactly what "cumulative" means has varied — some versions reset counters per flush so each document is a delta, others accumulate for the process lifetime. That difference completely inverts how you aggregate downstream, so confirm it for your version before you write a single `rate()` query.

Most of these counters are noise until they aren't. The handful that earn a permanent place on a dashboard are the ones that answer a question you'd otherwise have to guess at:

• Rate-limiter throttle counters (block and net) — the single most misread signal in the set. When these climb, your storage is not broken and your network is not degraded: a tenant hit the limit you configured and is being shaped exactly as designed. Without this counter, "my I/O is slow" and "you are being rate-limited" look identical from the guest, and you will spend an afternoon investigating a disk that is fine.
• Net drop and tap read/write failure counters — these distinguish "the guest is slow to drain its queues" from "the host is dropping packets on the floor." Guest-side backpressure and host-side loss have completely different fixes, and this is the only place the difference is legible.
• Seccomp fault counters — a security signal, not a performance one. A nonzero value means something tried a syscall the filter forbids. Usually that's a legitimate workload tripping an edge of the filter and worth a bug report; occasionally it's exactly what it looks like. Either way you want to know, and you want to know which sandbox and which tenant.
• vCPU exit counts — a direct read on how hard the guest is making the hypervisor work. A guest with an anomalous exit rate is doing something structurally expensive (heavy MMIO, a spin loop, a pathological device access pattern) and is a good candidate for a noisy-neighbor investigation.
• Block read/write counts — the boring baseline. Useful mostly as a denominator: throttle events per thousand operations is a meaningful number, throttle events on their own is not.
• API server counters — how many requests the VMM served and how it went. Mostly interesting during boot and snapshot operations, where a rejected call is often the real cause of a create that failed for opaque-looking reasons.
• Balloon statistics — if you use the balloon device, this is your view into guest memory pressure from outside the guest, which is otherwise a surprisingly hard thing to see.

For ad-hoc poking, `jq` over the stream is exactly the right amount of tooling. With one enormous caveat about who else is reading:

# Watch throttling and security counters live. Field names below are
# illustrative -- check `jq keys` against your own build's output first.
tail -f /run/pandastack/sbx-4f2c9a/metrics.fifo \
  | jq -c '{
      t:       .utc_timestamp_ms,
      blk_thr: .block.rate_limiter_throttled_count,
      net_thr: .net.rate_limiter_throttled_count,
      tap_err: .net.tap_read_fails,
      seccomp: .seccomp.num_faults
    }'

# WARNING: on a FIFO this makes you a SECOND reader. Two readers on one pipe
# split the byte stream between them -- you will silently corrupt your real
# collector's input and get half-parsed JSON in both places. On a live VM,
# tee from the collector instead, or do this only on a throwaway sandbox.

# First, always: find out what your version actually emits.
head -c 4096 /var/log/fc-metrics-scratch.json | jq 'keys'

The collector: a drain loop that can't wedge the guest

Here's the piece that matters. This is the shape of the reader I'd write for anything running real workloads: opened before the VMM exists, non-blocking, keeper write end held, and structurally incapable of applying backpressure to the hypervisor. It also does the one thing Firecracker fundamentally cannot do for you — attach identity. The VMM knows it is a virtual machine. It has no idea it is customer `acme-corp`'s test runner. Only your control plane knows that, so tagging happens here or it doesn't happen.

package fcmetrics

import (
	"bufio"
	"encoding/json"
	"log"
	"os"
	"syscall"
	"time"
)

// Sample is one flushed Firecracker metrics record, tagged with the identity
// the VMM cannot possibly know. Firecracker sees a guest; only the control
// plane knows whose guest it is.
type Sample struct {
	SandboxID string          `json:"sandbox_id"`
	Template  string          `json:"template"`
	OrgSlug   string          `json:"org_slug"`
	Host      string          `json:"host"`
	At        time.Time       `json:"at"`
	Raw       json.RawMessage `json:"raw"` // the VMM document, verbatim
}

type Tags struct{ SandboxID, Template, OrgSlug, Host string }

// Tail opens the read end of a metrics FIFO and drains it for the life of
// the VM. Call it BEFORE you exec firecracker: if nobody holds the read end,
// the VMM blocks in write() on a full pipe buffer and your observability
// stack wedges the workload it was installed to observe.
func Tail(path string, tags Tags, out chan<- Sample) error {
	// O_NONBLOCK so open() returns immediately even with no writer yet --
	// that is what lets us get here before Firecracker exists. Go puts the
	// FIFO on the netpoller, so later reads park the goroutine rather than
	// pinning an OS thread.
	rd, err := os.OpenFile(path, os.O_RDONLY|syscall.O_NONBLOCK, 0)
	if err != nil {
		return err
	}
	defer rd.Close()

	// Keeper write end. A reader on a pipe with zero writers gets EOF, so
	// without this we'd exit the moment Firecracker closes or between
	// flushes -- leaving the pipe unattended, which is the whole problem.
	keeper, err := os.OpenFile(path, os.O_WRONLY|syscall.O_NONBLOCK, 0)
	if err != nil {
		return err
	}
	defer keeper.Close()

	sc := bufio.NewScanner(rd)
	sc.Buffer(make([]byte, 0, 64<<10), 1<<20) // records are chunky; don't truncate

	var dropped uint64
	for sc.Scan() {
		line := sc.Bytes()
		if len(line) == 0 {
			continue
		}
		raw := make([]byte, len(line))
		copy(raw, line) // Scanner reuses its buffer; copy before it escapes.

		s := Sample{
			SandboxID: tags.SandboxID,
			Template:  tags.Template,
			OrgSlug:   tags.OrgSlug,
			Host:      tags.Host,
			At:        time.Now().UTC(),
			Raw:       raw,
		}

		// The load-bearing select. NEVER block here. A stalled downstream must
		// cost us samples, not the guest's write(). This trade gets made once,
		// in code, on purpose -- not implicitly during an incident.
		select {
		case out <- s:
		default:
			dropped++
			if dropped%1000 == 1 {
				log.Printf("fcmetrics: downstream stalled, dropped=%d sandbox=%s",
					dropped, tags.SandboxID)
			}
		}
	}
	return sc.Err() // nil on clean shutdown; the caller owns VM teardown
}

Note what the `default:` branch buys you. It converts "my ClickHouse insert is slow" from a guest-visible stall into a counter going up and a log line, which is the correct severity for that event. It also means `dropped` is itself a metric worth alerting on — a collector that's quietly discarding samples is broken, just not in a way that hurts anyone's workload. That's the whole design goal: make the failure land on you, not on the tenant.

The cardinality trap on the way to Prometheus

Now you have tagged samples and a pipeline, and here is where a lot of otherwise-careful teams take out their metrics backend. The tempting move is to emit each counter as a Prometheus time series labelled with the sandbox ID, so you can query any VM. On a fleet churning short-lived microVMs, that's a brand-new time series every few hundred milliseconds, times however many counters Firecracker emits, forever. Prometheus is a wonderful piece of software with a very clear opinion about unbounded label cardinality, and that opinion is that it will fall over.

The split that works: aggregate at the host. Sum device counters and histogram the latencies per agent, per template, per region — labels with small, bounded value sets — and ship those to your time-series database. Then send the per-sandbox detail, with the full sandbox ID and tenant, to a wide-event store built for high cardinality: ClickHouse, or whatever columnar thing you already run. Your dashboards and alerts query the aggregate; your "what happened to sandbox `4f2c9a` at 03:14" investigation queries the events. You get both, and neither system is asked to do the thing it's bad at.

Snapshots reset the counters (and that's your problem, not the VMM's)

One more thing that surprises people on a snapshot-restore platform. The metrics are process-scoped: they belong to the Firecracker process, and a restore is a new process. So a restored microVM starts a fresh counter series from zero, with a new metrics FIFO, configured by whichever supervisor brought it up. There is no continuity across a snapshot boundary, because from the VMM's point of view there is no boundary — there's just a process that started.

This matters more on PandaStack than most places, because snapshot-restore isn't an optimization there, it's the only boot path — every create restores a baked snapshot on demand, p50 179ms end-to-end and ~203ms p99, with the restore step itself around 49ms. Only a template's very first spawn does a genuine cold boot at roughly 3s. Forks are the same story: 400–750ms same-host, 1.2–3.5s cross-host. Every one of those is a fresh process with fresh counters. Stitching a sandbox's history across restores, forks, hibernate/wake cycles, and host migrations is a control-plane job — you join on your sandbox ID, not on anything the VMM gives you, because the VMM genuinely does not know it has a past.

The upside of pushing identity down from the control plane is that this stitching becomes trivial. If every sample already carries a stable sandbox ID and tenant, the lineage is a `GROUP BY` away, and the counter resets are just series boundaries you handle at query time. If you only tagged by Firecracker process, you have thousands of anonymous, unjoinable counter series and no way back.

from pandastack import Sandbox

# On PandaStack the agent owns both FIFO readers, so the collector's lifetime
# is coupled to the VM's by construction -- the wedge described above isn't
# reachable. Per-sandbox detail lands in the event store; the aggregate goes
# to Prometheus on the agent's /metrics endpoint.
with Sandbox.create(template="base", ttl_seconds=300) as sbx:
    sbx.exec("dd if=/dev/zero of=/tmp/x bs=1M count=512 oflag=direct")

    # Guest-side view (what the workload thinks happened):
    print(sbx.exec("cat /proc/diskstats").stdout)

# Host-side view (what the hypervisor knows -- incl. whether the I/O above
# was throttled rather than slow), aggregated, no per-VM label explosion:
#   curl -s http://<agent>:9100/metrics | grep -E 'pandastack_(sandbox_boot|uffd)'

The bottom line

Firecracker's observability model is a small, opinionated trade: no HTTP server, no scrape endpoint, no attack surface — just structured JSON pushed to a descriptor and the assumption that a competent supervisor is holding the other end. That trade is correct for a VMM whose entire value proposition is minimalism, and it hands you a genuinely load-bearing responsibility in exchange. Open the read end first. Keep it open. Never let downstream backpressure reach the pipe. Tag every sample with the identity only your control plane knows. Aggregate before Prometheus and keep per-VM detail in a store built for it. And remember that the counters reset on every restore, because from the hypervisor's perspective each microVM was born five milliseconds ago with no memory of its former life — which, on a snapshot platform, is very nearly true. Verify every field name against the upstream docs for your build before you page on it; the one thing worse than no metrics is metrics you trust that were never right.