Observability for Machines That Live 200 Milliseconds

The ticket says: my sandbox died. It arrives forty minutes after the fact. The machine it refers to existed for about nine hundred milliseconds, on a host that has since created and destroyed several thousand others, and the rootfs it wrote its logs to was a copy-on-write clone unlinked before the customer finished typing. Somewhere in your infrastructure there is an answer to this ticket. It is definitively not on that machine, because that machine has not existed for most of an hour.

I'm Ajay; I build PandaStack, a Firecracker microVM platform where every create is a snapshot restore — around 179ms p50, 203ms p99 — and a large share of the VMs we run are gone in under two seconds. The thing nobody warns you about when you move to ephemeral compute is that observability breaks structurally rather than incrementally. Every tool most of us learned was designed on the assumption that the host outlives the question, and that assumption is now false. This post is about which streams actually exist, how to get bytes out of a guest that is about to be killed, and how to do it without turning your metrics bill into a second infrastructure problem.

Every assumption in your stack is about long-lived hosts

Start with the failures, because they are specific and they compound.

• Pull-based scraping assumes the target is alive when you ask. A scraper polls on an interval you configure — tens of seconds, typically; check your own scrape config. A VM that lived four hundred milliseconds is never scraped once. Not sampled poorly: never observed, and a series that never existed leaves no gap in the graph to notice.
• Log files assume a durable filesystem. The guest wrote to /var/log/app.log on a reflinked rootfs destroyed at teardown. A hard kill is the world's fastest log rotation, and there is no rotated copy.
• Tailing is a race against a kill. A tail process inside the guest works perfectly until the moment it matters, then dies with everything else — and the last thing it read is not the last thing that happened.
• Identity is not stable. Ten thousand VM identities an hour, each with its own label set, is a cardinality bomb rather than a monitoring target — time-series databases are priced on distinct series, and a fresh UUID per workload multiplies that by your throughput.
• Sidecar agents assume time to start and time to flush. A collector that takes a second to initialise and flushes on a five-second timer has, inside a two-second VM, spent its whole life starting up before being killed holding the interesting bytes.

None of this means the tools are bad. Prometheus, Loki, Vector and the OpenTelemetry collector all have patterns for short-lived work — push gateways, agent modes, host-level collection — and you should verify the specifics against each project's own docs, which move faster than blog posts. It means the default deployment shape, one agent per host scraping targets that stay put, does not describe your fleet, so you have to design the collection path rather than inherit it.

Four streams, and the outage that comes from conflating them

The most useful thing you can do early is stop saying "the logs" and start naming four distinct streams. They have different producers, transports and failure modes, and — critically — different availability. Two of them exist when the guest is completely dead.

1. The VMM/host view. Firecracker's own process log: API requests and their errors, device configuration, boot timings, the reason a VM exited. It lives on the host and survives the guest by definition. It is also the only stream that exists when the guest never boots — a machine that never ran cannot explain itself.
2. The guest serial console. Where the kernel writes: panics, OOM kills, early-boot device failures, init dying. The VMM captures it from outside, so it works when the network is down, the guest agent is broken, userspace never started. Slow, unstructured, interleaved — and the only channel you have while everything else is still coming up.
3. The application's own stdout and stderr inside the guest. What users mean by "logs" and the stream everyone builds first. It also needs the most machinery: the app has to run, something has to capture it, something has to move it off before the machine ends.
4. Structured lifecycle events from the control plane. created, restored, resumed, paused, killed, ttl-expired, oom — emitted by your own code, on hosts that stay up, with schemas you control. The spine everything else hangs off, and the cheapest stream to make reliable, because none of it needs the guest to cooperate.

Now the outage. When a customer says "my sandbox died," the answer is almost always in stream 1 or stream 2, and almost nobody looks there first. The instinct is to read the application log — which, in the failure cases that generate tickets, is empty. Empty because the app never started, because init failed, because the kernel panicked, because the platform killed the VM for exceeding a memory ceiling. All of that is in the host log and the console. The application log's silence is the evidence, not the absence of it.

Firecracker gives you both host-side streams through its API socket, and the console through the same serial device the guest kernel is told to use on the boot line. Configure them before the machine starts:

# All of this happens BEFORE InstanceStart -- the logger and metrics endpoints
# take a FIFO or a file path that must already exist. Create them first.
SOCK=/run/firecracker/vm-abc123.sock
VMDIR=/var/lib/pandastack/vms/abc123
mkdir -p "$VMDIR"
touch "$VMDIR/firecracker.log" "$VMDIR/metrics.ndjson"

# 1. The VMM's own log. Levels: Error | Warning | Info | Debug.
#    show_log_origin puts the source file/line in each line -- worth it when
#    you are correlating a VMM error against a specific API call.
curl -s --unix-socket "$SOCK" -X PUT http://localhost/logger \
  -H 'Content-Type: application/json' -d '{
    "log_path": "'"$VMDIR"'/firecracker.log",
    "level": "Info",
    "show_level": true,
    "show_log_origin": true
  }'

# 2. VMM metrics, emitted as JSON lines. This is NOT a Prometheus endpoint --
#    nothing scrapes it; Firecracker appends to the path you give it. Point it
#    at a FIFO and have a host-side reader forward it, or read the file after
#    the VM exits. Counters cover the API, block/net devices, vsock, seccomp.
curl -s --unix-socket "$SOCK" -X PUT http://localhost/metrics \
  -H 'Content-Type: application/json' -d '{
    "metrics_path": "'"$VMDIR"'/metrics.ndjson"
  }'

# 3. The serial console. The guest kernel writes panics here; the VMM captures
#    it from outside the guest, so it works when the guest is beyond help.
curl -s --unix-socket "$SOCK" -X PUT http://localhost/boot-source \
  -H 'Content-Type: application/json' -d '{
    "kernel_image_path": "/var/lib/pandastack/kernel/vmlinux-5.10",
    "boot_args": "console=ttyS0 reboot=k panic=1 pci=off"
  }'

# Redirect the process stdio when you exec firecracker, and ttyS0 lands in a
# host file that outlives the guest by exactly as long as you want it to:
#   firecracker --api-sock "$SOCK" >"$VMDIR/console.log" 2>&1
#
# Field names and levels are version-specific -- check them against the
# Firecracker API spec for the version you actually run.

Two things in there are load-bearing. Firecracker's metrics endpoint is not a scrape target — nothing pulls it; the VMM appends JSON lines to a path you own. That inversion is exactly right for ephemeral machines, and worth internalising as a pattern rather than an inconvenience. And the console is captured by redirecting the VMM process's own stdio, so it belongs to the host process supervising the VM: when the guest dies badly, the file is already complete on a filesystem that is not going anywhere.

Getting bytes out of a guest

For the application stream you have four real options, and the differences between them matter far more than the differences between log formats.

Write to a file, have the host tail it. Familiar, and the only question that matters is where the file lives. If the app writes into a rootfs deleted at teardown, you have built a log guaranteed to disappear at the exact moment it becomes interesting. Put it on a path the host can read and it survives the VM. Tailing from inside the guest is fine for live viewing and terrible as durability, because the tail process is inside the thing about to stop existing.

A vsock channel to a host-side collector. This is the interesting one. AF_VSOCK connects a guest to its hypervisor directly — no network stack, no IP address, no routing, no NIC. The guest connects to a well-known port on the host CID; Firecracker exposes the other end as a Unix domain socket. Low latency, no guest network configuration, and — the part that actually decides it — it works when the sandbox has no egress at all. If the whole point of your platform is that untrusted code cannot reach the internet, a telemetry transport that needs an outbound TCP connection is a hole you cut in your own boundary.

Push directly to a collector over the network. The guest holds an endpoint and a token and ships to your OTLP or Loki ingest itself — the simplest thing in the world when the guest already has network access, a structural mistake when it does not. You have coupled untrusted workload code to your telemetry endpoint, handed it a credential, and given it a legitimate-looking egress path. Fine for trusted first-party workloads; for tenant code it is the transport you spend the next year regretting.

The host reads the serial console. Barely a shipping mechanism — unstructured, interleaved, slow enough that high-volume writes change your program's timing. But last resort is a real category, and some workloads write one short summary line to the console at exit precisely because that path has no dependencies left to fail.

• Host tails a guest file — Latency: seconds, whatever the poll interval is. Works without guest network: yes. Survives a hard kill: only if the file is on a host-visible path, not inside a doomed rootfs. Best for: bulk application logs where you control the mount layout.
• vsock to a host collector — Latency: sub-millisecond, no network stack in the path. Works without guest network: yes, that is the point. Survives a hard kill: whatever the host already read, yes; whatever sat in the guest buffer, no. Best for: no-egress sandboxes and structured events you want off promptly.
• Guest pushes over the network — Latency: a round trip plus batching. Works without guest network: no, by definition. Survives a hard kill: only what was flushed and acknowledged. Best for: trusted first-party workloads — a security problem for tenant code, which gains a credential and an egress path.
• Host reads the serial console — Latency: immediate, though the write perturbs the guest. Works without guest network: yes, and without a guest agent, and without userspace. Survives a hard kill: yes, the capture is host-side and already on disk. Best for: kernel panics, early-boot failures, a final summary line at exit.

The guest side of a vsock forwarder is small enough that you should write it rather than adopt something. A sketch of the host-side listener, where the interesting logic lives:

// Host-side vsock log collector. One goroutine per guest connection; the
// guest's context ID is the identity, so a compromised guest cannot claim to
// be a different sandbox by putting someone else's ID in a log line.
package main

import (
	"bufio"
	"log"
	"time"

	"github.com/mdlayher/vsock" // AF_VSOCK bindings
)

const (
	logPort = 10514
	maxLine = 64 << 10 // 64 KiB; a longer "line" is a bug or an attack
	quotaB  = 32 << 20 // 32 MiB per VM lifetime, then we drop and say so
)

func main() {
	l, err := vsock.ListenContextID(vsock.Host, logPort, nil)
	if err != nil {
		log.Fatalf("vsock listen: %v", err)
	}
	for {
		c, err := l.Accept()
		if err != nil {
			log.Printf("accept: %v", err)
			continue
		}
		go handle(c)
	}
}

func handle(c *vsock.Conn) {
	defer c.Close()

	// Identity comes from the hypervisor, never from the payload.
	cid := c.RemoteAddr().(*vsock.Addr).ContextID
	sandboxID, ok := lookupSandboxByCID(cid)
	if !ok {
		return // a CID we do not recognise gets no ingest path
	}

	sc := bufio.NewScanner(c)
	sc.Buffer(make([]byte, 0, 4096), maxLine)

	var wrote int
	for sc.Scan() {
		line := sc.Bytes()
		if wrote += len(line); wrote > quotaB {
			emitEvent(sandboxID, "log_quota_exceeded", wrote)
			return // drop the connection, record that we dropped it
		}
		// The tenant controls these bytes. Treat them as a value in a field,
		// not as a line in a file that a downstream parser re-splits on \n.
		sink.Write(Record{
			SandboxID: sandboxID,    // ours
			At:        time.Now(),   // ours -- guest clocks lie, especially
			Stream:    "app",        //        after a snapshot restore
			Message:   string(line), // theirs, and untrusted
		})
	}
}

Two design points worth stealing: identity comes from the hypervisor rather than the payload, and the timestamp comes from the host. Guest clocks are wrong right after a snapshot restore, when the guest resumes believing it is whatever time it was when the snapshot was taken. Sort a customer's logs by a guest-supplied timestamp and you will eventually show them a session that happened in the past.

The flush-before-you-die problem

Here is the failure that will cost you the most debugging time, and it is not exotic. Buffered writers lose the last few seconds, and the last few seconds are always the interesting ones. A process that crashes at t=1.8s in a VM torn down at t=2.0s, holding an 8 KiB stdio buffer nobody flushed, produces a log that ends cleanly in the middle of normal operation. The VM had a rough time and left no note.

Many runtimes buffer stdout when it is not a terminal, which is exactly the case inside a sandbox. Then your log library batches on top of that, and your shipper batches on top of that: three layers politely holding the evidence. Four things help, in rough order of how much:

1. Make the exit path synchronous. Handle SIGTERM, flush, and wait for the flush to be acknowledged before exiting — then give teardown a grace period long enough for it to finish. Highest-value change here, usually about ten lines.
2. Keep a last-words ring buffer on a host-visible path: a fixed-size region holding the final N kilobytes, readable after the guest is gone. It costs nothing while things are fine and it is the entire postmortem when they are not. Store it with the lifecycle event as a first-class artifact, not as log lines that happened to be near the end.
3. Do not just run everything unbuffered. That solves the loss and adds a syscall per write, which on a chatty program is measurable and over a serial console is worse. Line buffering plus an explicit flush at exit is the honest middle.
4. Emit the "about to die" event from the control plane, not the guest. Why a VM was killed — TTL expiry, an operator delete, a memory ceiling, a failed health check — is known by your platform, not by the process being killed. That event turns "the log just stops" into an answer.

Cardinality, and the difference between a metric and an event

Do not put sandbox_id in a Prometheus label. A time-series database stores one series per distinct combination of label values, so a per-VM identity on a high-churn fleet means a new series for every workload you ever run, retained for the whole retention window. The graph you wanted — "is boot latency regressing" — never needed per-VM series. It needed a histogram.

The clean split is between metrics and events, and ephemeral fleets need both because they answer different questions.

• Metrics are aggregates computed on the host with bounded label sets: per-template, per-region, per-tier histograms of boot and restore duration; counters of creates, failures, OOM kills. Labels should be things you can enumerate on a whiteboard. This is alerting and dashboards.
• Events are one row per thing that happened, with high-cardinality fields as ordinary columns rather than index dimensions: sandbox id, tenant, template, exit reason, per-stage durations. A columnar store — ClickHouse and its relatives are the obvious fit — takes millions of these cheerfully, because a UUID column is a column and not a new series. This is forensics and billing.
• Sample the expensive detail. Full traces on a million daily VMs is a data warehouse you did not mean to build. Keep events for everything, detailed spans for a small percentage plus every failure — tail sampling, where the keep decision happens after you know whether it went wrong, is the shape this workload wants.

Rule of thumb: if the answer is a number over time, it is a metric and the labels must be small; if the answer is "which one, and what happened to it," it is an event and belongs in a store built for wide rows. Making one system do both is how you end up with a metrics bill larger than your compute bill.

Tracing across the hypervisor boundary

Trace context does cross into a guest, and the mechanism is unglamorous: propagate it as you would into any subprocess. The API request that creates a sandbox carries a traceparent; you pass it in as an environment variable or a file written before the workload starts, and code inside picks it up as the parent of its own spans. Nothing about virtualization makes this harder than fork/exec. What is harder is that the guest may have no network path to your collector — which puts you back on vsock, spans being structured events with a parent id.

The honest limitation is proportion. In a VM that lives two hundred milliseconds, the guest's own spans are a sliver and the platform's spans are nearly the whole trace: allocate a network slot, configure the tap device, reflink the rootfs, fork the VMM, load the snapshot, resume, probe the port. That sounds like a disappointment and it is the opposite — those are exactly the spans you want when boot latency regresses, because the regression is almost always in one of them. A trace covering only the user's code would show a slow request with an unexplained gap at the front.

On a two-hundred-millisecond machine, the platform is not the overhead around the trace. The platform is the trace.

Your log source is also your attacker

Everything above assumes logs are data you collect. On a multi-tenant platform they are also input a stranger controls — the volume, the content, and the exact bytes your pipeline treats as structure.

• Cap volume per VM and per tenant, and record the drop as an event rather than discarding silently. A tenant logging a gigabyte in ninety seconds is either a bug or a denial-of-service against your ingest, and both belong in billing and alerting rather than your on-call rotation.
• Never render guest output as HTML. Log viewers grow features — clickable links, expandable JSON, ANSI colour — and each is a parser applied to attacker-controlled bytes inside your dashboard's origin, where your session cookie lives.
• Assume injected newlines. A guest writing an embedded newline plus a plausible prefix can forge log lines that appear to come from your platform. Escape control characters at ingest and carry the message as a field value, not as text you concatenate into a line something downstream re-splits.
• Do not trust guest-supplied identity, severity or timestamps for anything that matters — those come from the collector. A tenant may label their own lines however they like inside their own field; the labels must never change how your system routes, prices or alerts. Watch the reverse direction too: error text returned to a tenant leaks host paths, template names and node identifiers.

What this looks like in practice

Concretely, on PandaStack: each VM has a host-side firecracker.log written by the VMM process, which is what we read when someone asks why a machine never came up. The application's own output goes to a guest file tailed over the exec channel and streamed to clients as server-sent events, so live logs need no route into the guest. Lifecycle events land in ClickHouse as wide rows, where sandbox id is a column and nobody's metrics bill notices. Three streams, three transports, three retention policies, deliberately not unified.

The client-side shape of that, for a job that has to produce its logs before its machine stops existing:

from pandastack import Sandbox

# A short-lived job. The TTL is a backstop enforced by the platform: if this
# process panics, the machine still dies. Cleanup you have to remember to run
# is cleanup that does not happen during an incident.
sbx = Sandbox.create(
    template="base",
    ttl_seconds=120,
    metadata={"job": "etl-batch", "trace_id": current_trace_id()},
)

try:
    # Propagate trace context into the guest the same way you would into any
    # subprocess -- an env file the workload reads before it starts.
    sbx.filesystem.write(
        "/work/trace.env",
        f"TRACEPARENT=00-{current_trace_id()}-{current_span_id()}-01\n",
    )
    sbx.filesystem.write("/work/input.json", payload)

    # Two things make the log survivable, and neither is exotic:
    #   - stdbuf -oL: line buffering, so a crash does not eat the last 8 KiB
    #   - the log file is a real path we read back explicitly, not something
    #     we hope a background shipper flushed in time
    r = sbx.exec(
        ". /work/trace.env && "
        "stdbuf -oL -eL python /work/job.py 2>&1 | tee /work/run.log",
        timeout_seconds=90,
    )

    # Read the artifact BEFORE the machine goes away. The last N kilobytes are
    # the postmortem; treat them as a first-class output of the job, stored
    # next to the lifecycle event rather than streamed and forgotten.
    tail = sbx.filesystem.read("/work/run.log")[-64_000:]
    emit_event(
        kind="job.finished",
        sandbox_id=sbx.id,        # a column in the event store,
        exit_code=r.exit_code,    # NOT a Prometheus label
        last_words=sanitize(tail),
    )
finally:
    # Ordering matters. Everything you want must be off the machine before
    # this line runs, because after it there is no machine to ask.
    sbx.kill()

The ordering in that finally block is the whole lesson in four lines: read what you need, then destroy. Nearly every observability bug I have debugged on ephemeral infrastructure reduces to someone doing those two in the other order, or assuming a background process would get the bytes out in time. It did not, because there was no time. Assume the machine is gone, assume the last buffer is lost, and make sure the streams that answer your first question — did it boot, and who killed it — never needed the guest alive at all.