CPU Steal Time in microVMs: Your VM Isn't Slow, It's Waiting

A tenant opens a ticket that says "your platform is slow." That sentence contains no information. The whole job of the next ten minutes is turning it into one of three much better sentences: their code is slow, their storage is slow, or they never got a CPU in the first place. There is a single number that separates the third case from the other two, it costs nothing to collect, and most people building on microVMs never look at it. It is steal time.

Steal time is the amount of time a guest's vCPU was runnable — it had work queued, it wanted a core — and the host scheduler did not give it one. Not blocked on disk, not waiting on the network, not idle. Ready and passed over. The CPU wasn't slow; it was in a meeting. I'm Ajay, and I build PandaStack on Firecracker, where dense overcommit is the economic model, so steal is not an anomaly I hunt down and eliminate — it's a dial I have to keep in a range I chose on purpose. This post is what the number actually measures, the paravirtualised plumbing that makes it observable at all, how to read it without fooling yourself, and what to do in what order when it climbs.

What steal time actually measures

Start from the model, because steal only makes sense once you accept it: a guest vCPU is a host thread. Firecracker spawns one ordinary Linux thread per vCPU, each sitting in a KVM_RUN ioctl loop, and the host's CFS/EEVDF scheduler decides when that thread gets a physical core — exactly as it would for nginx or a Python process. Nothing anywhere in the stack promises that a guest with 8 vCPUs receives 8 cores' worth of time. The full mechanics are in our writeup on the Firecracker vCPU scheduling model (/blog/firecracker-vcpu-scheduling-model).

Now the guest's point of view. The guest kernel runs its own scheduler and its own accounting, and it believes it owns its CPUs. When its vCPU thread sits on a host runqueue waiting its turn, the guest experiences that as wall-clock time in which its own tasks did not advance — but from inside, there is no local evidence of why. Its process was runnable. The clock moved. Nothing ran. Without help, that time gets misattributed, usually smeared into whatever bucket the guest was last in, and every graph inside the VM lies to you.

Steal time is the fix: the host tells the guest the truth. It publishes, per vCPU, a running count of nanoseconds during which that vCPU was runnable but not scheduled, and the guest subtracts that from its own accounting. The result is a guest that can say, precisely, "I lost 14% of my wall clock to a queue I cannot see." That's a diagnosis, not a symptom.

How the guest can possibly know: the paravirt steal clock

It's worth being concrete about the mechanism, because it explains both why the number is trustworthy and why it is sometimes silently zero. Steal accounting is paravirtualised — the guest and the hypervisor cooperate through a shared page of memory. On KVM (which is what Firecracker sits on), the guest kernel allocates a small structure and hands the hypervisor its physical address by writing to a KVM-specific MSR. That structure is the steal-time page.

From then on the host updates it out of band. Whenever KVM loads or unloads that vCPU, it takes the delta of how long the underlying task spent waiting on a runqueue and adds it into the shared struct's steal field, in nanoseconds. The guest never traps to read it; it just reads its own memory, which is the entire point — the measurement costs a load instruction, not a VM exit. (On arm64 the same idea arrives through a PV time interface rather than an MSR, but the shape is identical: host writes, guest reads, no exits.) If you want the background on why avoiding an exit for something this frequent matters, see KVM VM exits explained (/blog/kvm-vm-exits-explained).

The guest kernel then folds that counter into its per-CPU time accounting, which is what surfaces in /proc/stat. That handshake has two requirements, and both can fail quietly:

• The host must expose the feature. KVM has to advertise the steal-time capability to the guest. If it doesn't, the guest never registers a page and never sees a nonzero value.
• The guest kernel must be built to use it. Paravirt time accounting is a kernel config option, and minimal microVM kernels — the stripped-down ones people bake precisely because they boot in milliseconds — are exactly the kernels most likely to have it off.

Where the number surfaces

Once the plumbing works, steal shows up in the same three places everywhere:

• The st column in top and vmstat. The one most people have seen and never looked at, sitting at the far right of the CPU line past us, sy, id and wa.
• The 8th value on /proc/stat's cpu line. The raw counter, in jiffies, monotonically increasing since boot. This is what every tool above is reading, and it's the one to instrument because it needs no packages installed in the guest.
• cpu_steal (or node_cpu_seconds_total{mode="steal"}) in whatever agent you run. Same counter, shipped to your metrics backend. If your dashboards break CPU down by mode and steal is not one of the series, add it — it is the only mode that describes the host rather than the guest.

The /proc/stat line orders its fields user, nice, system, idle, iowait, irq, softirq, steal, guest, guest_nice. Steal is the 8th, and because these are cumulative counters you must diff two samples over an interval — a single read tells you about all of time since boot, which is never the question you're asking. Here is the whole measurement with no dependencies:

#!/bin/sh
# steal% over a 5s interval, read straight from /proc/stat inside the guest.
# The aggregate "cpu" line is:
#   cpu  user nice system idle iowait irq softirq steal guest guest_nice
# awk sees "cpu" as $1, so the 8th value (steal) is $9.

snap() {
  awk '/^cpu /{ s=$9; t=0; for (i=2; i<=NF; i++) t+=$i; print s, t }' /proc/stat
}

a=$(snap)
sleep 5
b=$(snap)

echo "$a $b" | awk '{
  d_steal = $3 - $1          # steal jiffies burned in the window
  d_total = $4 - $2          # all jiffies accounted in the window
  pct = (d_total > 0) ? 100 * d_steal / d_total : 0
  printf "steal: %.2f%%  (%d of %d jiffies)\n", pct, d_steal, d_total
}'

# Same number, less arithmetic, if the tools are installed:
#   vmstat 1 5      -> the "st" column
#   top -bn2 | grep '^%Cpu'   -> the "st" field of the second sample
#   mpstat -P ALL 1 -> per-CPU %steal, which is where asymmetry shows up

Reading the CPU line: which number means what

This is the actual triage table. When someone says "slow," you are choosing between four buckets, and each one hands the ticket to a different team:

• user time — Means: the guest's own userspace burned the CPU. It got the cores it asked for and spent them. Fix: this is the tenant's code — profile the application, not the platform. High user with low steal is the healthiest kind of slow, because it is at least honest.
• system time — Means: the guest kernel burned the CPU on the application's behalf — syscalls, page faults, network stack, virtio device work. Fix: look for syscall-heavy or IO-chatty patterns, small writes, or a chatty virtio path. Still the workload's shape, just one layer down.
• iowait — Means: the CPU was idle with at least one task blocked on disk IO. It is a flavour of idle, not of busy, and it is a storage signal rather than a CPU one. Fix: chase the block layer — device queue depth, cache mode, the size and pattern of the reads. Do not respond to iowait by adding CPU.
• steal — Means: the vCPU was runnable and the host scheduler did not run it. Nothing about the guest explains this number; it is a fact about the host. Fix: this is yours, not the tenant's — a neighbour is eating the cores, or you oversubscribed the box, or you set a quota you forgot about.

Every other column tells you about the workload. Steal is the only one that tells you about the landlord.

Why a microVM platform sees steal at all — on purpose

Here is the part people find uncomfortable: on a dense microVM fleet, steal above zero is not a bug report, it is the business model working. Density comes from overcommit — you allocate far more vCPUs across guests than the host has cores, betting that most guests are idle most of the time. That bet is usually right, because an idle guest's vCPU thread is parked in a blocking wait and costs essentially nothing. The economics of that bet are covered in overcommit and microVM density (/blog/overcommit-microvm-density).

On PandaStack, guests are baked with 8 vCPUs, and those 8 are explicitly burst capacity rather than a reservation. A sandbox that wants to run a parallel build gets to use a lot of the host for a short time; the moment it stops, that capacity is somebody else's. Under contention, cgroup cpu.weight shares the cores fairly rather than letting whoever spins hardest win. And because CPU is billed by active CPU-seconds actually burned, an idle guest holding 8 vCPUs costs its tenant nothing — which is only possible because those vCPUs were never dedicated in the first place.

Steal is the price of that arrangement, surfacing exactly when several guests want cores simultaneously. Which reframes the goal. The engineering question is not "how do I drive steal to zero" — you drive steal to zero by dedicating a core per vCPU and throwing away the density that makes the platform cheap. The question is: what steal level is acceptable for this workload? A batch build does not care about a few percent. An interactive terminal session does. A latency-SLA API in front of paying users cares a great deal. Those are three different answers on the same host, and choosing them deliberately is the job.

Four ways the number will mislead you

It's a percentage of guest time, so idle guests exaggerate

Steal is reported as a fraction of the guest's accounted time, not of wall clock or of host capacity. On a nearly idle guest the denominator is dominated by idle jiffies, so a tiny absolute amount of waiting can render as a startling percentage — or, in the other direction, a genuinely contended guest that mostly sleeps can look fine. Always read steal next to the guest's actual utilisation. Ten percent steal on a guest pinning all its vCPUs is a real problem; ten percent steal on a guest doing nothing is a rounding error with good PR.

The first sample after a restore is not a measurement

Every PandaStack sandbox is created by restoring a baked snapshot rather than cold-booting — roughly a 49ms restore step inside a p50 of 179ms and a p99 around 203ms, versus about 3s for a genuine first cold boot. That restore hands the guest a set of counters that were frozen at bake time, and the first delta you compute after resume spans a gap that never happened from the guest's perspective. The same applies across a pause/resume and across a fork (400–750ms same-host, 1.2–3.5s cross-host). Discard the first interval after any restore, pause, or fork, and start your rate calculation from the second sample.

A snapshot-restored clock produces nonsense rates

Worse than a bad first sample: a guest resumed from a snapshot wakes up believing the time is whatever it was when the snapshot was taken, until something re-syncs it. Every rate you compute is a delta divided by elapsed time, and if the guest's notion of elapsed time is wrong — or jumps discontinuously when it does re-sync — your steal percentage is arithmetic on a broken denominator. You get negative rates, absurd spikes, and a very convincing graph of an incident that did not occur. This is a general hazard for anything you measure inside a restored guest, and it's covered properly in our post on Firecracker guest clock and time drift (/blog/firecracker-guest-clock-and-time-drift-explained).

Containers cannot show you this at all

A container has no vCPU. Its processes are host threads scheduled directly by the host, and /proc/stat inside the container is, in the usual setup, the host's /proc/stat — so the steal column describes the host, not your container, and typically reads zero regardless of how starved you are. When a container is CPU-throttled by its cgroup quota it simply stops getting scheduled, and the only evidence lives in nr_throttled and throttled_usec on the host side, where the tenant cannot see it. Their process appears slow, their user time looks unremarkable, and the platform offers no in-guest signal to explain the gap.

That is a genuinely underrated argument for VM-level isolation: the guest kernel boundary is what gives a tenant a first-person, self-service answer to "am I being starved?" — without you granting them host access to prove it. Steal time is one of the few metrics that is better precisely because there is a hypervisor in the way.

Sampling steal across a fleet

One guest's steal number is an anecdote; the shape across many guests on the same host is the diagnosis. Run the identical probe over a set of sandboxes and read the distribution, not the maximum:

import shlex
from pandastack import Sandbox

# One shell snippet, run identically in every guest: two /proc/stat samples
# five seconds apart, printing steal as a percentage of accounted time.
SAMPLE = r"""
snap() { awk '/^cpu /{s=$9;t=0;for(i=2;i<=NF;i++)t+=$i;print s,t}' /proc/stat; }
a=$(snap); sleep 5; b=$(snap)
echo "$a $b" | awk '{d=$3-$1; t=$4-$2; printf "%.2f\n", (t>0 ? 100*d/t : 0)}'
"""


def steal_pct(sbx) -> float:
    # timeout_seconds must exceed the 5s sample window plus exec overhead.
    r = sbx.exec("sh -c " + shlex.quote(SAMPLE), timeout_seconds=30)
    if r.exit_code != 0:
        raise RuntimeError(f"steal probe failed: {r.stderr}")
    return float(r.stdout.strip())


boxes = [Sandbox.create(template="base") for _ in range(8)]
try:
    readings = [(sbx.id, steal_pct(sbx)) for sbx in boxes]
finally:
    for sbx in boxes:
        sbx.kill()

# Sort descending: if ONE sandbox is high and the rest are near zero, the
# problem is that guest's own scheduling, not the host. If they are ALL
# elevated together, the host is the story.
for sid, pct in sorted(readings, key=lambda r: -r[1]):
    print(f"{sid}  steal={pct:6.2f}%")

Interpretation is the whole exercise. If one sandbox is elevated and its neighbours are flat, the host has cores and something about that guest's own scheduling or placement is off. If every sandbox on the host climbs together, the host is oversubscribed for its current runnable working set, and no amount of tuning inside any one guest will help. If steal correlates with the arrival of one specific tenant's workload, you have found your noisy neighbour, and the remedy is weights or placement rather than capacity.

The host-side knob, and the counter next to it

Everything you do about steal happens on the host, in cgroup v2, on the Firecracker process. Two knobs, one counter:

# Host side: the two knobs, and the counter that tells you which one bit.
CG=/sys/fs/cgroup/pandastack/vm-abc123

# Proportional share. Only bites under contention -- an idle host still lets
# this guest burst across its full baked vCPU count. Default weight is 100.
echo 400 > "$CG/cpu.weight"      # 4x the share of a default neighbour

# Hard ceiling: at most 200ms of CPU per 100ms period == ~2 cores, enforced
# even when cores sit idle. Use sparingly: it throws away the burst model.
echo "200000 100000" > "$CG/cpu.max"

# The host-side truth for this cgroup:
cat "$CG/cpu.stat"
#   usage_usec      -> what you actually bill (active CPU-seconds burned)
#   nr_throttled    -> how many periods hit the cpu.max ceiling
#   throttled_usec  -> time this guest was held at the ceiling

# Read throttled_usec NEXT TO the guest's steal number. Contention and quota
# throttling both look like "the guest did not get a core", but the fix is
# opposite: contention wants weights or another host; throttling wants you to
# raise (or delete) the ceiling you set.

The distinction in that last comment is the one people burn time on. Contention and quota throttling both present to the tenant as "my code didn't run," but they have opposite remedies, and only reading the host's throttled_usec alongside the guest's steal tells them apart. The full toolbox — weights, quotas, affinity, NUMA placement, and the density tradeoff each one buys — is in CPU pinning and noisy neighbours in microVM fleets (/blog/microvm-cpu-pinning-noisy-neighbor).

What to do about it, in order

The ordering matters more than any individual step, because the expensive move is last for a reason:

1. Measure per-guest, continuously, before you need it. Ship the /proc/stat steal counter from every guest as a first-class series alongside user, system and iowait. The reason is unglamorous: steal is only useful as a comparison, and you cannot compare against a baseline you never recorded. Verify while you're at it that the counter is not stuck at zero because the guest kernel lacks paravirt time accounting.
2. Alert on sustained steal, never on spikes. A burst of steal means several guests wanted cores at the same moment, which is a busy platform behaving correctly. What you want paged on is elevation held across many minutes on a workload that cannot absorb it. Pick a threshold and a window as a starting point to tune against your own baseline — the right value depends entirely on your workload mix and how much of the burst model you intend to sell, so treat the first number you choose as a hypothesis and correct it after a week of real data.
3. Weight or pin the specific offenders. Once the data names a tenant, raise cpu.weight for the latency-sensitive neighbours or constrain the offender with cpu.max, and pin the genuinely jitter-intolerant workloads to their own cores. Reserve cores for your own control path too — the restore/boot pipeline is host work, and a saturated host inflates create latency for everybody, which turns one greedy tenant into a platform-wide incident.
4. Only then add hosts. Capacity is the correct answer when the runnable working set genuinely exceeds the box, and it is the wrong answer to every one of the cases above — a second host does not fix a misconfigured quota, an unpinned latency workload, or one tenant mining. Add hardware when the measurement says the platform is full, not when it says you have a bug.

The metric that tells you whose problem it is

Every operational metric on a multi-tenant platform is really answering a routing question: whose problem is this? User time routes to the tenant. Iowait routes to storage. Steal routes to you, and it is the only one of the four that does, which is exactly why it is the number worth wiring up before you think you need it. Collect it on every guest, read it next to utilisation, distrust the first sample after any restore, and remember that on a platform whose whole value proposition is dense burst capacity, a little steal is the sound of the machine doing its job. What you are watching for is not its presence. It's its persistence.