Your benchmark ran at a different clock speed than production

Someone sends you a benchmark. Same binary, same instance type, same region, run twice a week apart, and the second run is meaningfully slower. Nothing changed in the code. The obvious suspects get investigated in the usual order: a dependency bump, a kernel upgrade, some regression in the platform underneath. Days go into it. The actual answer is that the two runs happened at different clock speeds, and nothing in the measurement recorded that, because almost nobody records it.

I'm Ajay; I build PandaStack, a Firecracker microVM platform where creating a sandbox is a snapshot restore rather than a boot. A large share of the performance complaints I get are, underneath, this: a vCPU is not a fixed unit of speed, people assume it is, and every mental model built on that assumption produces wrong answers about benchmarks, about capacity, and about billing.

A vCPU is an entitlement, not a rate

In Firecracker a vCPU is a host thread running a KVM ioctl loop. That thread gets scheduled onto a physical core by the host kernel, and while it is on that core, the core runs at whatever frequency the hardware and the host's power management have negotiated for the current instant. Nothing about "you were allocated 2 vCPUs" says anything about how fast those cores tick.

So the unit is time-on-core, not work-per-second. A guest asking for four cores is asking to occupy four cores concurrently. Whether that yields the throughput it yielded yesterday depends on a stack of things the guest cannot see: which driver owns frequency on the host, which governor that driver is running, how many sibling cores are currently busy, how much thermal and power headroom the socket has left, and — on some parts — whether the instruction mix has tripped a frequency offset.

None of those are exotic. They are the default behaviour of every modern server CPU. The exotic thing would be a machine that ran at one constant frequency, and the industry spent twenty years engineering away from exactly that.

P-states and the thing that picks them

The hardware exposes a ladder of performance states — P0 at the top, descending through lower voltage-frequency pairs. Something has to choose a rung, continuously, per core, and that something is the cpufreq subsystem on Linux. Two layers matter: the scaling driver and the governor.

The driver is the part that knows the hardware. Modern Intel parts usually run intel_pstate, modern AMD parts amd-pstate, and older or virtualised setups fall back to acpi-cpufreq. This distinction trips people constantly, because intel_pstate in its default mode does not really use the kernel's generic governors at all — it implements its own algorithm and exposes just two names, "powersave" and "performance". Reading "powersave" on an intel_pstate host and concluding the machine has been left in some low-power mode is a very common misdiagnosis. Under intel_pstate, "powersave" is the normal load-following mode, not a floor.

Where the generic governors do apply — acpi-cpufreq, amd-pstate in passive mode, intel_pstate in passive mode — the interesting one is schedutil. Older governors like ondemand sampled CPU utilisation on a timer and reacted to it. schedutil instead reads the scheduler's own utilisation signal, the same per-entity load tracking the scheduler uses for placement decisions, and requests a frequency directly from it. That is a better design, and it is still fundamentally reactive: a task has to run and accumulate utilisation before the signal rises enough to justify a higher frequency. The governor cannot know your job is short and urgent.

Then there is HWP, Intel's hardware-managed P-states, or Speed Shift. When HWP is active the CPU package itself picks the frequency, with the OS supplying hints about desired range and energy-performance preference. Hardware control ramps far faster than a software governor can, because it is not waiting on a scheduler tick or a sampling window. If you are chasing frequency behaviour on short workloads, whether HWP is in play is one of the first things to establish.

# Who owns frequency on this host? intel_pstate and amd-pstate implement
# their own scaling algorithm; acpi-cpufreq defers to the kernel governors.
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver

# The governor. On intel_pstate, "powersave" is the normal load-following
# mode -- it is NOT a low-power lock, despite the name. "performance" asks
# for the top P-state and holds it.
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors

# Per-core right now, in kHz. scaling_cur_freq is what the kernel believes it
# requested; cpuinfo_cur_freq reads the hardware and usually needs root.
grep . /sys/devices/system/cpu/cpu*/cpufreq/scaling_cur_freq

# The window the kernel is allowed to ask within.
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq

# Is opportunistic boost even enabled? Two different files depending on driver.
cat /sys/devices/system/cpu/intel_pstate/no_turbo 2>/dev/null   # 1 = turbo OFF
cat /sys/devices/system/cpu/cpufreq/boost        2>/dev/null   # 1 = boost ON

# Hardware P-states (Speed Shift). If "hwp" is in the flags, the package is
# choosing frequency itself and ramps much faster than any governor.
grep -m1 -o ' hwp ' /proc/cpuinfo || echo "no HWP"

# Sanity: on a virtualised guest, most or all of the above will simply not
# exist. That absence is itself the finding. See the cloud section below.

Turbo is opportunistic, and it is shared

The number on the spec sheet — the big one, the max turbo — is a single-core figure. It is what one core can reach when the rest of the socket is idle enough to donate its power and thermal budget. That is not the number you get on a machine running fifty sandboxes.

Server CPUs publish turbo as a set of bins keyed on how many cores are active. One or two active cores get the headline frequency. All cores active get a lower one, sometimes substantially lower. The mechanism is a budget: the socket has a power limit and a thermal limit, and boosting is spending shared headroom. When your neighbours spend it, your ceiling drops. Nobody did anything wrong; the hardware is behaving exactly as specified.

Your maximum clock speed is a function of how busy your neighbours are. On a multi-tenant host, that means your performance ceiling is set by strangers.

Power limits add a time dimension on top. Intel parts express this as PL1 and PL2 with a time constant: a short-term power limit you may exceed for a while, then a sustained limit you settle back to once the running average catches up. AMD's Precision Boost does something analogous with its own budgets. The practical shape is that a burst of work can run genuinely faster than the same work run continuously, which means a thirty-second benchmark and a thirty-minute production load are measuring two different machines.

There is an instruction-mix dimension too. On several Intel server generations, sustained wide-vector work — AVX2 and especially AVX-512 — drops the core to a lower licence frequency, because those execution units draw more power per cycle. Later parts reduced the penalty considerably, and it was always workload-dependent rather than a flat tax, but if your benchmark links a numerics library that dispatches to AVX-512 and your production path does not, you are again comparing two machines. This is also the reason CPU feature masking has a performance dimension and not only a portability one.

Arm server parts are a useful contrast here. Graviton and Ampere-class chips are generally marketed on a single sustained all-core frequency rather than a boost figure, which removes most of this variance by construction. You give up the burst; you get predictability. For a platform selling latency percentiles, that trade is more attractive than it first looks.

Ramp latency, and why short work never sees the good clock

Here is the part that matters most for sandboxes and gets discussed least. Changing frequency is not instantaneous and it is not free. A software governor has to observe load, decide, and request a transition; the hardware then has to move voltage and settle. Historically the observe-and-decide step alone lived on the order of tens of milliseconds, which is why HWP exists and why schedutil replaced timer sampling.

Now consider the workload profile that a sandbox platform actually serves. A create finishes in about 179ms at p50 on our fast path. An agent tool call runs for a few hundred milliseconds. A serverless function invocation does its work and exits. A CI step might run for seconds, but the interactive stuff — the stuff whose latency users feel — is bursty and short.

A core that was idle when your work arrived starts low. Your job runs. The governor notices utilisation climbing and asks for more. And then your job finishes, possibly before the request has been fully honoured. You ran at the low clock for a meaningful fraction of your total runtime, or all of it. This is the exact opposite of the benchmark case, where you hammer the same core in a loop for thirty seconds and it has long since settled at the top of the ladder.

What this does to a sandbox platform specifically

Short-lived sandboxes are the worst case

Everything about the ephemeral-sandbox model puts you on the wrong side of frequency ramping. The sandboxes are short. They land on cores that may have been idle. They do a burst of work and go away. There is no long-running process to build up the utilisation history that would justify a high P-state, and by the time the system agrees you deserve one, you are gone.

There is a mitigating detail worth stating honestly, because it cuts against the drama: our create path is mostly not CPU-bound. Allocating a pre-built network namespace, reflinking a rootfs, forking Firecracker, loading a snapshot and probing a port is dominated by syscalls, page faults and I/O rather than by arithmetic throughput. Frequency moves that number, but it moves user code inside the sandbox far more. If your workload is "restore a VM and run a Python script", the script is where clock speed shows up.

Dense hosts run at the all-core bin

A platform's whole economic argument is density. Ours certainly is: memory admission and cgroup weights exist so that many guests share a host without any of them being starved. But density is precisely the condition under which the socket sits in its all-core turbo bin rather than its single-core one.

So the first sandbox on a freshly provisioned host is, genuinely, running on faster silicon than the fiftieth sandbox on a full one — before any scheduler contention, any steal time, any cache pressure enters the picture. Users experience this as "the platform got slower under load" and reach for the software explanation, because the software explanation is the one that is visible. Some of that gap is not software at all.

Heterogeneous fleets break naive capacity math

Now scale it up. A fleet is rarely one SKU. You add capacity over years; instance families get refreshed; you run in several regions with different hardware availability. Every one of those hosts advertises cores, and your scheduler counts them.

Ours does exactly that. The placement score is a load-spreading function over free CPU and free memory, with a small bonus for hosts that can stream memory on restore. Free CPU there is a count of unclaimed vCPU entitlements. It says nothing whatsoever about how fast those entitlements execute. Two hosts reporting the same free capacity can deliver visibly different throughput for identical work, and the scheduler has no term for that today. I would rather write that down than pretend otherwise.

The consequence for capacity planning is that "we have N vCPUs of headroom" is a weaker statement than it sounds. It is a statement about concurrency, not about throughput. If you plan capacity from it, you are planning how many things can run at once, not how much work will get done.

Why CPU-second billing is fairer than it looks — and where it still isn't

This is the one place where frequency variance argues in the customer's favour, and it took me a while to see it. We bill CPU by active CPU-seconds actually burned, not by the vCPU count a template was baked with. Each live Firecracker process sits in its own cgroup and a reconcile loop scrapes usage_usec out of cpu.stat; that delta is the billable quantity. Memory bills on committed GiB-hours, because memory is genuinely reserved; CPU bills on use, because CPU is genuinely burst.

Under that model, a customer whose work landed on a busy host does not pay for the reservation of eight cores they never got. They pay for the microseconds they actually consumed. Compared with a fixed vCPU-hour price, which charges the same whether the silicon delivered its headline clock or its all-core one, metering consumption is the more honest instrument.

The honest caveat: usage_usec is microseconds on a core. It is time, not work. If the core was running slower, the same computation consumes more microseconds and therefore costs more, not less. Time-based metering does not fully insulate anyone from frequency variance — it just stops charging for capacity that was never delivered. Billing by retired instructions would close the remaining gap and would be a considerably worse product, because nobody can forecast a bill denominated in instructions.

# What the meter actually reads on one of our hosts. Each live Firecracker
# process gets a child cgroup under the agent's delegated service cgroup,
# with cpu.weight proportional to the template's baked vCPU count.
svc=/sys/fs/cgroup$(awk -F: '/^0::/ {print $3}' /proc/self/cgroup)

# weight = 100 x vCPUs, so an 8-vCPU template lands on 800. Weights are
# PROPORTIONAL shares: they bind only under contention. On a quiet host a
# sandbox bursts across every physical core available to it.
cat "$svc"/vm-<sandbox-id>/cpu.weight

# usage_usec is the billable counter: MICROSECONDS ON A CORE. Sample it twice
# and diff. Note what it does not contain -- cycles, instructions, or any
# notion of how fast the core was ticking while those microseconds elapsed.
grep usage_usec "$svc"/vm-<sandbox-id>/cpu.stat
sleep 10
grep usage_usec "$svc"/vm-<sandbox-id>/cpu.stat

In a cloud VM you usually cannot see any of this

Everything above assumes you can read cpufreq. On bare metal you can. Inside a virtualised instance — which is where most people's hosts and all of their guests live — the cpufreq sysfs tree is typically absent entirely, because frequency is the hypervisor's business and the guest has no say in it. The MSRs that turbostat wants are usually not accessible either.

Two traps follow from that. First, the "cpu MHz" line in /proc/cpuinfo inside a guest is frequently just the nominal frequency, restated forever, changing never. It looks like an answer and is not one. Second, the TSC is invariant on modern parts: it ticks at a fixed rate regardless of the core's actual clock. That is what makes it a reliable clocksource, and it also means you cannot infer frequency from it. The ratio that does encode real frequency is APERF over MPERF, multiplied by nominal — and those are exactly the MSRs your guest is not allowed to read.

What you can always do is measure the effect rather than read the cause. Run a known quantity of work and time it. The absolute number means nothing; the ratio between an idle host and a loaded one, or between host A and host B, means quite a lot.

# Where perf is available and permitted, it does the arithmetic for you.
perf stat -e task-clock,cycles,instructions -- ./your_workload
#   cycles / task-clock-in-seconds  = effective GHz for that run
#   instructions / cycles           = IPC, which moves for OTHER reasons
#                                     (cache misses, contention) -- do not
#                                     conflate a frequency drop with an IPC drop

# Where it is not, calibrate against a fixed amount of arithmetic. Run this on
# a quiet host, run it again on a busy one, and compare the two numbers. The
# unit is arbitrary; the RATIO is the measurement.
python3 - <<'PY'
import time
def rate(n=30_000_000):
    t = time.perf_counter()
    x = 0
    for _ in range(n):
        x += 1
    return n / (time.perf_counter() - t) / 1e6
warm = [rate() for _ in range(3)]      # discard: this is the ramp
runs = sorted(rate() for _ in range(9))
print("Miter/s  min", round(runs[0], 1),
      " median", round(runs[len(runs)//2], 1),
      " max", round(runs[-1], 1))
PY

How to benchmark so the number survives contact with production

None of this makes benchmarking pointless. It makes single-number benchmarking pointless. The fix is a short list of habits, and the reason to adopt them is that they turn an unreproducible anecdote into a measurement someone else can argue with.

1. Warm up and throw the warmup away. The first N iterations are measuring caches filling, connection pools opening, JITs compiling and the core climbing its P-state ladder. Discard them explicitly rather than hoping they average out — they do not average out, they skew the mean upward and leave a fake tail.
2. Pin the governor if you own the host, and say so. Setting every core to the performance governor removes one large variable. If you cannot pin it — and inside a cloud VM you cannot — record that you could not, because the alternative is silently reporting the governor's mood as your system's performance.
3. Report percentiles, never a mean. p50, p90, p99 and max. A mean hides exactly the bimodality that frequency effects create, where most runs get the good clock and a minority do not.
4. State the host state alongside the number. How many other guests were on the box, what they were doing, how long the host had been up, whether turbo was enabled. A latency figure without its conditions is not a result, it is a screenshot.
5. Run the loaded case on purpose. Benchmark with realistic neighbour load, not on an empty machine, because the empty machine is the one configuration your users will never be in.
6. Measure both shapes. A steady-state loop and a cold burst are different questions. Publish both, because your users' traffic is the burst and your marketing instinct is the loop.

import os
import statistics
import time

from pandastack import Sandbox   # pip install pandastack; PANDASTACK_API_KEY set

WARMUP, N = 20, 200

def one_run() -> float:
    """End-to-end: create a sandbox, run something trivial, tear it down."""
    t0 = time.perf_counter()
    sbx = Sandbox.create(template="code-interpreter", ttl_seconds=60)
    try:
        sbx.exec("python3 -c 'print(1)'")
        return (time.perf_counter() - t0) * 1000.0
    finally:
        sbx.kill()

# 1. Warm up and DISCARD. TCP pools, chunk caches on the host, and the core's
#    P-state all need to reach steady state before a sample means anything.
for _ in range(WARMUP):
    one_run()

samples = sorted(one_run() for _ in range(N))

def pct(p: float) -> float:
    return samples[min(len(samples) - 1, int(len(samples) * p))]

# 2. Percentiles, not a mean. The mean is where bimodal distributions go to
#    hide, and frequency effects are bimodal almost by definition.
print("n", N,
      "p50", round(pct(0.50)), "p90", round(pct(0.90)),
      "p99", round(pct(0.99)), "max", round(samples[-1]))
print("mean", round(statistics.mean(samples)),
      "stdev", round(statistics.stdev(samples)))

# 3. Publish the CONDITIONS with the number, or it is not a result. Fill these
#    in honestly -- "unknown" is a legitimate and useful answer.
print("host", os.uname().nodename,
      "| neighbours: unknown",
      "| governor: unreadable (virtualised guest)",
      "| turbo: unknown",
      "| warmup discarded:", WARMUP)

Reading everyone else's numbers, including mine

Apply the same list as a reader. When a sandbox vendor publishes a cold-start figure, the questions that decide whether it means anything are: was the host otherwise idle, was there warmup, is that a mean or a percentile, how many samples, and what was the neighbour load. A published number that answers none of those was measured in the most flattering configuration available, because that is what happens by default when nobody is watching for it.

That includes our own. Our 179ms p50 and roughly 203ms p99 for create come from our hosts, running our fleet's real mix, and I try to publish the conditions alongside them on the benchmarks page rather than let the number float free. It is still a measurement of a particular fleet on particular silicon at a particular density. The right response to any vendor number, mine included, is to run the harness above against your own account and your own workload.

The thing I would push back on hardest is cross-vendor comparison tables where every row was produced by a different person on a different day. Those are not comparisons. Frequency variance alone can move a CPU-bound result by more than the gap the table is claiming to demonstrate.

What we actually do about it

Being concrete about our own position, including the parts that are unresolved.

• Templates bake a fixed vCPU count — 8 on our first-party templates — and it is a burst ceiling, not a reservation. Firecracker cannot change vCPU or RAM at snapshot restore, so this is a build-time decision; the agent overrides a create request's CPU and memory to match the baked snapshot rather than silently ignoring them.
• Under contention, cgroup v2 cpu.weight arbitrates. Each VM's cgroup gets a weight of 100 times its vCPU count, so an 8-vCPU tier genuinely outweighs a 2-vCPU one instead of it being a thread-count lottery. Weights bind only when cores are contended; on a quiet host a guest bursts across whatever is free.
• CPU bills on active CPU-seconds scraped from cpu.stat, memory on committed GiB-hours. One rate card across classes.
• Per-sandbox core pinning exists in the agent — a configured pool of cores, round-robin assigned, vCPU threads pinned via affinity — but it is off unless a pool is configured, and it is aimed at cache and TLB jitter rather than at frequency.
• We do not pin the CPU governor on our hosts, and on virtualised hosts we largely cannot. Frequency is the underlying provider's decision. Pretending otherwise would be the marketing answer.
• The scheduler scores free cores and free memory. It has no notion of how fast a host's cores are. On a heterogeneous fleet that is a real gap, and the honest mitigation today is consumption-based CPU billing rather than a smarter score.

If I were to fix the scheduler gap properly, the shape would be a periodic calibration probe per host — a fixed unit of work, run on a schedule, producing a relative performance factor that placement could weight by. That is not built. It is the sort of thing that sounds trivial and turns out to need careful thought about when to sample, how to avoid the probe itself perturbing what it measures, and what to do when the factor drifts under load, which is precisely when you would most want to trust it.

The summary

A vCPU buys you time on a core, not a rate of work. The rate moves with the governor, with the turbo bin your neighbours have pushed the socket into, with power and thermal budgets that have their own time constants, with instruction mix on some parts, and with how long your job runs before the frequency has ramped at all.

For short-lived sandboxes that stack is close to worst case, and the effect is systematically flattering in benchmarks and systematically unflattering in production — an idle host with a warmed-up loop is the best clock you will ever see, and a dense host serving a 200ms burst is close to the worst. The gap between those two is real hardware behaviour, and it gets misattributed to platform software constantly.

So: warm up and discard, pin the governor when you can and admit when you cannot, report percentiles with the host conditions attached, and treat any published sandbox benchmark that omits the host state as a marketing artifact rather than a measurement. And if you are building the platform rather than measuring it, bill for what was consumed instead of what was promised, because that is the one design decision here that survives the customer landing on a slow host.