CPU Pinning and Noisy Neighbors in microVM Fleets
Here's the thing that surprises people about Firecracker: a guest vCPU is not a special isolated core, it's just a thread. When Firecracker boots a two-vCPU microVM, that's two host threads, and they get scheduled by the host's Linux scheduler (CFS, or EEVDF on newer kernels) alongside every other guest's threads and the host's own processes. Which means the entire beautiful hardware-isolation story — separate guest kernel, KVM boundary, the works — says nothing about CPU time. Two guests can be perfectly isolated from each other's memory and filesystem while one of them steals the other's compute wholesale.
That's the noisy-neighbor problem, and on a dense multi-tenant host it's not a corner case — it's the default behavior you have to actively manage. Overcommit is a promise you make to every tenant that they'll all be your favorite; the host CFS scheduler is where that promise goes to get audited. One tenant's CPU-bound job, spin loop, or (the classic) crypto miner can starve its neighbors, fatten everyone's p99, and — worse for a sandbox platform — inflate the tail latency of the boot/restore path itself. I'm Ajay; I build PandaStack, a Firecracker platform where dense overcommit is the whole economic model, so noisy-neighbor management isn't optional, it's the operational discipline. This post is the toolbox: pinning, cgroup weights and quotas, NUMA, and the density tradeoff that decides which you reach for.
Why a vCPU can starve its neighbors
Density comes from overcommit: you sell more vCPUs than you have physical cores, betting that most guests are idle most of the time. That bet is usually right and it's what makes microVMs cheap — an idle guest's vCPU thread costs almost nothing. But the bet has a failure mode. When several guests want CPU at once, or one guest wants all of it forever, the host scheduler has to divide real cores among more demand than exists, and by default it does so fairly-per-thread — which is not the same as fairly-per-tenant or fairly-per-SLA.
- A busy guest is a greedy thread. A guest running a tight compute loop keeps its vCPU threads runnable constantly, so the scheduler keeps handing them time slices. Neighbors that wake up to do a little work wait behind it, and their latency climbs.
- Firecracker's rate limiters don't help here. The built-in rate limiters bound block and network I/O — they throttle disk and packet throughput, not CPU. CPU fairness is entirely a host-scheduler and cgroup concern; if you reach for the FC rate limiter expecting it to tame a compute-bound neighbor, you've reached for the wrong knob.
- The boot/restore path competes too. Your create pipeline — restore a snapshot, resume the guest, probe readiness — runs as host work on the same cores. A neighbor pinning the CPU can inflate the latency of creating new sandboxes, turning one greedy tenant into a platform-wide slowdown of the thing you most want to stay fast.
- It's silent until it's a page. Nothing errors. p50 stays fine because most requests still land on idle cores. p99 quietly rots as a fraction of requests queue behind the noisy neighbor — which is exactly the tail you can't screenshot but can definitely get paged for.
The controls, from soft to hard
cgroup v2 cpu.weight — proportional shares
The softest and often best knob. Put each guest's Firecracker process (its vCPU threads) in its own cgroup and set `cpu.weight`. Weight is proportional: under contention, a guest gets CPU in proportion to its weight relative to the others competing for the same cores. Crucially, when there's no contention, a guest can still burst and use idle cores — weight only bites when demand exceeds supply. This preserves overcommit density (idle capacity is still usable) while ensuring that under pressure, no single guest can take more than its fair proportional share. It's the density-friendly answer to noisy neighbors.
cgroup v2 cpu.max — hard quota
When you need a firm ceiling, `cpu.max` sets a hard quota: this cgroup may use at most N microseconds of CPU per M-microsecond period, and gets throttled when it hits the cap — even if cores are idle. This bounds a guest's absolute CPU consumption, which is what you want for a tenant tier with a contractual CPU limit, or to defang a runaway loop so it simply can't exceed its allotment. The cost is that it caps burst: a guest that could have used idle capacity is throttled anyway, so you trade some efficiency for predictability. Use quota for hard limits, weight for fair sharing; often you want both — a weight for fair contention behavior and a max as a hard backstop.
CPU pinning / affinity — dedicated cores
The hardest control: pin a guest's vCPU threads to specific physical cores (via `taskset`, `sched_setaffinity`, or a `cpuset` cgroup) so they run only there and nothing else runs there either. This buys the best latency predictability and cache locality — the guest's working set stays warm in the pinned cores' caches, and it never waits behind an unrelated thread. It's the right move for a latency-SLA tier. But it's expensive in the currency that makes microVMs cheap: pinning dedicates cores, which kills overcommit for those cores, fragments your capacity (bin-packing pinned guests is harder than sharing a pool), and leaves pinned cores idle when their guest is idle. Pin the tiers that pay for predictability; share everything else.
NUMA pinning + memory locality
On multi-socket hosts there's a second axis: NUMA. A vCPU thread scheduled on one socket accessing memory allocated on another pays a cross-socket latency penalty, and it varies run to run, which shows up as tail-latency jitter. Pinning a guest's vCPUs and its memory to the same NUMA node keeps memory access local and latency consistent. If you're also using hugepages to back guest memory, keeping those pages NUMA-local compounds the win. This matters most for latency-sensitive tiers and larger guests; for small bursty sandboxes it's usually noise.
The density-vs-predictability tradeoff
- No isolation (pure overcommit) — Predictability: worst; one greedy guest fattens everyone's p99. Density: best; every idle vCPU costs ~nothing. Fit: trusted/low-contention or dev.
- cpu.weight (proportional shares) — Predictability: good under contention, still allows burst. Density: excellent; idle capacity stays usable. Fit: the default for dense multi-tenant fleets.
- cpu.max (hard quota) — Predictability: strong absolute ceiling. Density: good, but caps burst so some idle capacity goes unused. Fit: contractual CPU tiers, defanging runaways.
- Hard pinning (cpuset/taskset) — Predictability: best; dedicated cores, warm caches, no queueing behind strangers. Density: worst; kills overcommit for pinned cores. Fit: latency-SLA tiers only.
- NUMA pinning — Predictability: removes cross-socket jitter. Density: reduces placement flexibility. Fit: large or latency-sensitive guests on multi-socket hosts.
The honest summary: hard pinning gives the best latency and destroys the economics that make microVMs worth using. Most dense platforms therefore live on cgroup weights (fair sharing that preserves burst and density), add a quota where a hard ceiling is required, protect the control plane and boot path with a reserved cpuset, and reserve full pinning + NUMA for a paid latency tier where the customer is buying predictability and you're pricing in the lost density. Pinning everything to solve noisy neighbors is like solving a crowded restaurant by giving every diner their own building — effective, and you're no longer running a restaurant.
Seeing the knobs
The mechanics are small (illustrative — wire these to each guest's cgroup/process in your agent, and verify paths against your cgroup v2 layout):
# Assume each microVM's Firecracker process lives in its own cgroup v2 group:
CG=/sys/fs/cgroup/pandastack/vm-abc123
# 1) Proportional fair share (density-friendly): default weight is 100.
# Under contention this guest gets CPU in proportion to its weight; when
# idle cores exist, it can still burst.
echo 100 > "$CG/cpu.weight"
# 2) Hard ceiling (predictable, caps burst): at most 150ms of CPU per 100ms
# period == ~1.5 cores, throttled even if cores sit idle.
echo "150000 100000" > "$CG/cpu.max"
# 3) Hard pinning for a latency-SLA guest: bind its vCPU threads to cores 4-5
# (a cpuset), and keep tenant work OFF the cores reserved for system work.
echo "4-5" > "$CG/cpuset.cpus"
# Or per-thread with taskset against the vCPU thread ids:
# taskset -pc 4 <vcpu0-tid>; taskset -pc 5 <vcpu1-tid>
# Reserve cores 0-1 for the agent + snapshot-restore path so a noisy tenant
# can't inflate CREATE latency for the whole host:
echo "0-1" > /sys/fs/cgroup/system-reserved/cpuset.cpusWhy this ties back to the numbers that define a sandbox platform: create latency is the product. A snapshot-restore create is p50 179ms and p99 about 203ms (the restore step around 49ms; a true cold boot only on first spawn, around 3s); a same-host fork is 400–750ms (cross-host 1.2–3.5s). Every one of those is host work that a saturated core inflates. Keeping the boot/restore path on protected cores is how you keep that p99 a p99 and not a story about the tenant who started mining at 3am.
Putting it together
Firecracker isolates memory and the kernel; it does not isolate CPU time, because a vCPU is just a host thread the Linux scheduler divides among everyone. On a dense, overcommitted host that makes noisy neighbors the default risk, not an edge case — one greedy guest can fatten every neighbor's p99 and, if you're careless, inflate the create latency that is your whole product. Manage it with the softest knob that works: cgroup `cpu.weight` for fair, burst-preserving sharing across the fleet; `cpu.max` where you need a hard ceiling; a reserved cpuset to keep the agent and the snapshot-restore path off cores tenants can saturate; and full pinning plus NUMA only for a latency tier that pays for it. Hard-pinning everything would make noisy neighbors disappear and take your density — the reason you chose microVMs — with them. The discipline isn't picking one knob; it's matching the knob to the tier, so most of the fleet runs dense and cheap while the workloads that need predictability get it.
Frequently asked questions
Why can one Firecracker microVM slow down others on the same host?
Because a guest vCPU is just a host thread scheduled by the host Linux scheduler (CFS or EEVDF), not a dedicated isolated core. Firecracker's hardware isolation covers memory and the kernel boundary, but says nothing about CPU time. On a dense host that overcommits vCPUs to physical cores, a guest running a tight compute loop keeps its vCPU threads constantly runnable, so the scheduler keeps giving them time slices and neighbors queue behind it. Their latency climbs — the classic noisy-neighbor effect — and it's silent: p50 often stays fine while p99 rots as a fraction of requests wait behind the greedy guest. Firecracker's built-in rate limiters bound disk and network I/O, not CPU, so CPU fairness is entirely a host-scheduler and cgroup concern.
Should I pin every microVM's vCPUs to dedicated cores?
Usually not — pinning everything solves noisy neighbors by destroying the economics that make microVMs worth using. Hard pinning gives the best latency predictability and cache locality because a guest's vCPU threads run only on their cores and nothing else runs there, which is ideal for a latency-SLA tier. But it dedicates cores, which kills overcommit, fragments capacity (pinned guests are harder to bin-pack), and leaves pinned cores idle when their guest is idle. Most dense platforms use cgroup cpu.weight for fair, burst-preserving sharing across the fleet, add cpu.max where a hard ceiling is needed, and reserve full pinning plus NUMA locality only for a paid latency tier where the lost density is priced in.
What's the difference between cpu.weight and cpu.max for controlling a guest?
cpu.weight is a proportional share: under contention, a guest gets CPU in proportion to its weight relative to others competing for the same cores, but when cores are idle it can still burst and use them. That preserves overcommit density while preventing any single guest from taking more than its fair share under pressure — it's the density-friendly default. cpu.max is a hard quota: the guest may use at most N microseconds of CPU per period and gets throttled at the cap even if cores sit idle, which bounds absolute consumption for a contractual CPU tier or defangs a runaway loop, at the cost of capping burst. They compose well: use a weight for fair contention behavior and a max as a hard backstop.
How do I stop a noisy tenant from slowing down new sandbox creation?
Reserve cores for the host's own work and keep tenant workloads off them. Your create pipeline — restoring a snapshot, resuming the guest, probing readiness — runs as host work on the same cores tenant guests use, so a guest saturating the CPU can inflate the latency of creating new sandboxes across the whole host. Put the agent and the snapshot-restore path in a dedicated cpuset (or give them a strongly reserved weight) so they run on cores tenants can't saturate. It's the cheapest, highest-leverage anti-noisy-neighbor move: a greedy tenant should be able to slow itself down, not turn one busy guest into a platform-wide slowdown of every create. This is what keeps a p99 create latency a real p99 rather than a story about one busy neighbor.
Do Firecracker's rate limiters help with CPU noisy neighbors?
No — that's a common wrong turn. Firecracker's built-in rate limiters throttle block-device and network throughput; they bound how much disk I/O and how many packets a guest can push, which is genuinely useful for I/O noisy neighbors. But they don't touch CPU scheduling at all. A compute-bound guest spinning in a loop isn't doing I/O, so no rate limiter will slow it down. CPU fairness lives entirely in the host scheduler and the cgroup cpu controller — cpu.weight for proportional sharing, cpu.max for a hard quota, and cpuset/affinity for pinning. If you're trying to tame a CPU-hungry neighbor, reach for cgroups and scheduling, not the Firecracker rate limiter.
49ms p50 cold start. Fork, snapshot, and scale to zero.