Firecracker vCPU Hotplug, and How CPU Scaling Actually Works

The ticket always arrives in the same shape. A build inside a sandbox is slower than it is on a laptop, and the conclusion is "give it more CPU." On a container platform that's a one-line change to a resource limit. On Firecracker the first answer is: you cannot add a vCPU to a running microVM, there is no API for it, and there is not going to be one. The second answer — the useful one — is that "more CPU" was never about vCPU count, and the knob you wanted lives on the host, not in the guest.

I'm Ajay, I built PandaStack — this post is about why Firecracker deliberately has no vCPU hotplug, why snapshots make that constraint even stricter, and what the two actual levers are for giving a workload more compute: how many vCPU threads exist (fixed at boot) and how much host CPU time those threads get (adjustable at runtime, via cgroups). Getting those two confused is the source of most "why is my 8-vCPU VM slow" confusion.

vcpu_count is a boot-time decision, and boot happens once

Firecracker's control surface is a small REST API over a Unix socket. Before you start the machine you configure it: the kernel and boot args, the block devices, the network interfaces, and the machine shape — `vcpu_count` and `mem_size_mib` — via `PUT /machine-config`. Then you `PUT /actions` with `InstanceStart` and the VMM builds the machine. From that instant, the shape is settled. The pre-boot configuration surface closes, and there is no post-boot endpoint that adds a CPU.

FC_SOCK=/run/firecracker/fc.sock

# 1. Set the machine shape. This is the ONLY moment vCPU count is negotiable.
curl --unix-socket "$FC_SOCK" -i \
  -X PUT 'http://localhost/machine-config' \
  -H 'Content-Type: application/json' \
  -d '{
        "vcpu_count": 8,
        "mem_size_mib": 4096,
        "smt": false,
        "track_dirty_pages": true
      }'

# 2. Boot it. Eight host threads enter KVM_RUN; the guest kernel enumerates 8 CPUs.
curl --unix-socket "$FC_SOCK" -i \
  -X PUT 'http://localhost/actions' \
  -H 'Content-Type: application/json' \
  -d '{"action_type": "InstanceStart"}'

# 3. Change your mind. This is the part that does not work.
curl --unix-socket "$FC_SOCK" -i \
  -X PATCH 'http://localhost/machine-config' \
  -H 'Content-Type: application/json' \
  -d '{"vcpu_count": 16}'
# The machine is running, so the machine-config surface is no longer writable --
# and there is no "hotplug a vCPU" action to fall back to. Field names and
# accepted values drift between releases; check the Firecracker API spec for yours.

What `vcpu_count` actually creates is threads. Firecracker spawns one host thread per vCPU, and each of those threads sits in a loop: call `KVM_RUN` on its vCPU file descriptor, let the hardware execute guest instructions until something forces an exit (an MMIO access to a virtio queue, a halt, a signal, a timer), handle the exit in userspace, call `KVM_RUN` again. That's the whole model, and it's covered in more depth in /blog/firecracker-vcpu-scheduling-model. The important consequence here is that the vCPU is a host thread, and threads are created when the machine is built.

The guest side is even more final. At boot, the guest kernel is handed a static description of its processors — an MP table or ACPI tables on x86_64, a flattened device tree on aarch64 — plus the CPUID leaves the VMM chose to expose. It walks that description exactly once: per-CPU areas, scheduler domains, run queues, APIC IDs. Nothing in that sequence is designed to be re-run. A guest kernel doesn't poll for new processors; it has to be told, through a device it already trusts, that one has appeared.

Why there's no hotplug: it's a device problem, not a CPU problem

Adding a CPU to a live guest is not one feature. It's a small stack of them, and every layer has to cooperate.

1. The VMM has to create a new vCPU fd and thread mid-flight, initialize its register state to something the guest will accept as a freshly-arrived processor, and wire it into an interrupt topology the guest already cached.
2. The platform needs a way to notify the running guest at all. On x86 that means ACPI: a CPU object in the namespace, a general-purpose event, an interrupt into the guest's ACPI driver saying "re-evaluate the processor list." That is a non-trivial firmware surface.
3. The guest OS needs the matching support compiled in, and on Linux the new CPU usually still arrives offline — something in userspace has to write to /sys/devices/system/cpu/cpuN/online before a task is ever scheduled on it.
4. Everything downstream has to cope with the topology changing under it: NUMA layout, CPU masks, IRQ affinity, and every thread pool that sized itself at startup from a CPU count that is now a lie.

Firecracker's entire design thesis is that it doesn't have that stack. It ships a deliberately tiny device model — a handful of virtio devices, a serial console, the minimum needed to boot and do useful work — because it was built to run untrusted, multi-tenant code where the VMM is the security boundary. Every emulated device is code that parses guest-controlled input in the host's most privileged userspace process. CPU hotplug would add a firmware and device surface that exists to solve a problem serverless workloads already solve a different way: if you need more machine, start another machine.

The most secure device is the one you never implemented. Firecracker's feature list is short on purpose — it's an attack-surface budget, not an oversight.

Other VMMs make the opposite trade and are right to. QEMU has a full-fat device model with mature ACPI support and implements CPU hotplug on supported machine types — you add a CPU object to a running guest, then online it inside the guest. Cloud Hypervisor was designed for elastic cloud guests and exposes a resize path built on a boot-vCPUs versus max-vCPUs split, so the topology has room reserved for growth. Both descriptions are qualitative and both projects move quickly; verify the mechanism and its limits against their current docs before designing around it.

Snapshots freeze the shape even harder

If you only ever cold-boot microVMs, "pick the vCPU count before InstanceStart" is mildly annoying. If you restore from snapshots — which is how any platform gets sub-second creates — it becomes a hard structural property of your templates.

A Firecracker snapshot is a memory file plus a serialized VM state file, and that state file contains the per-vCPU state: registers, model-specific registers, the local APIC, the whole saved condition of each virtual processor. The count is not metadata you can edit on the way in; it's the shape of the data. Restore reconstructs exactly the vCPUs that were saved. You cannot restore a 2-vCPU snapshot as an 8-vCPU machine any more than you can restore a process core dump with extra threads bolted on.

This is the same constraint that makes baked RAM fixed — memory size is likewise part of the saved machine, which is why ballooning rather than resizing is the memory story (see /blog/firecracker-memory-hotplug-vs-ballooning). On PandaStack it shows up as a rule that surprises people exactly once: the agent overrides the CPU and memory values in a create request to match the baked snapshot. If the `base` template was baked at 8 vCPU and 4 GiB, that is what you get, whatever the API call said.

The two levers: how many threads, and how much time

Here's the reframe that makes all of this tractable. "CPU" in a virtualized system is two separate quantities that people say with the same word:

• Lever 1 — how many vCPU threads exist. This is `vcpu_count`, set before boot and frozen into the snapshot. It determines the guest's parallelism ceiling: how many tasks can be genuinely running at the same instant, and what number every thread pool in the guest reads out of nproc.
• Lever 2 — how much host CPU time those threads actually receive. This is pure host-side scheduling: cgroup v2 `cpu.weight` for proportional shares under contention, and `cpu.max` for a hard quota. It is adjustable at any moment, on a running VM, with no guest cooperation whatsoever.

Lever 2 is the one the slow-build ticket actually wanted. And it's a file write. Firecracker's jailer already places the VMM process into its own cgroup; from the host you can retune it live while the guest keeps running, oblivious.

# One microVM's cgroup (the jailer places the firecracker process here).
CG=/sys/fs/cgroup/pandastack/sbx-4f2c

# The guest believes it has 8 CPUs, because it has 8 vCPU threads.
awk '{print "pid", $1}' "$CG/cgroup.procs"

# cpu.max is "<quota_us> <period_us>" -- a hard ceiling across ALL vcpu threads.
cat "$CG/cpu.max"                       # "max 100000" = no quota
echo "200000 100000" > "$CG/cpu.max"    # ceiling: 2 CPUs' worth of time, total

# cpu.weight is a proportional share (1-10000, default 100). It only bites
# when the host is contended -- an idle host gives a weight-1 cgroup everything.
cat "$CG/cpu.weight"
echo "400" > "$CG/cpu.weight"           # 4x the share of a default neighbour

# Prefer weights to quotas for bursty work: drop the hard cap, keep the share.
echo "max 100000" > "$CG/cpu.max"

# What it actually burned. usage_usec is the honest billing signal --
# CPU-seconds consumed, not vCPUs allocated.
grep -E 'usage_usec|nr_throttled|throttled_usec' "$CG/cpu.stat"

Once you see it this way, oversubscription stops being a dirty word and becomes the plan. Bake generously: PandaStack's templates carry 8 vCPU precisely because that's the burst a `npm run build` wants for ninety seconds and never wants again — the guests are not each reserving eight cores. Under contention, `cpu.weight` arbitrates between neighbours and the machine gets allocated to whoever is actually running. It's also why the honest billing unit is CPU-seconds burned, not vCPUs allocated: eight idle vCPU threads cost the host approximately nothing, so charging for the ceiling would be charging for a number with no cost behind it. Memory is the opposite — a committed GiB is committed whether you touch it or not — which is why the two bill on different units. More in /blog/cgroups-v2-explained-for-sandboxing.

The footgun: nproc lies to your build

The two levers are independent, and the guest can only see one of them. Inside the microVM, `nproc` reports 8 because there are genuinely 8 vCPUs. The host cgroup that's currently handing those 8 threads two cores' worth of time is not visible from inside the guest — it isn't the guest's cgroup, it's the VMM's, on the other side of a hardware boundary. So every piece of software that autosizes itself from the CPU count will size to the ceiling and then queue.

This cuts the opposite way from what people expect. Modern runtimes have gotten good at container awareness — reading `cpu.max` out of the cgroup filesystem and sizing thread pools to the quota rather than the host's core count. That machinery is useless in a VM. Not broken: useless. There is no cgroup limit inside the guest to read. The throttle is entirely outside, and the guest scheduler will cheerfully believe it has eight processors to hand out while the host gives it two cores' worth of runtime, sliced.

• `make -j$(nproc)` and `cargo build -j 8` — eight compiler processes, each with its own resident set, all making progress in slow motion. The real damage is often memory, not time: eight concurrent link steps against a fixed baked RAM ceiling is how you meet the guest OOM killer.
• Go's GOMAXPROCS — defaults from the CPU count the runtime observes, so the scheduler runs 8 Ps and happily oversubscribes. Usually harmless, occasionally not: lock contention and GC assist work scale with P count.
• Node's os.cpus().length and cluster workers — fork-one-per-CPU forks eight, each with a full V8 heap, in a guest sized for one or two.
• Test runners — `pytest -n auto`, `jest --maxWorkers=100%`, `vitest` pools: all read the same number, all wrong in the same direction.
• Postgres and JVM heuristics — parallel worker and GC thread counts derive from the visible processor count: fine at the ceiling, wasteful under a tight quota.

The fix isn't clever, it's explicit: if a workload's concurrency matters, pin it. `make -j4`, `GOMAXPROCS=4`, `--maxWorkers=4`. Treat the number the guest reports as the maximum it is ever allowed to be, and set the real one from what you know about the host budget.

Steal time: the guest's one honest signal

The guest can't see the host's cgroup, but it isn't completely blind. Steal time is the accounting for "this vCPU was runnable and wanted to execute, and the host did not schedule it." It's the eighth field on the aggregate `cpu` line in /proc/stat, the `%st` column in top, and the `st` column in vmstat. When your throughput drops but user and system time look normal and idle isn't rising, steal is where the missing seconds went.

It's the right signal to build on precisely because it distinguishes the two failure modes people confuse. High user time means your code is slow. High iowait means you're waiting on storage. High steal means you're being arbitrated — either the host is oversubscribed right now, or someone set a `cpu.max` quota and you're hitting it; from inside the guest those look identical. Both are host-side answers, and neither improves by optimizing your inner loop.

vCPU hotplug across Firecracker, QEMU, Cloud Hypervisor, and containers

• Runtime vCPU hotplug — Firecracker: none; `vcpu_count` is pre-boot only and there's no post-boot action to add one. QEMU: implemented on supported machine types via its full ACPI device model, with the guest still needing to online the new CPU. Cloud Hypervisor: supported through a boot-vCPUs versus max-vCPUs split plus a resize call, designed in from the start. Containers: not applicable — there are no vCPUs at all, only host threads against a quota.
• How you change capacity at runtime — Firecracker: only host-side, via cgroup v2 `cpu.weight`/`cpu.max` on the VMM process. QEMU: hotplug, plus the same host-side cgroup levers. Cloud Hypervisor: resize API, plus the same host-side levers. Containers: rewrite `cpu.max`/`cpu.weight` and you're done, instantly, with no guest involvement.
• Snapshot interaction — Firecracker: vCPU count is part of the serialized VM state, so a snapshot restores at exactly its baked count. QEMU and Cloud Hypervisor: migration and snapshot machinery must reconcile topology with saved state, which constrains when a resize is safe — check their docs. Containers: checkpoint/restore exists, but there is no CPU topology to preserve.
• Device-model cost — Firecracker: minimal by design; no ACPI CPU objects, no hotplug controller, less code parsing guest input in the host. QEMU: large and feature-rich, which is what makes hotplug possible. Cloud Hypervisor: middle ground, Rust, elasticity as a design goal. Containers: no device model at all — the boundary is the shared host kernel.
• Best fit — Firecracker: dense multi-tenant, untrusted, short-lived workloads where you'd rather start a second VM than grow the first. QEMU: long-lived general-purpose VMs where in-place resize beats rebuilding. Cloud Hypervisor: modern cloud guests that genuinely need to grow and shrink. Containers: trusted first-party workloads where a quota change is the whole scaling story and a shared kernel is an acceptable boundary.

The container row explains why this topic feels weird coming from Kubernetes. A container has no CPU count at all. "2 CPUs" is a quota — a ratio of runtime to a scheduling period — rewritable mid-flight because there is nothing to renegotiate with: the kernel doing the scheduling is the kernel the workload runs on. That flexibility is the direct dividend of the shared kernel, which is also the isolation boundary you gave up to get it. A container is a polite suggestion to the kernel; a microVM is a locked room with a fixed number of chairs.

Practical guidance: pick the shape at bake time

The working rule is that vCPU count is a ceiling you choose once, not a reservation you're paying for. So choose it generously, arbitrate with weights, bill on what burned, and stop trying to resize.

1. Pick vCPU count at bake time, for the peak burst the workload will ever want. Eight is a reasonable default for a general-purpose template.
2. Do not treat it as a reservation. Eight idle vCPU threads cost the host essentially nothing; the cost is memory, which really is committed.
3. Arbitrate with `cpu.weight` rather than capping with `cpu.max` unless you need a hard ceiling — quotas throttle bursty work that would otherwise finish and get out of the way.
4. Pin application-level concurrency explicitly (`-j`, GOMAXPROCS, maxWorkers) instead of letting nproc decide. The guest's CPU count is the ceiling, not the current allowance.
5. Watch steal time from inside and `cpu.stat` from outside. They're the two halves of the same story.
6. If you genuinely need a different machine shape, bake a different template. That's the supported resize path, and it's cheaper than it sounds — the alternative is cold-booting every guest.

In an API this comes out as: you don't request a vCPU count, you request a template. The snapshot decides the shape, which is why creates land at a p50 of 179ms instead of the ~3s a cold boot costs.

from pandastack import Sandbox

# You pick capacity by choosing a template, not by asking for N vCPUs.
# The baked snapshot's shape wins -- that's what makes restore-on-create possible.
sbx = Sandbox.create(template="code-interpreter", ttl_seconds=900)

# The guest enumerated its CPUs at boot, from the snapshot's saved state.
print(sbx.exec("nproc").stdout.strip())          # -> the baked ceiling

# Anything that autosizes from nproc sizes to the CEILING, not to the host
# time you're actually being given right now. So pin it yourself.
sbx.filesystem.write("/work/build.sh", "make -j4 all")
res = sbx.exec("cd /work && sh build.sh", timeout_seconds=600)
print(res.exit_code, res.stdout[-2000:])

# Steal time: the one host-side fact the guest can observe.
# Field 8 of the aggregate /proc/stat cpu line.
print(sbx.exec("awk '/^cpu /{print $9}' /proc/stat").stdout.strip())

# Need a different shape? Snapshot the work, not the machine, and
# fork it -- same-host forks land in 400-750ms.
snap = sbx.snapshot()
sbx.kill()

None of this is Firecracker being underpowered. It's Firecracker being honest about what a virtual machine is: a fixed set of virtual processors, created at boot, competing for real ones. The elasticity lives on the host where the real CPUs are — and it's better there anyway. A file write beats an ACPI event, and "start another microVM" beats "grow this one" when starting one costs a fraction of a second.

If this was useful, the neighbouring pieces go deeper on each half: /blog/firecracker-memory-hotplug-vs-ballooning covers the identical constraint on the memory side and why ballooning is the answer there, /blog/cgroups-v2-explained-for-sandboxing is the practical guide to `cpu.weight`, `cpu.max`, and the memory controllers you'll pair them with, and /blog/firecracker-vcpu-scheduling-model gets into the KVM_RUN loop, the thread layout, and how the host scheduler actually decides which guest runs next.