How Firecracker Schedules vCPUs: The Threading Model

Here's the mental model that makes everything about Firecracker CPU behavior click: a guest vCPU is just a host thread wearing a convincing hat. When your microVM thinks it has 2 CPUs, what actually exists on the host is two ordinary Linux threads inside the Firecracker process, each sitting in a tight ioctl loop, each schedulable by the host kernel exactly like any other thread. There is no magic CPU allocator, no hypervisor scheduler carving up cores. It's threads all the way down, and the host's normal CFS scheduler runs the show.

That single fact explains why you can oversubscribe vCPUs wildly across many idle microVMs, why there's no CPU pinning by default, why a noisy neighbor needs cgroups to contain, and why vCPU count is a fixed property of a baked snapshot. I'm Ajay — I build PandaStack on Firecracker, so I've spent a lot of time staring at these threads under load. Let's walk the whole model, from the KVM_RUN loop up to host-level density.

The thread layout: VMM + one thread per vCPU + API/IO

A running Firecracker microVM is a single host process with a small, fixed set of threads. There's no thread pool that grows with load — the layout is deterministic and boringly simple, which is exactly the point for a security-focused VMM.

• The VMM (main) thread — sets up the VM via KVM, owns the device emulation state machine, and pumps the virtio device queues (block, net, vsock) plus the epoll event loop that drives I/O.
• One vCPU thread per guest CPU — if machine-config says vcpu_count: 2, Firecracker spawns exactly two vCPU threads, each bound to one KVM vCPU file descriptor, each running its own KVM_RUN loop.
• The API thread — serves the control-plane HTTP API over a Unix socket (the PUTs that configure the machine, boot source, drives, and issue snapshot/restore). It's idle once the VM is running.
• A couple of housekeeping threads — e.g. the metrics/heartbeat flusher. These barely register on the scheduler.

So a 2-vCPU microVM is roughly: 1 VMM thread + 2 vCPU threads + 1 API thread + housekeeping. When people say Firecracker has a "one thread per vCPU" model, this is what they mean — the vCPU-to-thread mapping is 1:1 and static for the life of the VM. No guest can conjure a third vCPU thread at runtime; the count is fixed at boot.

Inside a vCPU thread: the KVM_RUN loop

Each vCPU thread does one thing forever: it calls the KVM_RUN ioctl on its vCPU fd. That ioctl says to the kernel, "switch this hardware thread into guest mode and run guest instructions until something happens that you can't handle in hardware." On Intel/AMD this is a VM-entry; the CPU executes the guest's kernel and userspace natively at full speed. There's no interpretation, no instruction emulation on the hot path — the guest runs on real silicon.

KVM_RUN returns to the vCPU thread only when the guest does something that needs host attention — a VM-exit. The classic cases are MMIO and PIO exits: the guest touched a memory or port address mapped to a virtio device (say it kicked the virtio-net queue to send a packet). KVM can't service that in hardware, so it exits back to userspace with an exit reason, and the vCPU thread hands the operation to the VMM's device model, which does the emulation and then re-enters KVM_RUN. Halts (the guest idles with HLT/WFI) also exit; the thread blocks until an interrupt wakes it, which is why an idle guest burns essentially zero host CPU.

vCPU thread (conceptually):

  loop {
      ioctl(vcpu_fd, KVM_RUN)      // VM-entry: run guest natively
      // ... returns on a VM-exit ...
      match exit_reason {
          MMIO  => vmm.handle_mmio(addr, data)   // virtio device access
          PIO   => vmm.handle_pio(port, data)
          HLT   => block_until_interrupt()       // idle guest -> ~0% host CPU
          _     => ...
      }
  }

The host owns scheduling: CFS decides who runs

Firecracker does not schedule vCPUs. It creates the threads and calls KVM_RUN; from there the host Linux scheduler — CFS (or EEVDF on newer kernels) — decides which runnable thread gets a physical core and for how long. A vCPU thread is, to the host, indistinguishable from any other runnable thread. It competes for cores against every other vCPU thread on the box, against the VMM threads, against your API server, against kubelet, against everything.

This is elegant because it means all the mature Linux scheduling machinery just applies. Fairness, nice levels, cgroup CPU controllers, cpusets, load balancing across cores — you get it for free, with no VMM-specific scheduler to tune or trust. The flip side: with no other controls, the host treats a 64-vCPU noisy microVM and a 1-vCPU quiet one as 64 threads vs 1 thread competing for time. Fair at the thread level is not the same as fair at the tenant level, which is where cgroups come in.

Why there's no CPU pinning by default

Out of the box, a vCPU thread can run on any host core, and CFS will migrate it between cores to balance load. Firecracker deliberately doesn't pin — pinning is a host-level operational decision, not something a general-purpose VMM should impose. For dense, bursty, multi-tenant fleets (the Lambda-style workload Firecracker was built for), letting CFS float threads across all cores maximizes utilization: an idle vCPU's core is instantly available to a busy one.

But floating has costs. A migrated vCPU thread lands on a cold cache and pays for it; cross-NUMA migration is worse. For latency-sensitive or benchmark-stable workloads, you often want to pin. Because vCPUs are just threads, you pin them with the standard Linux tools — no Firecracker feature required. Find the vCPU thread TIDs (Firecracker names them fc_vcpu 0, fc_vcpu 1, …) and taskset or cpuset them:

# The Firecracker process PID for one microVM:
FC_PID=$(pgrep -f 'firecracker --api-sock')

# List its threads; vCPU threads are named fc_vcpu N.
ps -T -p "$FC_PID" -o tid,comm | grep fc_vcpu
#   TID  COMMAND
#  4812  fc_vcpu 0
#  4813  fc_vcpu 1

# Pin each vCPU thread to a dedicated physical core (taskset).
taskset -pc 2 4812   # fc_vcpu 0 -> core 2
taskset -pc 3 4813   # fc_vcpu 1 -> core 3

# Or, better for fleets: place the whole microVM in a cpuset cgroup (v2)
# so every thread it spawns inherits the CPU set atomically.
mkdir -p /sys/fs/cgroup/fc-vm-abc
echo '2-3' > /sys/fs/cgroup/fc-vm-abc/cpuset.cpus
echo '0'   > /sys/fs/cgroup/fc-vm-abc/cpuset.mems   # NUMA node 0
echo "$FC_PID" > /sys/fs/cgroup/fc-vm-abc/cgroup.procs

The jailer: where cgroup placement actually happens

In production you don't run the firecracker binary directly — you run it under the jailer, Firecracker's companion that drops the VMM into a locked-down sandbox (chroot, dropped capabilities, a seccomp filter, its own PID/network namespaces) before the VM ever boots. The jailer is also where CPU cgroup placement lives: it can put the Firecracker process into a specific cgroup at launch (via --cgroup and --cgroup-version), so every thread the VMM spawns — VMM, vCPUs, API — inherits that cgroup's CPU controls. That's how you bound a microVM's CPU without touching individual TIDs after the fact.

The two knobs that matter for CPU:

• cpu.max (cgroup v2) — a hard bandwidth cap: "this cgroup may use at most N microseconds of CPU per 100ms window." Set cpu.max to 50000 100000 and the whole microVM is capped at 0.5 of a core, no matter how many vCPUs it thinks it has. This is your ceiling against a runaway guest.
• cpu.weight — a proportional share (the CFS successor to cpu.shares): when cores are contended, weight decides who gets more time. It only bites under contention; an idle box lets everyone run freely. This is how you keep a busy tenant from starving a quiet one without hard-capping anyone.

Together, cpu.max plus cpu.weight are how you tame noisy neighbors: the cap prevents any single VM from eating the host, and the weight enforces fair-share when the box is hot. It layers cleanly with Firecracker's own I/O rate limiter, which throttles block and network throughput per device. CPU rate limiting via cgroups and virtio I/O rate limiting are complementary — one bounds compute, the other bounds bandwidth. See our writeup on the Firecracker rate limiter (/blog/firecracker-rate-limiter-explained) for the I/O side.

Oversubscription: packing more vCPUs than cores

Because a vCPU is just a thread, and an idle vCPU thread is blocked (consuming no core time), you can allocate far more vCPUs across your microVMs than you have physical cores. A 32-core host can comfortably run hundreds of 2-vCPU microVMs if their guests are mostly idle — which, for AI agent sandboxes, per-user notebooks, code interpreters, and preview environments, they usually are. The sum of allocated vCPUs vastly exceeds the core count, and nobody notices, because at any instant only a small fraction of those threads are actually runnable.

The limit isn't a vCPU count — it's the runnable working set. Density holds as long as the number of simultaneously computing guests stays under (roughly) your core count. When too many guests get busy at once, they don't fail; they slow down, sharing cores via CFS. That graceful degradation is exactly the behavior you want for bursty, mostly-idle fleets, and it's why memory — not CPU — is usually the real binding constraint on how many microVMs a host can hold. For the full economics, see overcommit and microVM density (/blog/overcommit-microvm-density).

A gotcha: vCPU count is baked into the snapshot

Here's the constraint that surprises people building on snapshot-restore: Firecracker cannot change vcpu_count or mem_size_mib at snapshot restore. The vCPU count is captured in the machine state when you snapshot, and the restore must recreate exactly that topology — same vCPUs, same memory. You set it once, at cold boot, in machine-config:

PUT /machine-config
{
  "vcpu_count": 2,
  "mem_size_mib": 2048,
  "smt": false,
  "track_dirty_pages": true
}

On PandaStack, every sandbox is created by restoring a baked template snapshot (not cold-booting), which is how a create is ~49ms restore / p50 179ms rather than a ~3s first cold boot. The direct consequence of the restore constraint: a template's vCPU count (and RAM) is a property of the baked snapshot, not something you pass per-create. If you ask for a sandbox from a 2-vCPU template, the agent honors the baked topology — the vCPU count comes from the snapshot, full stop.

from pandastack import Sandbox

# vCPU count and RAM are baked into the template's snapshot.
# The create restores that exact topology in ~49ms; you don't
# pass vcpu_count per-create because restore can't change it.
with Sandbox.create(template="code-interpreter") as sbx:
    info = sbx.exec("nproc && free -m | awk '/Mem:/{print $2\" MiB\"}'")
    print(info.stdout)   # e.g. "2\n2048 MiB" -> matches the baked snapshot

This is why different CPU sizes are different templates: to offer a 4-vCPU tier you bake a 4-vCPU snapshot, not tweak a flag at create time. If you want to shape the guest's view of the CPU (features, model, to keep snapshots portable across host generations), that's a separate mechanism — CPU templates — covered in Firecracker CPU templates explained (/blog/firecracker-cpu-templates-explained).

1 thread/vCPU vs QEMU vs container CPU shares

Firecracker's model is deliberately the simplest of the three. Verify specifics against each project's current docs — the shapes, not the numbers, are the point:

• Firecracker / KVM — A: strictly one host thread per vCPU, static at boot, each in a KVM_RUN loop. B: no built-in pinning or CPU scheduler; the host CFS schedules the threads. C: bound and shaped entirely with standard Linux tools (taskset, cgroup cpuset, cpu.max, cpu.weight), placed via the jailer.
• QEMU / KVM — A: also one thread per vCPU by default, but the model is richer and more configurable — many device backends, optional iothreads for block I/O, threaded/multiqueue virtio, and knobs for vCPU pinning and NUMA topology. B: far more surface area and features (full-featured VMM), at the cost of a much larger footprint and attack surface. C: use it when you need device breadth or fine-grained topology control; Firecracker strips all of that out on purpose. Verify against QEMU docs.
• Containers (cgroups) — A: no vCPUs at all — container processes are just host threads sharing the host kernel, scheduled directly by CFS. B: CPU limits are the same cgroup knobs (cpu.max as the CFS quota behind Docker's --cpus, cpu.weight behind --cpu-shares). C: the isolation boundary is the shared kernel, not a hypervisor — lighter, but a weaker security boundary for untrusted code. Verify against your runtime's docs.

Notice the through-line: Firecracker and containers both lean on the exact same Linux cgroup CPU controllers — cpu.max and cpu.weight. The difference is the isolation boundary underneath (a guest kernel behind KVM vs a shared host kernel), not the CPU-accounting mechanism. Firecracker's bet is that a VMM should be a thin, auditable layer over KVM and let the battle-tested host scheduler do the scheduling. Once you see vCPUs as ordinary threads, everything — pinning, oversubscription, noisy-neighbor control, density limits — follows from how Linux already schedules threads. No new scheduler to learn; just threads, cores, and cgroups.