Bare metal vs cloud VMs for running Firecracker

People ask me which cloud PandaStack runs on the way they'd ask which laptop I use. It gets treated as a procurement detail, settled by whoever has credits. But if you are running microVMs you are operating a hypervisor, and the thing directly underneath a hypervisor decides how fast it goes, how densely it packs, and in a few cases whether your fast paths exist at all. That is an architecture decision wearing a purchase order.

I'm Ajay, and I build PandaStack — a Firecracker platform where every sandbox create is a snapshot restore at 179ms p50. Production today runs on cloud VMs with nested virtualization switched on, so read what follows as a ledger written by someone who picked the cloud side and knows exactly which bills that choice generates. There is no clean winner here. There is a load shape, and the load shape picks for you.

The first question is whether /dev/kvm exists

Firecracker is not a hypervisor in the way VMware is a hypervisor. It is a userspace VMM that drives KVM: it opens /dev/kvm, asks the kernel for a VM file descriptor, creates vCPU descriptors, and then spends its life in ioctl calls. Take the device away and there is no fallback, no software emulation mode, no degraded path. The process fails at startup and says so.

Which makes the device the first thing to check on any host you are evaluating, before instance families or price sheets. Our own bootstrap scripts fail closed on it rather than letting you discover the problem three layers deeper: the Linux path refuses to continue unless /dev/kvm exists and is readable and writable by the current user, and the Mac path verifies nested KVM inside the Lima VM before it will build anything. Both of those checks were added after I wasted an afternoon debugging a confusing error that turned out to mean, simply, no device.

#!/usr/bin/env bash
# Five questions, in the order that makes the answers useful.

# 1. Does the CPU advertise virtualization extensions at all?
#    vmx = Intel VT-x, svm = AMD-V. Absent on most Arm cloud instances,
#    which use a different mechanism and a different set of caveats.
grep -o -m1 -E 'vmx|svm' /proc/cpuinfo || echo 'no vmx/svm flag'

# 2. Am I already inside somebody's VM? If this prints, I am L1, and any
#    Firecracker guest I start will be L2.
grep -o -m1 hypervisor /proc/cpuinfo || echo 'no hypervisor flag (looks like metal)'
systemd-detect-virt || true            # kvm | vmware | microsoft | none

# 3. Did the provider actually expose the device to me? Flags in
#    /proc/cpuinfo mean nothing on their own -- plenty of cloud VMs show
#    vmx and still have no /dev/kvm, because nesting is off at the host.
ls -l /dev/kvm || echo 'NO /dev/kvm -- Firecracker cannot run here'

# 4. Is nesting enabled in the kernel module? Only meaningful on a host
#    you own; on a rented VM this reports the module you are running,
#    not your provider's decision one level up.
cat /sys/module/kvm_intel/parameters/nested 2>/dev/null
cat /sys/module/kvm_amd/parameters/nested   2>/dev/null

# 5. Can a process actually open it? Permissions bite as often as
#    capability does -- the device is usually root:kvm 0660.
python3 - <<'PY'
import fcntl, os
KVM_GET_API_VERSION = 0xAE00        # _IO(KVMIO=0xAE, 0x00)
fd = os.open("/dev/kvm", os.O_RDWR | os.O_CLOEXEC)
print("KVM API version:", fcntl.ioctl(fd, KVM_GET_API_VERSION, 0))  # expect 12
PY

Read the results together rather than one at a time. Flags present and device missing means the provider chose not to expose nesting on that instance family — no BIOS setting or kernel argument of yours will change it. Device present and hypervisor flag present means you have nesting and you are paying for it. Device present and no hypervisor flag means you are on metal, and the rest of this post is about what that is worth.

What nesting actually costs, mechanically

The usual shorthand is L0, L1, L2. L0 is your provider's hypervisor on the physical machine. L1 is the VM you rented. L2 is your Firecracker guest. Nothing about that stack is exotic — it has been production-grade on x86 for years — but the arithmetic of a VM exit changes, and Firecracker's performance story is largely an exit-and-page-fault story.

On metal, a guest exit is a trap into your kernel's KVM, which handles it and resumes. Under nesting, an L2 exit traps to L0 first. L0 inspects it, decides whether it can service it directly or must reflect it into L1's KVM, and only then does your VMM see anything. Some exits get handled entirely at L0 and stay cheap. Others take the long path through two hypervisors. The distribution depends on which exits your workload generates, which is why published overhead numbers for nested virtualization range from single-digit percentages to genuinely bad, and why quoting a single figure is close to meaningless.

Memory is the more interesting half for a snapshot-restore fleet. Hardware nested paging is designed for one level of translation; running a second requires the L0 hypervisor to maintain a merged mapping between the guest's page tables and L1's. Faults that would be handled once become work at two levels. Now look at what a snapshot restore actually does: it maps a memory file, resumes, and lets the guest fault its working set back in — with userfaultfd, our handler services those faults by pulling 4 MiB chunks and installing them. That path is a page-fault machine by design. It is precisely the workload where a second translation level is least free.

What I can point at is where the time in our create pipeline sits, so you know which parts a hardware change could plausibly move. A create is roughly: allocate a pre-built network slot (~1ms), patch the tap device in its namespace (~6ms), reflink the rootfs (~4ms), fork and exec Firecracker (~25ms), POST /snapshot/load (~80ms), resume (~6ms), probe TCP :22 until the guest answers (~40ms). The snapshot load and the readiness probe dominate, and both are dominated by memory and I/O behaviour rather than by anything in our Go code. That is the part of the budget hardware owns.

There is a second scheduler problem stacked on the second hypervisor. Your Firecracker vCPU is a thread on your kernel's run queue; your kernel's run queue lives on vCPUs that are themselves threads on your provider's run queue. Two schedulers, neither aware of the other, both making fairness decisions about the same physical core. In steady state you mostly do not notice. In the p99 — which for a sandbox platform is the number customers actually feel — you notice, and it shows up as steal time you cannot fix from inside.

Which instances even expose the feature

This is where the abstract argument becomes a very short shopping list. Nested virtualization is not a universal cloud feature; it is a per-family capability with an explicit opt-in on some providers and no opt-in at all on others.

• Google Compute Engine exposes it as a per-instance-template flag rather than something you can turn on later, and it is restricted by machine family and CPU platform — the low-cost shared-core families and the Arm families are out. Our own MIG module sets advanced_machine_features { enable_nested_virtualization = true } on the agent template, and that single line is the difference between a fleet and a pile of broken hosts.
• EC2 does not expose nested virtualization on ordinary virtualized instances. The AWS answer to "I want KVM" is a .metal instance type, which is genuinely bare metal that happens to bill hourly. Our AWS Terraform variable documents this in the variable description itself, because it is the kind of constraint you want to hit while reading a file rather than while debugging a boot.
• Azure exposes nested virtualization on a subset of sizes, generally the v3-and-later general purpose and compute families. Sizes come and go from that list.
• Hosting providers that sell dedicated servers are, definitionally, bare metal — the extension may still be disabled in firmware on a fresh box, which is a support ticket rather than a design problem.
• Anything Arm-based deserves its own investigation. The virtualization story is different, the availability is different, and Firecracker's support matrix is narrower than on x86.

# GCP: nesting is an explicit property of the instance template.
# Miss it and every host in the MIG comes up without /dev/kvm.
resource "google_compute_instance_template" "agent" {
  machine_type     = var.machine_type      # NOT a shared-core family
  min_cpu_platform = var.min_cpu_platform

  advanced_machine_features {
    enable_nested_virtualization = true
  }
}

# AWS: there is no equivalent flag, because there is no equivalent feature.
# The instance type IS the decision.
variable "instance_type" {
  description = <<EOT
EC2 instance type. Must be a *.metal flavor for Firecracker (needs KVM).
EOT
  type        = string
  default     = "c5n.metal"
}

# Worth saying out loud: a .metal instance is bare metal with cloud
# economics. You get native KVM, real local disks and no neighbours above
# you, while keeping the API, the images, the object storage and the
# ability to hand the machine back this afternoon. You also pay the
# highest per-core-hour rate of any option on this page.

Treat every list above as a snapshot of a moving target. Providers add families, drop families, and change the rules for CPU platforms. The script in the previous section is authoritative for the machine in front of you; the documentation is a hint.

Disk is where a snapshot fleet feels the difference

Nested virtualization gets all the attention in this debate and storage quietly does more damage. A dense snapshot-restore fleet has two hard storage requirements, and cloud VMs make both of them awkward.

The first is copy-on-write cloning of the rootfs. When a sandbox is created we reflink the template's disk instead of copying it, which turns a multi-gigabyte copy into a metadata operation of a few milliseconds. Reflink is a filesystem feature — XFS with reflink enabled, or btrfs — and both files have to live on the same filesystem. A default cloud boot disk is ext4, which has no reflink, so on our GCE hosts cloud-init creates a large XFS image file and loop-mounts it to get the property back. It works. It is also a filesystem inside a file inside a network-attached block device, which is three layers where a bare-metal host would have one.

# What we do on a cloud VM whose boot disk is ext4 -- build the filesystem
# we need inside a file, because we cannot reformat the disk we were given.
truncate -s 300G /opt/pandastack.img
mkfs.xfs -m reflink=1 -m crc=1 -q /opt/pandastack.img
mount -o loop /opt/pandastack.img /var/lib/pandastack

# What the same thing looks like when the disk is actually yours.
mkfs.xfs -m reflink=1 /dev/nvme1n1
mount /dev/nvme1n1 /var/lib/pandastack

# Either way, verify the property rather than trusting the mkfs flags.
# FICLONE is the ioctl behind cp --reflink; if this is slow, it copied.
cd /var/lib/pandastack
truncate -s 2G probe.img
time cp --reflink=always probe.img clone.img   # want milliseconds, not seconds
du -sh --apparent-size probe.img clone.img     # 2G each
du -sh probe.img clone.img                     # shared blocks: far less
rm -f probe.img clone.img

The second requirement is read latency under fan-out. Restoring a snapshot means touching a memory file; baking and syncing templates means moving tens of gigabytes of seed artifacts around; forking a sandbox means reflinking a disk and restoring memory again. Local NVMe answers those in microseconds. Network-attached block storage answers them across a wire, with a latency floor and an IOPS budget you share, and it degrades in exactly the way that hurts most: fine when one restore is in flight, ugly when forty are.

We attacked that from the software side rather than the hardware side, because we were already on cloud VMs. Memory now streams on demand from object storage over HTTP range requests, with a zero-chunk map so empty pages are never fetched, a prefetch trace replayed in the background, and a shared on-disk chunk cache so the first restore on a host pays the network cost once for everybody. What we could not stream is the rootfs: copy-on-write needs a local block device, so disk artifacts stay local no matter how clever the memory path gets. That constraint is the honest reason local NVMe is not a nice-to-have for this architecture.

The other half of the ledger

Everything above argues one direction, so here is the return leg. Bare metal takes back things the cloud was quietly doing for you, and the bill arrives in operations rather than in dollars.

• Provisioning is minutes to hours, not seconds. This is the big one. You cannot absorb a traffic spike by scaling out, because the new capacity arrives after the spike ended. Elasticity stops being an API call and becomes a capacity plan.
• The machine is a pet whether you like it or not. Many providers cannot snapshot it, cannot resize it in place, offer no managed backups, and bill at full rate while it is powered off. If you want cattle discipline you have to impose it yourself: everything reproducible from git plus object storage, no state on the box that is not deliberately durable.
• You own the failure domain. A dead NVMe is your problem, an unstable DIMM is your problem, and the recovery time is a support ticket plus a hands-on window rather than a terminate-and-replace. Fleet-wide, this is fine; per-machine, it is a pager.
• Capacity becomes a monthly commitment. An elastic opex line turns into a fixed one, which flatters your unit economics when utilisation is high and punishes them when it is not. An idle owned box costs exactly what a busy one costs.
• You rebuild the free tier of the cloud. Images, private networking between hosts, object storage, metrics, secrets, DNS, a way to reinstall a machine unattended. None of that is hard. All of it is time, and it is time spent on things that were previously line items.

The last point deserves a paragraph rather than a bullet, because it is the one people underestimate. Our agent depends on managed object storage for template seeds and for user snapshots, and that dependency does not evaporate on bare metal — it becomes either a bucket you keep renting from a cloud you no longer run compute in, or an object store you now operate. Neither is free. The first gives you cross-provider egress; the second gives you a durability problem you now personally own.

It is a load-shape decision, not a religion

The question that actually settles this is not which option is cheaper per core-hour. It is: what fraction of your load is present at four in the morning on a Sunday?

Steady baseline belongs on hardware you own. The delta above the baseline belongs on hardware you can hand back within the hour. Everything else in this argument is detail.

That floor — the part of your fleet that is busy regardless of the hour — is the strongest case bare metal has. It is fully utilised, so the fixed cost is amortised properly. It is long-lived, so the provisioning latency does not matter. And it is the workload that benefits most from native KVM and local NVMe, because it is running constantly. Meanwhile the spiky part, the geographically-spread part, and the part you cannot forecast want an API that returns capacity in seconds, which is exactly what the cloud sells.

One wrinkle specific to platforms like ours, and it cuts against the bare-metal case: an architecture with genuinely no warm pool has a spikier load shape than one with idle VMs sitting around. When every create is a snapshot restore and hibernated workloads are deleted and rebuilt from object storage, there is no comfortable pool of already-running machines smoothing your demand curve. Your capacity requirement tracks concurrency almost directly. That makes the burst fraction larger, and the burst fraction is the part that wants cloud.

What a hybrid fleet actually demands

Most serious platforms land on both: a bare-metal floor with cloud burst above it. The idea is easy and I would guess most people underestimate the two pieces of engineering it requires.

A scheduler that can reason about unlike hosts

Our placement function today is a pure load-spreading score, and it is deliberately simple because our hosts are deliberately identical. It prefers free CPU, uses free memory as a secondary term, adds a small bonus to any agent that can stream a restore instead of downloading the whole memory file first, and drops any agent whose heartbeat has gone stale.

// api/internal/scheduler/scheduler.go -- the non-volume branch of Pick().
// Every create takes the same NATID + snapshot-restore fast path on any
// agent that has the template's seed, so placement is purely a
// load-spreading decision.
score = float64(freeCPU)*0.6 + float64(freeMem)/1024.0*0.3

// Tiebreaker: prefer an agent that can UFFD-stream the restore over one
// that must download the whole vm.mem first.
if a.Capacity.StreamRestoreEnabled {
    score += streamBoost      // 5.0
}

// Note what is missing: any notion of host class. Two agents advertising
// the same free CPU are interchangeable to this function, even if one of
// them is a nested guest on network-attached storage and the other has
// native KVM and local NVMe. In a homogeneous fleet that is correct and
// keeps the code honest. In a hybrid fleet it is a bug waiting for a
// customer to find it.

The bones for heterogeneity are already there, which I only noticed while writing this. Pick already filters by region, and it already filters by pool — we split hosts into stateful and ephemeral pools because a host holding customer volumes must never be a scale-in candidate, and a volume placement forces the stateful pool regardless of what the caller asked for. A bare-metal floor would arrive as exactly that kind of label. What is missing is the scoring term: something that says this class of host restores faster, prefer it until it is full, then spill. Filters exist; preference does not.

An artifact story that is identical in both places

The other half is harder to retrofit. A host is only interchangeable with another host if it can obtain the same artifacts the same way. For us that means template seeds — a baked snapshot's disk, its VM state and its memory file — published per generation to object storage behind a CURRENT pointer and a SHA256 manifest, so an agent can synchronise itself at boot, or lazily when a template it has never seen is requested.

In principle that design is provider-neutral: it is objects, ranges and hashes over HTTP. In practice our client code is written against one provider's API, so a genuinely hybrid fleet means porting the storage layer to an S3-compatible interface rather than flipping a config value. If you are building this from scratch and think you might ever own hardware, write that layer against S3 semantics on day one. It costs nothing at the start and it is unpleasant later, and I am telling you this as someone who did not.

The control plane itself is the easy part. Agents register in Postgres, heartbeat every ten seconds, hold a lease, and advertise their capacity; an agent in a rented rack is just another row provided the network path between it and the control plane is private, low-latency and not accidentally routed across the public internet. The database does not care where the machine lives. The scheduler and the artifact store do.

The decision procedure I actually use

1. Check /dev/kvm on a real instance of the family you are considering, not in the documentation. If it is missing, the family is out and nothing else matters.
2. Plot 30 days of concurrency and find the floor — the level your fleet never drops below. That number, plus a margin, is the bare-metal candidate. Everything above it is burst.
3. Ask whether your fast paths need local disk. If you do copy-on-write cloning or restore memory from files, they do, and a family without local NVMe is a compromise you will keep paying for.
4. Cost the floor honestly: owned hardware amortised over a month against equivalent always-on cloud instances, and add the operational work you will now be doing yourself. If the two are within twenty percent, stay on cloud — the flexibility is worth more than the margin.
5. Before buying anything, check that your scheduler can express host preference and that your artifact store is not welded to one provider's SDK. If either is false, fix that first. A hybrid fleet without those two is just two fleets and a pager rotation.

The summary

Running Firecracker on a cloud VM means your guests are nested. That costs you on every exit and on every page fault, it restricts you to the instance families that expose the feature, and it puts a second scheduler between your vCPU threads and a physical core. It also gives you capacity in seconds, hardware failure as somebody else's problem, and an ecosystem of managed services you would otherwise build.

Bare metal removes the nesting tax, hands you real local NVMe — which a dense snapshot-restore fleet effectively requires — and prices steady load far better than an always-on cloud instance does. It also removes elasticity, hands you the pager, converts a variable cost into a fixed one, and quietly adds a backlog of infrastructure work that used to be a line item.

So the answer is a load shape rather than a preference. Put the floor on hardware you own, put the spike on hardware you can return, and accept that the interesting engineering is not in either host — it is in the scheduler that can tell them apart and the artifact store that makes them interchangeable. We run entirely on cloud VMs today, and both of those pieces are exactly where our work would go first.