Firecracker vs VMware ESXi: the Device Model Decides

A fair number of people arrive at this comparison sideways. A renewal quote lands on somebody's desk, a director asks whether there is an open-source option, an engineer remembers that AWS Lambda runs on something called Firecracker, and now there is a meeting. The honest answer to that meeting is usually not the one anybody wants, so let me put it at the top: ESXi and Firecracker are aimed at genuinely different jobs, and if your estate is the kind of thing ESXi is good at, Firecracker will not replace it. Not because it is worse. Because it is not that shape.

I'm Ajay; I build PandaStack, a platform where every sandbox, app and managed Postgres instance is its own Firecracker microVM. So I'm not neutral, and I'll handle that the way I'd want someone to handle it in the other direction: PandaStack's numbers are mine and I'll give them precisely, VMware's capabilities I'll describe qualitatively and tell you to check, and the joke throughout is about the mismatch rather than about VMware. ESXi is a serious piece of engineering that has kept an enormous amount of the world's software running for two decades. It is also completely the wrong tool for booting ten thousand hostile Linux guests, which is fine, because it never claimed to be that tool.

What ESXi actually is

ESXi is a type-1 hypervisor that installs on bare metal and runs virtual machines. That sentence undersells it by roughly an order of magnitude, because almost nobody runs ESXi as a lone hypervisor. What people mean by "we run VMware" is ESXi plus vCenter — a cluster-wide control plane with an inventory, a permissions model, a scheduler in DRS that rebalances load across hosts, HA that restarts guests from a failed host, vMotion that relocates a running VM between hosts without dropping it, Storage vMotion for the disks, integration with SAN and NAS storage and vSAN, backup and replication ecosystems, and a hardware compatibility list that tells you exactly which servers, NICs and HBAs are certified to carry all of it.

Underneath that platform sits the thing that actually distinguishes it from Firecracker: an enormous device model. ESXi presents guests with a full synthetic PC — firmware, emulated chipsets, PCI and PCIe topology, SCSI and NVMe and SATA controllers, USB, video, sound, serial and parallel ports, and paravirtual devices for the drivers that know better. That is why an ESXi host can run Windows Server, an ancient Linux with no virtio drivers, a BSD, a vendor's sealed appliance shipped as an OVA, and a domain controller nobody has logged into since the person who built it left. It runs them because it looks like a real computer to anything that boots.

That breadth is the product. It is what lets an enterprise virtualize an estate it did not design, one machine at a time, without asking any of the software to cooperate. You cannot get that property by being small.

What Firecracker actually is

Firecracker is a Rust binary that runs on a Linux host and uses KVM to run one guest. It is a VMM, not a hypervisor in the ESXi sense — the Linux kernel underneath it is doing the hypervisor's job, and Firecracker is the userspace process that describes a machine to it. Its entire management interface is a REST API served over a Unix domain socket. There is no cluster, no inventory, no UI, no scheduler, no storage layer, and no notion that a second host exists anywhere in the world.

The device model is the headline. Roughly: virtio-block, virtio-net, virtio-vsock, a serial console, and a balloon device, exposed over MMIO. Not PCI — MMIO, because there is no PCI bus at all. The kernel command line PandaStack boots guests with says so out loud, and that single flag is a decent summary of the entire philosophy.

# The kernel args a Firecracker guest boots with on PandaStack
# (agent/internal/firecracker/driver.go):
console=ttyS0 reboot=k panic=1 pci=off

# "pci=off" is not an optimisation. There is no PCI bus to enumerate,
# so the guest kernel is told not to waste 200ms looking for one.
# No PCI bus means: no PCIe passthrough, no emulated SCSI controller,
# no GPU, no NIC that isn't virtio, and no BIOS option ROM to attack.

Every VM also runs as its own process, under its own user, in its own namespaces, behind a seccomp filter that allows a short list of syscalls. The design intent is explicit: Firecracker was built for AWS Lambda and Fargate, where guests are numerous, short-lived, and running code supplied by strangers. In that world every emulated device is a liability you have to defend rather than a feature you get to advertise.

ESXi's device model is its value proposition. Firecracker's device model is its threat model. Same noun, opposite sign.

The device model argument, which is the whole comparison

Everything else in this post is downstream of one question: how much hardware should a hypervisor pretend to have? ESXi answers "as much as possible, because our guests are arbitrary operating systems we do not control." Firecracker answers "as little as we can get away with, because our guests are hostile and we control the image." Both answers are correct given their premises, and each one buys exactly what the other one gives up.

The security half of the argument is straightforward and applies to any hypervisor, VMware's included: device emulation is where VM escapes historically live. Not in the CPU virtualization, which is small and heavily scrutinised, but in the code that parses what a guest writes to a virtual floppy controller or a graphics adapter written for hardware that stopped shipping before some of your engineers were born. The bigger the device model, the more of that code exists. This is not a VMware-specific claim and I'm deliberately not citing anybody's advisories at anybody — every large hypervisor, VMware and QEMU alike, has shipped device-emulation fixes, because that is what large device models do. Firecracker's answer is to have almost none of that code, and to run what remains as an unprivileged, seccomp-confined process. A guest that finds a bug in virtio-block is standing in a jailed process, as its own uid, with a syscall allowlist between it and the kernel.

The performance half is the same argument wearing different clothes. A boot that never enumerates PCI, never runs firmware, never probes a SCSI bus and never initialises a video device is a boot with almost nothing in it. Combine that with restoring a memory snapshot instead of booting at all and you land somewhere a full VM cannot go. On PandaStack, the snapshot restore step itself is about 49ms; the end-to-end create — network slot, copy-on-write rootfs clone, VMM start, restore, and waiting for the guest to answer — is 179ms at p50 and around 203ms at p99. The first spawn of a template, before a snapshot exists, is a real cold boot at about 3s. Those are my numbers on my hardware; measure your own before promising anyone anything.

And the bill for all of that arrives immediately. Here is what the hardware inventory looks like from inside one of these guests.

# Inside a Firecracker microVM. Exact device naming varies by kernel
# version, but the shape is the point.

$ cat /proc/cmdline
console=ttyS0 reboot=k panic=1 pci=off

$ ls /sys/bus/virtio/devices
virtio0  virtio1  virtio2        # block, net, vsock. That's the catalogue.

$ ls /sys/bus/pci/devices 2>/dev/null || echo 'no PCI bus'
no PCI bus

$ ls /dev/vda /dev/ttyS0
/dev/vda  /dev/ttyS0

$ lspci
# nothing. There is no bus to list, which is the intended outcome and
# also the reason your CUDA plans need a different host.

So: no GPU passthrough, and not "not yet" — there is no PCIe machinery to pass anything through. No USB, so no licence dongles. No arbitrary guest OS: you boot a Linux kernel you supply, directly, with no BIOS or UEFI path, which rules out Windows, most sealed appliances, and anything that expects to be installed from an ISO. No virtual display, so a guest that wedges before the serial console comes up is a guest you delete rather than debug. If any line in that paragraph is a requirement, the comparison is over and you want ESXi, or plain KVM/QEMU, and you can stop reading.

vMotion, and the availability question underneath it

vMotion is the feature people name when they explain why they will never leave vSphere, and they are right to name it. Being able to evacuate a running VM off a host you need to patch — without the guest noticing, without dropping its TCP connections, without a maintenance window an application owner has to approve — is genuinely transformative for an estate of long-lived machines. Pair it with DRS moving guests around for load and HA restarting them after a host dies, and you have an availability story that requires nothing at all from the software running inside the guest. That last clause is the actual magic. It works on a 2011 appliance whose vendor is out of business.

Firecracker has nothing of the sort, and the reason is the same reason it boots fast: pre-copy migration of a running guest is a large, stateful, dirty-page-tracking feature, and Firecracker's design centre is snapshot and restore as a fast-create primitive instead. I've written about the mechanical difference separately; the short version is that migration preserves one specific running guest, and restore produces a new guest from a frozen one. Those are not the same operation and it is worth being precise about that rather than pretending a snapshot is a migration with extra steps.

But they are two answers to one question, which is: what happens when a host has to go away? The microVM answer is to make the guest cheap enough that relocating it is unnecessary. If a create is 179ms and a same-host fork of a running guest is 400-750ms — 1.2-3.5s if it has to cross hosts and pull artifacts from object storage — then "drain the host by deleting everything on it and letting the next request rebuild elsewhere" is a real strategy rather than a euphemism for an outage. Nobody has ever needed to live-migrate a code-execution sandbox that will be dead in ninety seconds.

from pandastack import Sandbox

# The microVM answer to "this host is going away": don't move the guest,
# reconstruct it. Snapshot is the durable artifact; restore is the create path.
src = Sandbox.create(template="base", ttl_seconds=900,
                     metadata={"role": "builder"})
src.exec("mkdir -p /work && echo 'state that matters' > /work/notes.txt")

snap = src.snapshot()          # returns a snapshot id; guest keeps running
print("snapshot:", snap)

# Somewhere else, later, possibly on another host in the fleet:
restored = Sandbox.create(template="base", from_snapshot=snap, ttl_seconds=900)
print(restored.exec("cat /work/notes.txt").stdout.strip())

# Cheap enough that fan-out is the normal case, not a recovery procedure:
children = src.fork_tree(count=8)   # copy-on-write memory + rootfs per child
for c in children:
    c.kill()
src.kill()
restored.kill()

Here is the part where I have to be fair to vMotion, because the counter-argument only works for a specific kind of workload. "Just rebuild it elsewhere" is a lie whenever the guest holds state that cannot be reconstructed from somewhere else. A customer's database is exactly that guest. On PandaStack, managed Postgres does not get the cheerful treatment above: it lives on a durable volume, it is pinned to a host, it archives to object storage continuously, and failing it over to another host means rebuilding it from that archive rather than waving a snapshot around. Creating one takes 30-90s, not 179ms, because it blocks until Postgres is genuinely ready. If most of your estate looks like that database rather than like a sandbox, vMotion is solving a problem you actually have and I would not try to talk you out of it.

The operational model: a GUI versus a socket

vSphere is enterprise software. It has an installer, a web client, an inventory tree you can click through, alarms, role-based access control, an API and CLI with an ecosystem around them, certified hardware, professional services, training, and a support contract that means when it breaks at 03:00 there is a phone number and it is somebody's job to answer. Automating it looks like this — with the loudest possible caveat that command names, flags and semantics change across versions and tooling, so verify every line against current documentation rather than against a blog post's memory.

# vSphere automation, in shape rather than in detail. govc is the community
# CLI over the vSphere API. VERIFY every flag against current documentation
# before scripting any of this — this is illustrative, not a recipe.

export GOVC_URL='https://vcenter.example.internal'
export GOVC_USERNAME='svc-automation@vsphere.local'

# Deploy from a template that already lives in an inventory you didn't build,
# on storage the platform already owns, on a network it already knows about.
govc vm.clone -vm 'templates/ubuntu-24.04' -host esx-04 \
  -ds datastore-nvme-01 -net 'VLAN-Prod' -c 4 -m 8192 app-07

govc vm.power -on app-07
govc snapshot.create -vm app-07 'pre-upgrade'

# And the line with no Firecracker equivalent: move a RUNNING guest to
# another host, with the guest none the wiser.
govc vm.migrate -host esx-07 app-07

Firecracker's equivalent surface is: start a process with a socket path, PUT four or five JSON documents at that socket, and POST an action to boot. That is the entire management plane. Afterwards there is no record the VM exists, no health check, no restart policy, and no way to find it again except the pid you are holding. Everything vCenter was doing in the transcript above — inventory, storage, networking, placement, snapshots, lifecycle — is now a thing you write.

I want to be concrete about the size of that, because "you build the control plane" is a sentence that sounds like a sprint and is not. Networking alone: PandaStack pre-allocates 16,384 /30 subnets per host, each with its own network namespace, veth pair and tap device, built in advance — because doing that work on the create path costs about a hundred milliseconds and would dominate a 179ms create. That is one bullet on a list that also includes host scheduling and capacity accounting, template and rootfs distribution with copy-on-write cloning, snapshot storage and demand-paging of guest memory from object storage, TTLs and idle reaping and orphan cleanup after a host reboot, and the authenticated multi-tenant API your users actually call. Depending on who you are, that is either the interesting part of your job or a nine-month project standing between you and the thing you meant to build.

Licensing and cost posture, described honestly

I am not going to put figures in this section, because any number I wrote would be wrong for your situation and possibly wrong in general by the time you read it. What I can describe is the shape of the two postures, which is what actually drives the decision.

VMware is commercially licensed software, and its packaging, editions, entitlements and pricing model have changed materially since the Broadcom acquisition — including how products are bundled and how entitlements are subscribed to rather than bought outright. Whatever you remember about VMware licensing from a few years ago should be treated as expired. Get a current quote, read the current terms, and check what your renewal actually entitles you to. What the licence buys, and this is worth stating plainly rather than sneering at, is a certified hardware path, an integrated feature set you did not build, an ecosystem of backup and monitoring vendors that already support you, staff you can hire who already know the product, and a support relationship. Those are real goods and organisations pay for them on purpose.

Firecracker's posture is the mirror image. It is Apache-2.0 licensed, there is no per-core anything, and the acquisition cost is zero. The cost moves rather than disappears: it becomes engineers, and it becomes the operational burden of a system where every incident is yours. There is no vendor to escalate to at 03:00; there is you, a Rust binary's log output, and a serial console. If the reason you are reading this post is a renewal quote, the honest framing is that you are not comparing a licence fee against zero — you are comparing a licence fee against a platform team, plus the risk that the platform team's system does not do the thing your existing estate needs, which as established at the top it very likely does not.

Side by side

• Guest OS support — ESXi: essentially anything that boots on a PC, including Windows, BSDs, legacy Linux with no virtio drivers, and sealed vendor appliances shipped as OVAs. Firecracker: Linux guests only, booted directly from a kernel you supply, with no BIOS or UEFI path and no installer ISO story.
• Device model and attack surface — ESXi: a broad synthetic PC — firmware, PCI/PCIe topology, storage controllers, USB, video, sound, serial — which is precisely what makes arbitrary guests work, and equally what makes device emulation the historic home of hypervisor escapes across every vendor. Firecracker: virtio-block, virtio-net, virtio-vsock, serial and balloon over MMIO, in a seccomp-confined unprivileged process per VM.
• Boot time — ESXi: a normal VM boot dominated by firmware plus whatever the guest OS does; measure it with your own images rather than trusting anyone's figure. Firecracker: milliseconds of VMM work plus a kernel boot, and in production usually a snapshot restore instead — about 49ms for the restore step and 179ms p50 for a full create on PandaStack.
• Live migration — ESXi: vMotion and Storage vMotion, plus DRS and HA built around them; the guest needs to do nothing and notices nothing. Firecracker: none, by design. The nearest move is snapshot, ship the artifacts, restore elsewhere — a new guest, not the same one.
• Snapshot model — ESXi: an operator convenience with delta disks and a memory option, taken before risky changes and consolidated afterwards. Firecracker: the primary create path — freeze a booted guest once, then restore that memory image every time, with copy-on-write memory and disk so a fork costs 400-750ms same-host.
• Control plane — ESXi: vCenter, which is an inventory, scheduler, permissions model, alarm system, and web UI, plus a mature API/CLI ecosystem. Firecracker: a REST API on a Unix socket that controls one VM; the cluster, the scheduler, the storage layer and the tenant API are things you write or buy.
• GPU and passthrough — ESXi: PCIe passthrough, vendor GPU virtualization options, and USB passthrough for dongles; check current hardware and licensing requirements. Firecracker: none of it, and not planned — with no PCI bus there is nothing to pass through.
• Licensing posture — ESXi: commercially licensed with editions and entitlements that have changed materially under Broadcom, buying certified hardware, an integrated feature set, an ecosystem and support; verify current terms and get a quote. Firecracker: Apache-2.0, zero acquisition cost, with the expense relocated into the engineering and on-call required to build everything vCenter already did.
• Density target — ESXi: tens of substantial VMs per host, sized like real servers, each expected to stay up. Firecracker: thousands of small guests per host, expected to be created per request and deleted without ceremony.

Choose ESXi if / choose Firecracker if

Choose ESXi — or vSphere generally — if your guests are heterogeneous, long-lived, and individually meaningful. If there is a Windows Server anywhere in the estate. If a vendor ships you an appliance as an OVA and "just rebuild it from a Dockerfile" is not a sentence that survives contact with their support contract. If you need to patch a host next Tuesday without asking eleven application owners for a window. If somebody needs a GPU or a USB dongle. If the operators are administrators rather than platform engineers, and a web console is how the work gets done. If losing one guest is an incident with a name. All of that is what the product is for, and building a replacement for it out of open-source parts is a way to spend three years arriving somewhere worse.

Choose Firecracker if your guests are homogeneous, short-lived Linux, created in response to a request, and hostile or at least machine-generated. An AI agent's shell. A code-interpreter cell running a model's best guess at what `rm -rf` does on a shared kernel — which is the entire argument for hardware virtualization here, since a container is ultimately a polite suggestion to a kernel you are also using. A CI job for an arbitrary pull request. A per-customer preview deploy. You care about creates per second, memory per guest and blast radius; you do not care about any individual guest's uptime and you never intend to log into one. Nobody is going to vMotion these. They will be gone before the migration finished.

And if you are in the second case but would rather not spend that nine months on scheduling, networking, snapshot storage and a tenant API, the third option is to consume it as a platform — which is the trade PandaStack makes: one Firecracker microVM per sandbox, created by snapshot restore, with copy-on-write forking and scale-to-zero attached, and the control plane already written. The isolation properties without the roadmap.

One last practical note, since it comes up in every one of these conversations: you can usually run Firecracker inside a VM on an ESXi host by enabling nested virtualization, or hardware-assisted virtualization exposure, for that guest. Verify the current requirements and caveats in VMware's documentation, because they depend on your version and CPU. It is an excellent way to prototype a microVM platform on hardware you already own and a terrible way to establish a production latency baseline — benchmark on bare metal before you promise anyone a number. Which is, in the end, the fairest summary of the whole comparison: these two things are far more often layered than opposed, and the versus in the title is mostly an artifact of how search works.