Firecracker vs Hyper-V Isolated Containers: Two Ecosystems, One Verdict

I build PandaStack, an open-source Firecracker microVM platform, so I spend most of my week on the Linux side of this question. But the most interesting thing about container isolation is not anything either ecosystem built — it is that two ecosystems sharing essentially no code arrived at the same verdict within a couple of years of each other, and almost nobody writes that up.

Microsoft shipped VM-backed containers because Windows containers on a shared kernel were not a boundary they were prepared to stand behind for arbitrary multi-tenant workloads. The Linux world shipped Firecracker, Kata Containers and gVisor for precisely the same reason, at roughly the same time, without coordinating. One side wrapped the container in a lightweight utility VM and kept the container API. The other side built a new hypervisor and threw away everything that was not strictly necessary. Same conclusion, opposite directions.

Before any of the comparison: Firecracker is Linux and KVM only. It does not run on a Windows host and it cannot boot a Windows guest. If the thing you need to isolate is a Windows binary — a .NET Framework service, an IIS site, a Windows-only build agent — then Firecracker is not on your shortlist at all, Hyper-V isolation is the answer, and the rest of this post is background reading. I would rather say that in the third paragraph than bury it on page four. What is left is worth reading anyway, because the two designs diverge sharply on the thing that decides whether a kernel per workload is affordable: how you get a fresh kernel without paying for a boot.

Two ecosystems, one verdict

A container is a polite suggestion to a kernel you are sharing with strangers. On Linux that suggestion is assembled from namespaces, cgroups, capabilities and a seccomp filter — all features of the very kernel they are supposed to be protecting. The attack surface is the whole syscall table, and the syscall table is large, actively researched and still growing. None of that is a criticism of containers. It is a statement about what containers are for: packaging and resource control, not adversarial isolation.

Windows had the structurally identical problem with a different vocabulary. Windows containers are built on silos — an extension of job objects plus virtualization of the object namespace, the registry and the filesystem view — and in process isolation mode the container's processes run on the host kernel. Same shape, same consequence: a kernel bug is a tenant boundary bug.

Microsoft's response was a second isolation mode in which each container gets its own kernel inside a lightweight VM, plus fairly direct guidance about which mode to use when you do not trust the workload. The Linux response was three separate projects that each move the boundary somewhere harder: Firecracker and Kata to a hypervisor, gVisor to a userspace kernel reimplementation. Two ecosystems, independently, reached for a kernel per tenant. That agreement is the headline; everything below is engineering detail.

What Windows containers actually do

Process isolation

Process isolation is the Windows analogue of a normal Linux container. The container's processes execute directly on the host kernel, with the namespace and registry virtualization providing the illusion of a private machine. It is the cheaper mode — no second kernel to boot, no second kernel's worth of memory — and on Windows Server it has historically been the usual default.

It also comes with a constraint that does not exist on Linux: because the container runs on the host's kernel, the host OS build and the container image's OS build have to be compatible. The rules have shifted across releases, so treat the specifics as something to look up rather than remember, but the practical effect is stable — you do not get to run an arbitrary Windows base image on an arbitrary Windows host and expect it to work.

Hyper-V isolation

In Hyper-V isolation, each container runs inside its own lightweight VM — Microsoft's term is a utility VM — with its own kernel. The container API you drive it with does not change much; what changes is that a hypervisor now sits between the container and the host kernel, and the container's syscalls land on a kernel that nothing else is using.

Two things make people choose it. The first is the security argument, which is the one everybody quotes. The second is more mundane and probably causes more adoption: because the container brings its own kernel, the host-build-versus-container-build compatibility constraint is relaxed. A fleet that wants a slightly older base image on a newer host has a mode that permits it. That is an operational reason, not a security one, and a lot of Hyper-V isolation in the wild is there because of build skew rather than threat modelling. On Windows client SKUs it has historically been required rather than optional — you did not get process isolation on a desktop — and that too has moved around across releases.

The utility VM is the same machinery as WSL2 and Windows Sandbox

The interesting part of the Windows design is how much of it is one mechanism reused. The API surface that creates and manages these VMs is the Host Compute Service, generally met through its Go bindings in the hcsshim project, which is what containerd on Windows drives underneath. The same utility-VM machinery shows up in places not branded as containers at all: Windows Sandbox is a disposable desktop built on it, and WSL2 runs a real Linux kernel in a lightweight VM on the same foundations.

Linux containers on Windows — the LCOW work, and the WSL2 backend Docker Desktop uses today — is that same trick pointed at a Linux kernel instead of a Windows one. Which is the quiet punchline of this comparison: when Windows needs to run Linux containers, it boots a Linux kernel in a VM, because that is the only way to get Linux syscalls. The boundary was never optional. It was just differently priced.

The tradeoff is exactly what you would expect and it is worth stating plainly rather than hedging: more isolation, more memory per container, slower start, and a second kernel per container that someone has to boot and patch. You have traded a packaging primitive for a virtualization primitive, and virtualization primitives cost RAM.

What Firecracker does differently

Firecracker started from the opposite end. Instead of taking a general-purpose hypervisor and making it light enough to put under a container, it is a purpose-built VMM written in Rust whose design premise is that you do not need most of a virtual machine. There is no BIOS and no legacy boot path — the VMM loads an uncompressed kernel directly. The device model is deliberately tiny: virtio block, virtio net, virtio vsock, a serial console, a keyboard controller that exists mainly so the guest can be told to reboot. No emulated graphics adapter, no sound card, no USB controller, no CD-ROM.

Around that sits a defence-in-depth story aimed at the host: a jailer that chroots and drops the VMM into its own namespaces and cgroup before handing over control, and a seccomp filter reducing the host-side VMM process to a short list of permitted syscalls. The threat model is explicit — you are protecting the host from a guest already assumed compromised.

On PandaStack that is Firecracker v1.16.0, a 5.10 guest kernel, and an Ubuntu 24.04 userland inside the guest. Host needs KVM. One thing worth noticing: because the guest brings its own kernel, there is no equivalent of the Windows build-compatibility constraint at all. The guest kernel version is a property of the template, not of the host distribution, and I can change one without touching the other. If you have fought Windows container base-image compatibility, that decoupling is the thing you would actually enjoy.

Snapshot restore is the part that actually matters

A tiny device model makes boots fast. Snapshot restore makes boots unnecessary, and that is the difference that changes the economics. Firecracker can serialize a running VM's memory and device state and restore it later, which means a fresh kernel per workload stops being something you boot and becomes something you un-pause.

PandaStack restores a baked template snapshot on every single create: p50 179 ms, p99 203 ms, with no warm pool of idle VMs anywhere in the system. The only slow path is the very first spawn of a template that has no snapshot yet, which cold-boots in roughly 3 seconds and bakes the snapshot on the way through, after which every subsequent create takes the restore path. Memory comes back as a private mapping so the kernel does copy-on-write on fault, or — when streaming is turned on — through a userfaultfd handler that pages 4 MiB chunks out of object storage on demand. The rootfs is a block device with copy-on-write underneath it (XFS reflink or dm-snapshot) and is always a local file, because CoW needs one. Streaming covers memory only. There is a longer write-up of that path at The Snapshot-Restore Boot Path: Every Sandbox in Under 200ms.

Since I am being honest about limits: restore has sharp edges and I have been cut by them. A restored guest resumes with whatever clock it had at bake time, which for a while meant guests failing TLS handshakes with certificate-not-yet-valid errors — a genuine incident, now fixed by force-syncing the clock on restore, resume and wake. Snapshots also freeze vCPU count and RAM, because Firecracker cannot change either at restore, so per-create `cpu` and `memory_mb` are overridden to whatever the template baked. RAM is the only template knob: `base` and `browser` 4 GiB, `code-interpreter` and `agent` 2 GiB, `postgres-16` 1 GiB, all at 8 vCPU.

# ---------------------------------------------------------------------------
# The Windows side. Two ways to ask for a kernel per container.
# Verify every flag and path against Microsoft's current docs -- this area has
# been renamed and re-defaulted more than once.
# ---------------------------------------------------------------------------

# 1. The one-liner. --isolation picks the MODE, not the image.
#    "process"  -> container processes run on the HOST kernel (Linux-container
#                  shaped). Cheapest. Requires host build and container-image
#                  build to be compatible -- this is the constraint that bites.
#    "hyperv"   -> container runs inside its own lightweight utility VM with
#                  its own kernel. More RAM, slower start, boundary is the
#                  hypervisor instead of the shared kernel.
#    "default"  -> whatever the daemon's default is for this SKU. On Windows
#                  Server that has historically been process; on client SKUs
#                  hyperv has historically been required. Do not rely on
#                  "default" meaning the same thing on two machines.
docker run --rm --isolation=hyperv mcr.microsoft.com/windows/servercore:ltsc2022 \
  powershell -Command "Get-ComputerInfo | Select-Object OsBuildNumber"

# The same image under process isolation is the compatibility test: if the host
# build and the image build are not compatible, THIS is the one that fails and
# the hyperv run above is the one that works. That is why a lot of production
# Hyper-V isolation exists -- not threat modelling, just build skew.
docker run --rm --isolation=process mcr.microsoft.com/windows/servercore:ltsc2022 \
  powershell -Command "Get-ComputerInfo | Select-Object OsBuildNumber"


# 2. The containerd shape. Isolation is selected by RUNTIME HANDLER rather
#    than a CLI flag, so the decision moves into cluster config -- a RuntimeClass
#    in Kubernetes terms. Sketch of config.toml; check the current schema.
cat <<'TOML'
[plugins."io.containerd.grpc.v1.cri".containerd]
  # Which handler a pod gets when it does not ask for one by name.
  default_runtime_name = "runhcs-wcow-process"

  # Process isolation: no utility VM. The shared-kernel option.
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runhcs-wcow-process]
    runtime_type = "io.containerd.runhcs.v1"

  # Hyper-V isolation: one utility VM per container, driven through HCS via
  # hcsshim. The sandbox_isolation knob is what flips it; the options table is
  # also where utility-VM sizing (CPU count, memory) lives -- and utility-VM
  # memory is the per-container overhead you are buying.
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runhcs-wcow-hypervisor]
    runtime_type = "io.containerd.runhcs.v1"
    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runhcs-wcow-hypervisor.options]
      sandbox_isolation = 1
TOML

# Note what did NOT change in either case: the image, the registry, the build,
# the orchestrator, the people. That low adoption friction is the single
# strongest argument for Hyper-V isolation and I am not going to pretend
# otherwise -- standing up a Firecracker fleet is a much larger project.

# ---------------------------------------------------------------------------
# The Firecracker side. This is the entire machine. Look how little there is.
# ---------------------------------------------------------------------------

cat > /tmp/vm.json <<'JSON'
{
  "boot-source": {
    "kernel_image_path": "/var/lib/pandastack/kernels/vmlinux-5.10",
    "boot_args": "console=ttyS0 reboot=k panic=1 pci=off nomodules random.trust_cpu=on init=/sbin/init"
  },
  "drives": [
    {
      "drive_id": "rootfs",
      "path_on_host": "/var/lib/pandastack/vms/abc123/clone.ext4",
      "is_root_device": true,
      "is_read_only": false
    }
  ],
  "network-interfaces": [
    {
      "iface_id": "eth0",
      "host_dev_name": "tap0",
      "guest_mac": "06:00:ac:10:00:02"
    }
  ],
  "machine-config": {
    "vcpu_count": 8,
    "mem_size_mib": 4096,
    "smt": false
  }
}
JSON

# What each piece is doing, and what is conspicuously absent:
#
# kernel_image_path  An UNCOMPRESSED kernel loaded directly by the VMM. There
#                    is no BIOS, no UEFI, no bootloader, no option ROM. The
#                    whole legacy boot path that a general-purpose hypervisor
#                    carries simply does not exist here. 5.10 on PandaStack.
#
# boot_args          pci=off is the tell: there is no PCI bus to enumerate,
#                    because every device is virtio-over-MMIO. nomodules means
#                    the guest will not go hunting for hardware that is not
#                    there. reboot=k + panic=1 make the guest DIE rather than
#                    sit at a prompt -- the correct behaviour for something
#                    disposable.
#
# drives             One ext4 block device. On PandaStack clone.ext4 is a
#                    copy-on-write clone (XFS reflink or dm-snapshot) of the
#                    template rootfs, which is why creating one is O(metadata)
#                    instead of O(gigabytes). It must be a LOCAL file -- CoW
#                    needs a local block device, so this is the one artifact
#                    that cannot be streamed from object storage.
#
# network-interfaces One tap device, living in a per-sandbox Linux network
#                    namespace with its own veth pair. Pre-allocated, so the
#                    create path only patches a MAC instead of building
#                    namespaces from scratch.
#
# machine-config     vcpu_count and mem_size_mib. These are FROZEN into any
#                    snapshot taken of this VM -- Firecracker cannot change
#                    either at restore time, so a per-create memory request
#                    against a snapshot-backed template is overridden to the
#                    baked value. RAM is a template property, not a call
#                    parameter.

# Boot it with no API server at all -- a config file and one process.
firecracker --no-api --config-file /tmp/vm.json

# There is no GPU line, no PCI passthrough line, no nested-virtualization line,
# and no sound/USB/CD-ROM. That is not an oversight, it is the product.

The comparison, graded on the axes that decide it

Latency shape and density are the same conversation

The two rows people stare at are start latency and memory, and they are both consequences of one choice: whether a kernel is something you boot or something you restore.

If a kernel is something you boot, your cost model is per-boot and your optimisation is to make the boot faster — which Microsoft has clearly spent real engineering on for utility VMs. There is a floor to that, because a boot is a kernel doing actual work, and the floor is why the two isolation modes still feel different in a tight loop. If a kernel is something you restore, the per-instance cost is reading back a memory image, which is a different curve entirely: dominated by page faults rather than kernel initialisation, and page faults are exactly what copy-on-write and demand paging are good at.

That is why PandaStack has no warm pool. A warm pool is a workaround for an expensive create, and when the create is a 179 ms restore the pool is just idle RAM with a scheduling problem attached. The flip side is that the restore model only works if you were willing to bake a template in advance, which is a constraint that the flag-on-your-existing-runtime approach does not impose on you.

On density, both worlds hit the same wall and it is RAM. A second kernel per tenant costs memory in Hyper-V isolation and it costs memory in Firecracker, and no amount of clever VMM design makes a guest's resident set go away. PandaStack's hard ceiling is 16,384 pre-allocated /30 subnets per agent host, a number you will never reach, because memory and CPU bind long before the subnet space does. The levers that actually move density are copy-on-write page sharing on restore and an idle clock aggressive enough to delete things — not the hypervisor. MicroVM Density: The Economics of Per-Tenant Isolation is the long version.

Which should you actually pick

This one resolves more cleanly than most head-to-heads, because the guest OS is a hard gate rather than a preference.

• Your workload is Windows. Firecracker is not an option — not a bad option, not an option. Hyper-V isolation is the answer, and the secondary benefit you will probably value more day to day is the relaxed host-to-container build compatibility.
• Your workload is Linux, you need a real multi-tenant boundary, high density and sub-second starts, and the creator of the sandbox is a program rather than a person. Firecracker-class microVMs. That is the case the snapshot-restore model was built for, and it is what I build.
• Your workload is Linux but you want to keep your existing container runtime and orchestrator. Kata Containers runs your OCI containers inside a VM with a shim that speaks CRI, which is the closest Linux analogue to Hyper-V isolation as an adoption story. gVisor takes the other road entirely and intercepts syscalls in a userspace kernel, trading some compatibility for no hypervisor requirement. Firecracker vs Kata vs gVisor: three isolation models grades all three properly.
• You need both Windows and Linux isolated workloads. You need two fleets, and you should plan and budget for two fleets rather than hunting for the substrate that covers both. It does not exist. A Linux host can run a Linux kernel in a VM, a Windows host can run a Windows or a Linux kernel in a VM, and neither can run the other's kernel on bare metal. Pretending one platform covers both is how you end up with a Windows build agent nobody wants to own.

And the honest non-recommendation: if your containers all run code your own team wrote, your tenants are not adversarial, and your compliance story does not require a hypervisor boundary, process isolation on Windows or a plain container on Linux is the right call and the VM is a tax. The verdict both ecosystems reached was about hostile or unvetted code. It was never "VMs for everything."

What the managed version looks like

The reason Hyper-V isolation adopts so easily is that somebody else already runs the hypervisor for you — it ships in the OS. The equivalent on the Firecracker side is a platform. Here is the same shape as that config file, with the fleet, the jailer, the netns pool and the snapshot store already somebody's job.

# The managed version of "give this workload its own kernel."
# Linux guests only -- there is no Windows path here and there will not be one.

from pandastack import Sandbox

# create() is a SNAPSHOT RESTORE, not a boot. The template was baked once; this
# call reads back its memory image and resumes it. p50 179 ms, p99 203 ms, and
# there is no warm pool behind it -- every create takes this same path. The one
# exception is the very first spawn of a template with no snapshot yet, which
# cold-boots in ~3 s and bakes the snapshot on the way through.
#
# cpu and memory_mb are NOT passed on purpose: `base` bakes at 4 GiB / 8 vCPU,
# and Firecracker cannot change either at restore, so a request would be
# overridden to the baked values anyway. RAM is a template knob, not a call arg.
sbx = Sandbox.create(
    template="base",
    ttl_seconds=600,        # IDLE clock, not a wall clock. Anything that
                            # touches the guest resets it; read-only status,
                            # metrics and log GETs deliberately do not. When it
                            # expires the reaper DELETES the VM -- unpushed work
                            # dies with it unless you passed persistent=True,
                            # raised the TTL, or snapshotted first.
    metadata={"job": "untrusted-build", "tenant": "acme"},
)

try:
    # One kernel, one tenant, one job. The guest is Ubuntu 24.04 userland on a
    # 5.10 kernel that nothing else on this host is sharing -- which is the
    # whole point, and the same point Hyper-V isolation is making on Windows.
    print(sbx.exec("uname -r").stdout.strip())          # 5.10.x

    # Anything slow or hostile gets fenced IN-GUEST. One-shot exec() does not
    # actually enforce timeout_seconds, and exec_stream only widens the HTTP
    # timeout -- so timeout(1) inside the guest is the only real bound.
    rc = sbx.exec_stream(
        "cd /work && timeout --kill-after=10s 900 sh -c 'make -j8 test'",
        on_stdout=print,
        on_stderr=print,
    )
    print("exit", rc)

    # A kernel boundary is not a network policy. Egress to the internet is OPEN
    # by default. Sibling sandboxes cannot reach each other's subnets and the
    # cloud metadata range is dropped at the host, but fencing your own VPC,
    # databases and internal APIs is still your work. Same on Windows: a
    # utility VM does not firewall your intranet for you.

finally:
    # Unconditional teardown. kill() -- there is no sbx.delete(). Write this
    # even though the idle reaper would eventually do it: "eventually" is a
    # billing decision you are making on behalf of your finance team.
    sbx.kill()

The bottom line

Two ecosystems that share no code, no leadership and no release calendar both concluded that a shared kernel is not a boundary you can sell to a multi-tenant customer, and both shipped a kernel per tenant. Microsoft did it by wrapping the existing container in a utility VM so the container API survived, which is the lower-friction engineering and the reason Hyper-V isolation is a flag rather than a migration. Firecracker did it by building a new VMM with almost no device model and then making the boot unnecessary through snapshot restore, which is the higher-ceiling engineering and the reason a kernel per request is affordable at all.

Pick on guest OS first, because that gate is absolute. Then pick on who you want running the hypervisor. If you are on Windows, I have nothing to sell you and I would not pretend otherwise. If you are on Linux, want a kernel per workload, and would rather not build a Firecracker fleet yourself, that is what PandaStack is: one microVM per sandbox, snapshot-restored at p50 179 ms with no warm pool, 8 vCPU of burst and the template's baked RAM, at $0.054 per vCPU-hour and $0.0162 per GiB-hour on one rate card. No GPU, no Windows guests, no nested virtualization, egress open until you close it. If any of those four is a requirement, I am the wrong substrate and you should go read somebody else's comparison.