Firecracker vs Bottlerocket: One Is a Hypervisor, One Is a Host OS

"Firecracker vs Bottlerocket" shows up in search often enough that I want to answer it properly, and the honest first answer is that the comparison is a category error. These two things do not compete. They are not two ways of doing the same job, they are not two vendors' takes on the same abstraction, and choosing one does not mean forgoing the other. Firecracker is a Virtual Machine Monitor — a userspace process that uses KVM to run a guest with its own kernel. Bottlerocket is an operating system — a minimal, immutable Linux distribution from AWS whose entire purpose is to be the host that runs containers. One is a thing you run on a host; the other is the host.

I'm Ajay, and I built PandaStack on Firecracker, so I have a bias I'll declare up front. But the point of this post isn't to tell you Bottlerocket is bad — it isn't, it's a genuinely well-made piece of engineering with a clear thesis, and I'd happily run it. The point is that when people type this comparison into a search box, they are almost always trying to answer a different question underneath it: "what should I use to run workloads I don't fully trust?" And for that question these two options give structurally different answers, because a hardened host OS makes the machine you share harder to break, while a hypervisor removes the sharing. An immutable read-only rootfs is a wonderful thing to have right up until the attacker is already inside the kernel you're sharing with them, at which point they don't need to write to your rootfs to have ruined your week.

What Firecracker actually is

Firecracker is a VMM written in Rust, originally built at AWS to run Lambda functions, and now open source and used well beyond it. When you launch a Firecracker microVM you get a real virtual machine: a guest kernel, a guest init, a guest userspace, all confined by hardware virtualization through KVM. The guest's only route to the outside world is a deliberately tiny set of emulated devices — a virtio block device, a virtio network device, a serial console, a vsock, and a handful of others. There is no emulated PCI bus full of legacy hardware, no BIOS, no USB controller, no graphics adapter. The device model was designed by subtraction, because every device you don't emulate is a device that can't have a bug.

Three properties matter for the comparison at hand. First, the boundary: the guest talks to a guest kernel, and a guest-kernel privilege escalation gets the attacker root inside a VM that contains one workload — nothing else on the host has a reachable handle. Second, the jailer: Firecracker ships a companion binary that puts the VMM process itself into a chroot, its own namespaces, a cgroup, and a seccomp filter, so even the hypervisor process is treated as something that might be compromised. That's defense in depth applied to the thing that is already the defense. Third, snapshots: Firecracker can freeze a running machine to a memory file plus a state file and restore it later, which is what turns "a whole VM per workload" from an expensive idea into a cheap one.

Firecracker is configured over an HTTP API on a Unix socket — you PUT a boot source, a machine config, some drives and network interfaces, then PUT an instance action to start it. It's blunt and it's legible, which is a nice property in something that is load-bearing for security.

# Firecracker is driven by a small HTTP API over a Unix socket.
# Start the VMM (the jailer wraps this in prod; shown raw for clarity).
firecracker --api-sock /tmp/fc.sock &

API="--unix-socket /tmp/fc.sock"

# 1. Machine config: vCPUs and RAM. Note these are frozen into any
#    snapshot you later take of this machine.
curl $API -X PUT 'http://localhost/machine-config' \
  -H 'Content-Type: application/json' \
  -d '{ "vcpu_count": 2, "mem_size_mib": 4096 }'

# 2. Boot source: an actual guest kernel. This is the line that makes
#    it a VM and not a container — the workload gets its own kernel.
curl $API -X PUT 'http://localhost/boot-source' \
  -H 'Content-Type: application/json' \
  -d '{
        "kernel_image_path": "/var/lib/pandastack/vmlinux-5.10",
        "boot_args": "console=ttyS0 reboot=k panic=1 pci=off"
      }'

# 3. Root filesystem as a virtio block device.
curl $API -X PUT 'http://localhost/drives/rootfs' \
  -H 'Content-Type: application/json' \
  -d '{
        "drive_id": "rootfs",
        "path_on_host": "/var/lib/pandastack/rootfs.ext4",
        "is_root_device": true,
        "is_read_only": false
      }'

# 4. Go.
curl $API -X PUT 'http://localhost/actions' \
  -H 'Content-Type: application/json' \
  -d '{ "action_type": "InstanceStart" }'

Notice what isn't in that config: any notion of an image registry, a scheduler, a package manager, a supervisor, or an update mechanism. Firecracker does one job. Everything around it — how you build the rootfs, where the kernel comes from, what schedules the machines, how the host gets patched — is somebody else's problem. Frequently that somebody is you.

What Bottlerocket actually is

Bottlerocket is AWS's answer to a completely different question: if a machine's only job is to run containers, what is the smallest, least-mutable, least-drift-prone operating system that can do that? The answer they landed on is a Linux distribution that is deliberately hostile to the traditional sysadmin workflow, and that hostility is the feature.

The design points, described qualitatively — and please verify the current specifics against AWS's own documentation rather than my summary, because this is exactly the kind of thing that evolves:

• The root filesystem is read-only and integrity-verified at boot via dm-verity, so tampering with the OS image doesn't silently succeed — it fails the verification.
• There is no package manager and no shell in the normal path. You cannot yum install a debugging tool onto a production node, which means nobody can, which means production nodes don't drift into snowflakes.
• Configuration happens through an API rather than by editing files over SSH. You set typed settings and the OS renders the config; you don't hand-write a unit file at 2am and forget you did.
• Updates are image-based and applied to an alternate partition set, with an A/B style switch and rollback, rather than in-place package upgrades that can leave a half-updated machine.
• Host-level access is deliberately awkward and mediated through a separate, privileged container rather than being ambient on the node.
• Orchestrator containers and host/admin containers are kept in distinct containerd instances, so the workload plane and the management plane aren't sharing one runtime's fate.

Read that list as a whole and the thesis is clear: eliminate configuration drift, shrink the installed surface to almost nothing, make the OS a versioned artifact rather than a pet, and make updates atomic and reversible. Those are excellent goals. If you run a large container fleet, a general-purpose distro on your nodes is a liability — it's full of packages nobody uses, it accumulates hand-edits, and patching it is a rolling, partially-observable process. Bottlerocket makes the host boring, and boring hosts are the ones that don't page you.

# Bottlerocket's config model: typed settings through apiclient,
# not files edited over SSH. Shapes are illustrative — check the
# current setting names in the Bottlerocket docs before you ship this.
apiclient set \
  kubernetes.cluster-name="prod-use1" \
  kubernetes.node-labels."workload-class"="untrusted" \
  kernel.lockdown="integrity"

# Kernel sysctls are settings too, not a file you sed into place.
apiclient set settings.kernel.sysctl."user.max_user_namespaces"="0"

# Read back what the machine believes about itself.
apiclient get settings.kubernetes

# Updates are image-based and applied to an alternate partition set:
# stage the new image, mark it active, reboot into it. If it's bad,
# the previous image is still sitting on the other partition.
apiclient update check
apiclient update apply --reboot

That last block is the part I actually admire. "The previous OS image is still sitting on the other partition" is the same structural idea as keeping the old machine around during a blue-green deploy — rollback is a pointer flip rather than a re-installation. It's the right shape for host updates, and it's a meaningfully better story than apt-get upgrade and a prayer.

The question people are actually asking

Almost nobody types "Firecracker vs Bottlerocket" because they're genuinely torn between installing an OS and linking a hypervisor library. They type it because they're running a multi-tenant platform, or a CI system that executes arbitrary customer repos, or an AI agent that runs model-written code, and they've been told that containers alone aren't enough, and they're surveying the options. So let me answer the real question directly.

Bottlerocket hardens the host. Firecracker changes the boundary. Those are different kinds of improvement and they are not substitutes for each other.

Here's why that matters concretely. On a Bottlerocket node running a container runtime, every container on that node still shares one Linux kernel. That is the design — it's a container host OS, and containers share a kernel; that's what makes them containers. All the immutability, the verified boot, the missing package manager, the API-driven config: those reduce the chance that the host is misconfigured, that a compromised workload finds a writable binary to persist in, or that an operator's hand-edit leaves a hole. They do not change the fact that a Linux local privilege escalation exploited from inside container A is executing in the same kernel that is isolating container B. Once an attacker is in ring 0 on the shared kernel, an immutable rootfs is a speed bump on the persistence path, not a wall on the lateral-movement path. They don't need to survive a reboot; they need the ten minutes before one.

This is not a knock on Bottlerocket, because it was never trying to solve that. Its README does not promise you cross-tenant isolation for hostile code; it promises you a minimal, updatable, drift-free container host. Judging it for not being a hypervisor is like judging a very good deadbolt for not being a moat. The failure mode I care about is teams reading "hardened, immutable, security-focused OS" and concluding that the multi-tenancy question is now handled. It isn't. The kernel is still shared, and the kernel is where the interesting bugs are.

Side by side

• Layer — Firecracker: a userspace VMM process that runs on a host OS. Bottlerocket: the host OS itself, on which runtimes and VMMs run.
• Unit it manages — Firecracker: one microVM, with its own guest kernel and init. Bottlerocket: one node, and the containerd instances on it.
• Isolation boundary it provides — Firecracker: hardware virtualization via KVM plus a minimal virtio device model. Bottlerocket: namespaces and cgroups on one shared host kernel, plus a much smaller host attack surface.
• What a kernel LPE gets an attacker — Firecracker: root inside one guest, still facing the hypervisor. Bottlerocket: root on the node's shared kernel, alongside every other container on it.
• Mutability story — Firecracker: each guest is disposable; you delete the VM and the state goes with it. Bottlerocket: the host is immutable and read-only, verified at boot, updated as an image.
• How you configure it — Firecracker: HTTP API on a Unix socket, per machine. Bottlerocket: typed settings via an API/apiclient, no SSH-and-edit workflow, no package manager.
• Update model — Firecracker: you replace the binary and re-launch guests. Bottlerocket: atomic image update to an alternate partition set with rollback.
• What it explicitly does not do — Firecracker: scheduling, image building, host patching, fleet management. Bottlerocket: change the kernel-sharing model of the containers it runs.
• Best fit — Firecracker: untrusted, multi-tenant, or model-generated code that needs a real kernel boundary. Bottlerocket: running a large fleet of first-party container workloads on hosts you want boring and drift-free.

They compose — that's the actual answer

Once you see the layers, the productive question stops being "which one" and becomes "how do they stack." A Bottlerocket node runs a container runtime. A container runtime can be configured with more than one handler. And some of those handlers — Kata Containers, firecracker-containerd — don't run your container as a namespaced process on the host kernel at all. They boot a microVM and run your container inside it, with Firecracker as the VMM.

So the composed stack looks like this: an immutable, verified, API-configured host OS, running a container runtime, which dispatches trusted first-party workloads to the ordinary shared-kernel path and untrusted tenant workloads to a VM-backed runtime class where each pod gets its own guest kernel under Firecracker. The host is drift-free and the hostile code is behind a hypervisor. You get both properties because they were never in tension — one is about the machine you operate, the other is about the wall between your customers.

{
  "_comment": "Sketch: two runtime classes on one hardened host. Trusted work takes the cheap shared-kernel path; untrusted work gets its own guest kernel. Names and shapes vary by runtime version — check the docs.",
  "runtimes": {
    "runc": {
      "boundary": "namespaces + cgroups on the host kernel",
      "use_for": "first-party services you wrote and reviewed",
      "kernel": "shared with every other runc container on this node"
    },
    "kata-fc": {
      "boundary": "Firecracker microVM via KVM",
      "use_for": "tenant code, CI for arbitrary repos, agent-generated commands",
      "kernel": "its own guest kernel, per workload"
    }
  },
  "scheduling_rule": "workload-class=untrusted -> runtimeClassName: kata-fc"
}

Two caveats before you go build that. First, verify the details against the current docs for whichever runtime and host OS you pick — VM-backed runtime classes have real constraints around device access, host networking, storage drivers, and privileged pods, and those constraints move between versions. Second, running the VM-backed path well is genuinely more operational work than running runc: you now own kernel images, snapshot artifacts, and a memory-footprint model per workload rather than per node. That work is worth doing when the code is hostile and not worth doing when it isn't, which is exactly why the two-runtime-class split is the right shape rather than converting everything.

What a VM per workload actually costs now

The historical objection to "just give every tenant a VM" was cost: VMs meant multi-second boots and RAM reserved up front, so you amortized them across many tenants, which is how we got back to sharing a kernel in the first place. Snapshot-restore breaks that loop. Instead of cold-booting a kernel per workload, you boot once, freeze the machine, and restore that frozen machine on demand.

On PandaStack every sandbox, app, and managed database is its own Firecracker microVM created by restoring a baked snapshot: the restore step lands around 49ms, end-to-end create is p50 179ms and p99 around 203ms, and only the very first cold boot of a template is around 3 seconds. Forking a running machine — copy-on-write memory plus a reflink'd rootfs — is 400–750ms on the same host and 1.2–3.5s cross-host. Networking is pre-allocated rather than built on demand: 16,384 /30 subnets per agent, each with its own network namespace, so per-sandbox network setup isn't a multi-hundred-millisecond tax. The upshot is that a hardware-isolated machine per unit of untrusted work is now in the same latency class as starting a container, which removes the last honest reason to put strangers on a shared kernel.

from pandastack import Sandbox

# One microVM per unit of untrusted work. Not a namespaced process on
# a shared kernel — its own guest kernel, restored from a snapshot.
sbx = Sandbox.create(template="base", ttl_seconds=900)
try:
    # Whatever this does, it does inside a kernel nothing else uses.
    r = sbx.exec(
        "python3 /work/customer_supplied.py",
        timeout_seconds=120,
    )
    print(r.exit_code, r.stdout[-2000:])

    # A hostile guest can wreck its own kernel; that's the point.
    # There is no neighbour in here to wreck.
    probe = sbx.exec("uname -r && ls /sys/fs/cgroup | head -5",
                     timeout_seconds=10)
    print(probe.stdout)
finally:
    # Deleted whole. No residue to scrub off a shared host, because
    # nothing that ran ever had a handle to one.
    sbx.kill()

The `ttl_seconds` there is doing quiet security work as well as cost work: a sandbox that reaps itself is one that can't be forgotten and left running someone else's crypto miner for a fortnight. Ask me how I know.

A decision guide you can actually apply

Skip the vs framing and answer these in order.

1. Is the code you're running written and reviewed by your own team? If yes, a shared kernel is a risk you already accept everywhere else, and hardening the host is the highest-leverage next move. Bottlerocket-class host OS, ordinary container runtime, done.
2. Does any workload on the node come from a party you wouldn't give SSH to — a customer, a public repo, an LLM? If yes, you need a kernel boundary for that workload specifically. Host hardening is necessary and insufficient.
3. Are hostile and trusted workloads on the same nodes? Then you need per-workload runtime selection, not a per-fleet decision. Two runtime classes on the same hardened host is the composed answer.
4. Do you need to run this on managed Kubernetes with a supported node image? Then the host OS choice is largely made for you, and Firecracker enters via a VM-backed runtime class rather than replacing anything.
5. Do you need per-workload machines fast enough that latency isn't the deciding factor? Then snapshot-restore is the mechanism to look for, in whatever platform you evaluate — cold-booting a kernel per request is the thing that made this expensive, and it's the thing that got fixed.
6. Would you rather not operate any of this? That's the case for a platform. Full disclosure: that's what I built, and it's built on Firecracker rather than on containers precisely because of the argument in this post.

Hardening the host and changing the boundary are both good ideas. Only one of them helps when the exploit is in the kernel you're sharing, and it's not the one that makes your rootfs read-only.

So: Firecracker vs Bottlerocket has no winner, because there's no contest. Bottlerocket is a very good answer to "my container hosts keep drifting and patching them is a nightmare." Firecracker is the answer to "the code I'm running might be trying to hurt me." If you have the first problem, a minimal immutable host OS will make your fleet noticeably calmer. If you have the second, no amount of host immutability substitutes for a hypervisor, and the good news is that snapshot-restore has made the hypervisor cheap enough that you no longer have to choose which problem to solve.