You Ship the Kernel: Firecracker Guest 5.10 vs 6.1

Nobody picks a kernel version for a container. Whoever provisioned the host made that call, and every container on the box inherits it without being consulted. A microVM inverts that. Firecracker boots an uncompressed kernel image you hand it on `/boot-source` — no bootloader, no GRUB menu, no distro package. The guest kernel is a file you built, sitting next to your rootfs: a build artifact of your platform, exactly like your API binary.

I'm Ajay; I build PandaStack, where every sandbox, managed Postgres instance, and hosted app is a Firecracker microVM booting a kernel we compiled. The question that keeps landing in my inbox from teams doing the same thing is the boring, load-bearing one: 5.10 or 6.1? Here's the long answer — why 5.10 is the well-trodden default, what a newer series concretely unlocks, what it costs, and the part everyone forgets until it's expensive: your snapshots are married to the kernel that took them.

You ship the kernel, which is wonderful and also your problem now

The Firecracker boot path is deliberately dull. You PUT a kernel path and a command line, PUT a root drive, PUT an InstanceStart action, and the VM is running. No firmware to negotiate with, no device enumeration to wait on, no initramfs unless you want one. That directness is much of why microVMs start in milliseconds: most of what a normal VM does at boot is discovering hardware whose exact shape you already know.

The consequence is that the kernel version is not a runtime setting. It's baked. Your fleet runs the kernel you compiled until you compile a new one and roll it out — which, if you use snapshots, means re-baking every template you have. On a good day that's the best property of the whole architecture: one config file, in version control, that you can read end to end. On a bad day it's a fleet-wide upgrade project with your name on it.

Why 5.10 is the well-trodden default

5.10 is the kernel the Firecracker project has been testing against for years. The reference configs exist, the boot path is thoroughly understood, the resulting image is small, and the config is known-good in the specific sense that matters: thousands of other people have booted approximately this exact kernel on approximately this exact virtual hardware, and the ways it breaks are already written down somewhere you can find with a search engine.

That's worth more than it sounds. When you boot a kernel a few hundred thousand times a day you aren't sampling the common path, you're sampling the tails — a one-in-a-million boot hang is a daily incident at that volume. Boring is a feature. A kernel whose interesting bugs were found in 2021 by someone else lets you spend your incident budget on your own code.

It's also small, which the snapshot path likes. On PandaStack a create restores a baked snapshot in roughly 49ms, for a p50 of about 179ms end to end and a p99 around 203ms; the first spawn of a template, which cold-boots and then captures the snapshot, takes about 3 seconds. A leaner kernel makes that cold path cheaper and the snapshot smaller, and snapshot size is what you pay to move a VM between hosts.

So the honest default is: if nothing inside the guest needs a feature that postdates 5.10, there is no prize for upgrading. You'd trade a known-good artifact for a novel one to gain features you aren't using.

What 6.1 and newer actually unlock

The case for moving is not "newer is better." It's specific capabilities that don't exist in 5.10 at any config setting, and one that technically does but usually isn't compiled in. Here are the ones that come up repeatedly.

virtio-mem: dynamic guest RAM is gated on the guest, not the VMM

A Firecracker guest's memory size is fixed at boot and, more painfully, at snapshot-bake time — the VM comes back with the RAM it was baked with, because a memory snapshot is a memory snapshot. Firecracker has grown memory hotplug support on the VMM side, but the guest half is a driver, and the virtio-mem guest driver landed around 5.16. A 5.10 guest cannot use it however new your Firecracker is. Verify the landing version against kernel.org and the current hotplug support status against Firecracker's documentation — both move.

This is the most consequential item on the list, because "guest RAM is baked in" cascades through everything else you design. It turns per-instance memory sizing into per-template memory sizing: a build that OOMs at 2 GiB means re-baking the template at 4 GiB, not passing a bigger number to an API. The alternatives are worse — virtio-balloon reclaims memory you already handed out but can't grow past the ceiling, and re-provisioning means a new VM. If growing a running guest is on your roadmap, the guest kernel is the whole blocker.

Netfilter modules, and why k3s falls over in a minimal microVM

Here's the failure that sends people to this post. You install k3s or Docker inside a microVM, everything looks healthy, and ClusterIP networking simply doesn't work. The logs are a wall of `iptables-restore` output that never once mentions the word kernel. The cause: kube-proxy writes every rule with `-m comment`, and your minimal guest config never enabled `CONFIG_NETFILTER_XT_MATCH_COMMENT`. The match doesn't exist, the restore fails, the rules never land, the service IP routes nowhere.

Be precise here, because it would be easy to sell this as a version story and it mostly isn't. `xt_comment` and its relatives — `xt_statistic`, `xt_recent`, conntrack, IPVS, ipset — are config options, not new features, and you can enable most of them in 5.10 today. The version angle is narrower: newer series carry nftables and netfilter work that doesn't exist in 5.10 at all, and the container ecosystem drifts toward what recent kernels provide. If you're rebuilding to close netfilter gaps anyway, that's the moment to ask whether you're rebuilding the right series.

io_uring, cgroup v2, drivers, and the support window

The rest of the case is cumulative rather than dramatic, which makes it easy to dismiss and expensive to keep dismissing:

• io_uring matured enormously after 5.10. If anything in your guest leans on it — modern database engines, some language runtimes, high-throughput proxies — a newer series gives you a far more complete implementation, with fewer of the sharp edges that were being sanded off in exactly that window.
• cgroup v2 works on 5.10, but userspace increasingly assumes it. Recent systemd, containerd, and runc are written against v2 semantics, and a current userspace on an older kernel is a supported configuration right up until the day it isn't.
• Newer virtio drivers and filesystem work — virtio-fs, erofs, overlayfs fixes that matter if your guest builds container images inside itself.
• A longer remaining support window. LTS series get security fixes for a defined period and then stop, and those timelines have been revised more than once. Check current end-of-life dates on kernel.org rather than any blog post, including this one.
• Newer syscall and feature surface generally — landlock, newer seccomp behavior, BPF capabilities — which matters if you do defense-in-depth inside the guest rather than relying only on the hypervisor boundary.

What the newer kernel costs you

A bigger image and a slightly longer boot. The vmlinux grows and the guest has more to initialize. On any single boot that's uninteresting; measure it on your own config anyway, because you pay it on every create forever, and it shows up again in snapshot size — which is what crosses the network on a cross-host fork.

A larger attack surface inside the guest. The hypervisor boundary protects your host and doesn't change here — this is guest-internal. But if your threat model includes a process in the microVM escalating against your own init, your agent, or another session in the same guest, more drivers and more syscall surface is more to defend.

Config churn, which is the one that actually bites. Options get renamed, split, merged, and removed between series. Copy a 5.10 config into a 6.1 tree, run `make olddefconfig`, and it will cheerfully answer every new symbol with its default — fast, convenient, and completely silent about the twenty things it just decided for you. Diff the config before and after, and read the diff. The whole value of owning your kernel is that someone read it.

5.10 LTS vs 6.1 LTS, dimension by dimension

Same job, two series. Verify anything version-specific against kernel.org and the Firecracker documentation before you commit — support windows and feature-landing versions shift, and this table will age.

• Firecracker testing pedigree — 5.10 LTS: the series the project has tested against longest, with reference configs and a boot path thousands of people have already debugged for you. 6.1 LTS: used in production by plenty of teams, but closer to the edge of the well-lit area, and more of the validation is yours.
• Boot time and image size — 5.10 LTS: smaller vmlinux, quicker to initialize, smaller snapshots to move between hosts. 6.1 LTS: bigger image and slightly more boot work — irrelevant once, measurable multiplied by every create your fleet does.
• virtio-mem memory hotplug — 5.10 LTS: unavailable at any config setting; the guest driver postdates the series, so guest RAM is fixed at boot and at bake time. 6.1 LTS: available, since the driver landed around 5.16 — the one gap here that is genuinely a version decision, not a config decision.
• In-guest netfilter completeness — 5.10 LTS: most matches exist but minimal configs routinely omit them, which is why a missing xt_comment kills k3s ClusterIP networking. 6.1 LTS: same config discipline, plus nftables and netfilter work that doesn't exist in 5.10 at all.
• io_uring maturity — 5.10 LTS: present but early; fine if nothing in the guest leans on it. 6.1 LTS: substantially more complete and better-behaved, which matters for engines that use it as their primary I/O path.
• Attack surface inside the guest — 5.10 LTS: smaller feature and driver surface, a genuine plus for a hardened single-purpose guest. 6.1 LTS: more surface, but newer in-guest hardening primitives if you build defense-in-depth above the hypervisor boundary.
• Support horizon — 5.10 LTS: an older LTS with less runway; confirm the current end-of-life date rather than assuming. 6.1 LTS: a longer remaining window — the least exciting and most operationally important row here.

Building a minimal guest kernel

The flow is identical for both series, which is the good news: the switching cost is re-validation, not re-learning. Start from Firecracker's reference config for the series, strip what a microVM has no hardware for, keep the virtio devices Firecracker actually emulates, and commit the result.

# Build a minimal Firecracker guest kernel from a stable LTS tag.
# Do this in a container or a scratch VM -- the kernel tree is large and
# the build litters your $HOME with nothing you want to keep.

SERIES=6.1          # or 5.10, same flow
TAG=v6.1.100        # pin an exact stable tag, never "latest"
ARCH=$(uname -m)

git clone --depth 1 --branch "$TAG" \
  https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git linux
cd linux

# Start from Firecracker's own reference config for this series and arch.
# These live in the Firecracker repo under resources/guest_configs/ --
# check the filenames there, they get renamed between releases.
curl -fsSL -o .config \
  "https://raw.githubusercontent.com/firecracker-microvm/firecracker/main/resources/guest_configs/microvm-kernel-ci-x86_64-${SERIES}.config"

# olddefconfig answers every NEW symbol in this series with its default.
# That is convenient and it is also how you ship 40 drivers you never
# read about. Diff the result before you trust it.
cp .config .config.reference
make olddefconfig
diff -u .config.reference .config | head -100

# Turn off the things a microVM has no hardware for.
./scripts/config --disable PCI
./scripts/config --disable ACPI
./scripts/config --disable MODULES        # build everything in, no modprobe
./scripts/config --disable SOUND
./scripts/config --disable DRM

# Keep the devices Firecracker actually emulates.
./scripts/config --enable VIRTIO
./scripts/config --enable VIRTIO_MMIO
./scripts/config --enable VIRTIO_BLK
./scripts/config --enable VIRTIO_NET
./scripts/config --enable VIRTIO_VSOCKETS
./scripts/config --enable HW_RANDOM_VIRTIO
./scripts/config --enable SERIAL_8250_CONSOLE

# If anything inside the guest runs iptables/nftables -- k3s, docker,
# a per-tenant firewall -- you need the match modules too, built in.
./scripts/config --enable NETFILTER_XT_MATCH_COMMENT
./scripts/config --enable NETFILTER_XT_MATCH_STATISTIC
./scripts/config --enable NETFILTER_XT_MATCH_RECENT
./scripts/config --enable NF_CONNTRACK
./scripts/config --enable IP_NF_NAT

make olddefconfig
make -j"$(nproc)" vmlinux        # x86_64: the uncompressed ELF at ./vmlinux
# make -j"$(nproc)" Image        # aarch64: ./arch/arm64/boot/Image

ls -lh vmlinux
file vmlinux                     # expect: ELF 64-bit LSB executable

# The config IS the artifact. Commit it, tag it, review changes to it.
cp .config "../guest-kernel-${TAG}-${ARCH}.config"
sha256sum vmlinux > "../vmlinux-${TAG}.sha256"

Two details catch people. `MODULES` off is usually right for a microVM — you know the device set at build time, nothing hotplugs, and building everything in removes a whole class of "the module wasn't there" failures. And Firecracker wants an uncompressed image: the ELF `vmlinux` on x86_64, the PE `Image` from `arch/arm64/boot/` on aarch64. Hand it a `bzImage` and the error message will not be as helpful as you'd like.

Pointing Firecracker at the kernel

Three PUTs and a start action. The boot args have to agree with the config: if you built without PCI and ACPI, say `pci=off acpi=off` so the kernel doesn't spend boot time hunting for buses that were never wired up.

# Point a Firecracker microVM at the kernel you just built.
# Firecracker takes an UNCOMPRESSED image: an ELF vmlinux on x86_64,
# a PE/Image on aarch64. A bzImage will be rejected.

SOCK=/tmp/fc.sock
rm -f "$SOCK"
firecracker --api-sock "$SOCK" &

# 1. Machine config -- vCPUs and RAM are fixed for the life of the VM
#    unless your guest kernel is new enough for virtio-mem.
curl -s --unix-socket "$SOCK" -X PUT "http://localhost/machine-config" \
  -H "Content-Type: application/json" \
  -d '{
        "vcpu_count": 2,
        "mem_size_mib": 1024,
        "smt": false
      }'

# 2. Boot source -- the kernel path plus the command line.
#    pci=off and acpi=off match a kernel built without them.
#    reboot=k + panic=1 turn "the guest died" into "the VM exited",
#    which is the only reboot semantics you want in a microVM.
curl -s --unix-socket "$SOCK" -X PUT "http://localhost/boot-source" \
  -H "Content-Type: application/json" \
  -d '{
        "kernel_image_path": "/var/lib/kernels/vmlinux-6.1.100",
        "boot_args": "console=ttyS0 reboot=k panic=1 pci=off acpi=off i8042.noaux=1 i8042.nomux=1 init=/usr/bin/pandastack-init"
      }'

# 3. Root device.
curl -s --unix-socket "$SOCK" -X PUT "http://localhost/drives/rootfs" \
  -H "Content-Type: application/json" \
  -d '{
        "drive_id": "rootfs",
        "path_on_host": "/var/lib/pandastack/vms/abc/rootfs.ext4",
        "is_root_device": true,
        "is_read_only": false
      }'

# 4. Go.
curl -s --unix-socket "$SOCK" -X PUT "http://localhost/actions" \
  -H "Content-Type: application/json" \
  -d '{"action_type": "InstanceStart"}' 

Then verify from inside the guest rather than from your build notes. Build notes describe the kernel you meant to ship; this describes the one you did.

from pandastack import Sandbox

# Ask the guest what it actually got. Run this the day you switch series,
# and again after every re-bake, because a config regression is invisible
# until the one workload that needed CONFIG_FOO shows up in a ticket.
CHECKS = [
    ("kernel", "uname -r"),
    ("cmdline", "cat /proc/cmdline"),
    ("virtio devices", "ls /sys/bus/virtio/devices"),
    ("virtio-mem present", "grep -c virtio_mem /proc/devices || echo 0"),
    ("io_uring syscall", "grep -c io_uring_setup /proc/kallsyms || echo 0"),
    ("cgroup version", "stat -fc %T /sys/fs/cgroup"),
    ("xt_comment", "iptables -A OUTPUT -m comment --comment probe -j ACCEPT "
                   "&& iptables -D OUTPUT -m comment --comment probe -j ACCEPT "
                   "&& echo ok"),
]

with Sandbox.create(template="base", ttl_seconds=300) as sbx:
    for label, cmd in CHECKS:
        r = sbx.exec(cmd, timeout_seconds=20)
        status = "OK " if r.exit_code == 0 else "FAIL"
        out = (r.stdout or r.stderr).strip().splitlines()
        print(f"{status} {label:20} {out[0] if out else '(empty)'}")

# The xt_comment probe is the one that catches you. kube-proxy writes
# every rule with -m comment, so a kernel without that match turns
# "install k3s in a microVM" into an hour of confused iptables-restore
# output that never mentions the kernel.

Snapshots are married to the kernel that took them

This is the cost that surprises people. A Firecracker snapshot is the guest's memory plus its device and vCPU state, captured from a machine that was running one specific kernel. Restoring it is not booting; it's resuming. There is no point in that sequence where a different kernel could be substituted, because the kernel is already in the memory you're restoring.

So kernel changes want schema-migration machinery: version the artifact, publish the new generation beside the old, cut templates over one at a time, keep the previous generation on disk. Forking adds a wrinkle — a fork inherits its parent's memory, so a fork tree rooted at an old-kernel snapshot stays on the old kernel however deep it goes. Same-host forks land in 400 to 750ms and cross-host forks in 1.2 to 3.5 seconds, and none of that latency includes noticing that the child is running last quarter's kernel.

A kernel upgrade in a container fleet is a host reboot. A kernel upgrade in a microVM fleet is a rebuild of every snapshot you have ever taken. Price it accordingly.

When to move, and when to stay put

Stay on 5.10 if your microVMs boot fast, run some code, and get destroyed. That's the majority case and a respectable place to be. If nothing in the guest needs a post-5.10 feature, upgrading buys you a longer support window and a novel artifact — and the support window is a real reason, but it's a scheduling decision, not an urgent one.

Move to 6.1 or newer when you can name the feature. The honest triggers:

1. You need to grow a running guest's RAM. virtio-mem needs a guest newer than 5.10, full stop; no amount of VMM-side support changes that.
2. You run container orchestration inside the guest — k3s, Docker, nested CI — and you're fighting netfilter gaps. Fix the config first, since that's usually the real bug, but if you're rebuilding anyway, rebuild on the series the ecosystem targets.
3. Something in the guest leans hard on io_uring, or your guest userspace is new enough to assume cgroup v2 semantics throughout.
4. Your support horizon is the problem: you need security fixes for longer than the older LTS will provide them — a compliance conversation as much as a technical one.
5. You're building in-guest hardening — landlock, newer seccomp, BPF-based policy — above the hypervisor boundary rather than relying on it alone.

If none of those apply, write down that you evaluated it, note which trigger would change your mind, and go do something else. The worst outcome here isn't picking the older series — it's picking the newer one for aesthetic reasons, re-baking every snapshot you own, and finding that the only observable difference is that your images got bigger.

Whichever you pick, own it properly: pin the tag, commit the config, review its diffs, and keep a probe that runs inside a real guest and asserts the features you depend on are present. The kernel is the one dependency in a microVM platform that no package manager will upgrade behind your back — which means nobody will upgrade it for you, either.