Guest kernel lockdown and module loading in Firecracker microVMs

Root inside a sandbox guest is not a breach. It is Tuesday. If you hand a microVM to a coding agent, a CI job, or a stranger's pull request, that workload is going to be uid 0 in its own kernel, and it is supposed to be — that's the deal a VM makes. The hypervisor is the boundary; the guest kernel is disposable. So the natural conclusion is that nothing root does inside the guest is worth defending against, and you can stop reading here.

I'm Ajay; I build PandaStack, which runs untrusted and model-generated code in Firecracker microVMs. That conclusion is about eighty percent right, and the missing twenty percent is expensive. What root can do to the guest kernel is the first half of every escape chain that has ever worked — you don't reach the VMM's virtio parsers from Python, you reach them from ring 0 after you've loaded a module or rewritten kernel memory. It's also the difference between a guest whose state you can reason about and one you can't. This post is about the specific levers: the lockdown LSM, module loading policy, why a small CONFIG surface is itself a control, and the snapshot wrinkle that quietly invalidates most people's hardening scripts.

Why guest kernel state matters even when the guest is disposable

Three reasons, in descending order of how much they should worry you. First, escape chains. Nobody escapes a hypervisor from a Python process; they escape it from kernel context, because that's where you can talk to the virtio device queues directly, craft the malformed descriptor, hammer the MMIO register, or hold a page mapped while the VMM reads it. Getting to ring 0 in the guest is step one and it's usually the easy step. Every mechanism that makes step one harder — no module loader, no /dev/mem, no kexec — removes a well-trodden path to the position from which the interesting attacks are launched.

Second, unpredictable guest state. A guest that can load modules and rewrite its own kernel is a guest whose behaviour you cannot reason about, which matters enormously if you snapshot it. A snapshot of a tampered kernel is a permanent, restorable copy of that tampering. Third, and least dramatic but most common in practice: your own debugging. When a restored VM behaves strangely, "could the workload have modified the kernel?" is a question you'd rather answer with "structurally, no" than with a forensic investigation.

The lockdown LSM: the kernel refusing its own root

Linux lockdown is a Linux Security Module, upstream since 5.4, that severs the traditional equivalence between uid 0 and the ability to modify the running kernel. It exists because Secure Boot's promise — only signed code runs in ring 0 — is worthless if root can simply write to /dev/mem and patch the kernel it just verified. Lockdown closes the userspace-to-kernel-memory doors that root historically walked through.

The important structural point is that lockdown is not a capability check. It isn't asking whether you hold CAP_SYS_MODULE or CAP_SYS_RAWIO; it's a separate hook (security_locked_down()) placed at each dangerous operation, and it says no to a full-capability, uid 0, no-namespace-tricks process. That's unusual on Linux and it's exactly what you want in a guest where the workload is legitimately root. Dropping capabilities in a sandbox where the user is meant to be admin is theatre. Lockdown isn't, because it constrains the administrator by design.

It has three modes, and they nest: none, integrity, confidentiality. Each is a superset of the last, and the level is a one-way ratchet — you can raise it at runtime, you can never lower it without a reboot.

Integrity mode: you may not modify the running kernel

Integrity blocks the operations that let userspace change kernel code or data. The set is worth knowing by name, because each one is a real technique someone has used:

• /dev/mem, /dev/kmem, /dev/port — direct physical-memory and I/O-port access from userspace. The original kernel-patching primitive; if you have this, you have ring 0 without needing a bug.
• Unsigned module loading — with lockdown active, module signature enforcement is applied regardless of whether CONFIG_MODULE_SIG_FORCE was set at build time. An unsigned .ko is refused.
• kexec — loading and booting a second kernel from the first. Trivially defeats every other guarantee: if you can boot your own kernel, the hardened one you booted from is irrelevant.
• Hibernation — writing a kernel image to disk and resuming from it is, from lockdown's perspective, an unsigned-kernel-loading primitive wearing a hat.
• Raw PCI config access, ioperm/iopl, MSR writes, and custom ACPI tables — the hardware-adjacent side doors. Mostly moot in a microVM with no PCI bus, which is precisely the point about surface.
• BPF writes to kernel memory — the bpf_probe_write_user side of BPF, plus debug-interface writes to kernel memory. BPF is a legitimate tool that also happens to be a supervised way to touch the kernel.
• Unsafe module parameters — parameters that themselves set hardware addresses, which is /dev/mem with extra steps.

Confidentiality mode: you may not read the running kernel either

Confidentiality adds everything integrity blocks plus the read side — the things that would leak kernel memory, and with it the secrets and addresses that make an exploit reliable. /proc/kcore goes away. kprobes go away. BPF reading kernel memory goes away. perf's kernel-side access, tracefs, and the kernel-memory read paths of debug interfaces go with them. It is a genuinely aggressive mode and it will break your observability tooling, which is the honest tradeoff: you're trading the ability to introspect the kernel for the guarantee that nobody else can either.

For a microVM guest running untrusted code, confidentiality is usually the right default, because the observability you'd lose inside the guest is observability you should be collecting from the host anyway. For a guest running a workload you own — a managed database, your own app — integrity is the sane setting, because you'll want profilers to work.

# --- Run this INSIDE the guest to see what its kernel actually permits ---

# 1. Lockdown state. securityfs has to be mounted for this file to exist.
mountpoint -q /sys/kernel/security || mount -t securityfs none /sys/kernel/security
cat /sys/kernel/security/lockdown 2>/dev/null || echo "lockdown LSM: not present"
# none [integrity] confidentiality      <- brackets mark the ACTIVE mode
# If the file is missing, either the LSM isn't built in or it isn't in the
# active LSM list -- check the boot line the kernel recorded for itself:
grep -o 'lsm=[^ ]*\|lockdown=[^ ]*' /proc/cmdline

# 2. Module surface. If the loader was compiled out, this file does not exist.
ls /proc/modules 2>/dev/null || echo "CONFIG_MODULES=n: no module loader at all"
sysctl kernel.modules_disabled 2>/dev/null || echo "no modules_disabled knob"

# 3. The one-way switches worth checking (and setting at bake time).
sysctl kernel.modules_disabled kernel.kexec_load_disabled \
       kernel.unprivileged_bpf_disabled kernel.kptr_restrict \
       kernel.dmesg_restrict kernel.perf_event_paranoid 2>&1

# 4. Prove a denial rather than assuming one. Under integrity lockdown a
#    read of /dev/mem fails even for uid 0 -- root is not the check here.
dd if=/dev/mem of=/dev/null bs=1 count=1 2>&1 | tail -1
dmesg | grep -i 'lockdown' | tail -5
#   Lockdown: dd: /dev/mem,kmem,port is restricted; see man kernel_lockdown.7

Module loading in a microVM: three answers, in order of preference

Loadable modules exist because a distro kernel cannot know what hardware it will meet. A Firecracker guest kernel is the opposite of that situation: the hardware is fixed, virtual, and known when you compile. Firecracker gives the guest virtio-blk, virtio-net, virtio-vsock, an entropy source, and a serial console over MMIO — that's essentially the whole machine. There is no card someone might plug in later. So the question isn't really "how do I secure module loading" but "why do I have a module loader at all."

The simplest answer is the best one: build the drivers you need directly into the kernel and set # CONFIG_MODULES is not set. That doesn't disable module loading — it removes the loader. There is no init_module or finit_module to call, no /proc/modules, no modules tree in the rootfs, no signature policy to get wrong, and no clever trick to re-enable it, because the code isn't there. It is the rare security control with negative complexity cost.

If you genuinely need runtime modules — a custom filesystem, an eBPF-adjacent driver, a workload that builds its own kernel module as part of the job — the second answer is signature enforcement: CONFIG_MODULE_SIG=y with CONFIG_MODULE_SIG_FORCE=y, so a module without a valid signature from a key in the kernel's keyring is rejected, not merely tainted. Without SIG_FORCE, an unsigned module loads fine and just sets a taint flag, which is a diagnostic, not a control. (Lockdown at integrity or above enforces signatures anyway, which is a nice belt-and-braces property: even a kernel built without SIG_FORCE gets the enforcement once lockdown engages.)

The third answer is the runtime one: kernel.modules_disabled. Setting this sysctl to 1 permanently disables module loading and unloading for the life of the boot — it is a one-way switch, and unlike most sysctls there is no path back to 0 short of a reboot. The intended pattern is to load whatever modules you need during early boot and then slam the door. It only exists when CONFIG_MODULES=y, which is a useful tell: if the sysctl is missing, you already took the better option.

• Enforcement point — CONFIG_MODULES=n: compile time, absolute. The syscalls do not exist, so there is nothing to bypass, misconfigure, or forget to set. CONFIG_MODULE_SIG_FORCE=y: load time, cryptographic. Modules still load; they just have to be signed by a key baked into the kernel keyring.
• What it costs you — CONFIG_MODULES=n: any driver you didn't compile in is permanently unavailable, and a workload that builds its own module simply cannot run. CONFIG_MODULE_SIG_FORCE=y: you now operate a signing key, a build step that signs, and a rotation story — and the private key becomes a thing worth stealing.
• Failure mode if you get it wrong — CONFIG_MODULES=n: loud and immediate. The guest won't boot or a feature won't work, and you find out on the first cold boot. CONFIG_MODULE_SIG_FORCE=y: quiet. Forget the FORCE and unsigned modules load with only a taint flag; nobody notices until an incident review.
• Fit for a Firecracker guest — CONFIG_MODULES=n: near-perfect, because the virtual hardware set is fixed at build time and never changes underneath you. CONFIG_MODULE_SIG_FORCE=y: the right answer only when a real requirement forces modules to exist, which for a purpose-built microVM guest is rarer than people assume.

# --- Guest kernel: what root inside the VM is allowed to do to the kernel ---
# (a fragment, not a full .config -- reconcile with `make olddefconfig`
#  against your kernel version; symbol availability shifts between releases)

# The lockdown LSM itself. Needs CONFIG_SECURITY=y, and "lockdown" has to
# actually appear in the active LSM list to do anything.
CONFIG_SECURITY=y
CONFIG_SECURITY_LOCKDOWN_LSM=y
CONFIG_SECURITY_LOCKDOWN_LSM_EARLY=y      # engage during early boot, not after init
CONFIG_LOCK_DOWN_KERNEL_FORCE_INTEGRITY=y # compile-time default mode
CONFIG_LSM="lockdown,yama,bpf"            # order/contents depend on what else you build

# The simplest answer to "how do I stop module loading?": don't build the loader.
# CONFIG_MODULES is not set

# ...and if you DO need modules, make unsigned ones unloadable.
# CONFIG_MODULES=y
# CONFIG_MODULE_SIG=y
# CONFIG_MODULE_SIG_FORCE=y               # reject unsigned/badly-signed modules outright
# CONFIG_MODULE_SIG_ALL=y                 # sign everything during the build
# CONFIG_MODULE_SIG_SHA256=y

# Doors lockdown closes at runtime that you can also just never install.
# CONFIG_DEVMEM is not set                # no /dev/mem
# CONFIG_DEVPORT is not set               # no /dev/port
# CONFIG_PROC_KCORE is not set            # no /proc/kcore view of kernel memory
# CONFIG_KEXEC is not set                 # can't boot a second kernel from the first
# CONFIG_KEXEC_FILE is not set
# CONFIG_HIBERNATION is not set           # nothing to hibernate to in a microVM anyway
# CONFIG_KPROBES is not set               # no live kernel instrumentation
# CONFIG_DEBUG_FS is not set              # debugfs is a large, unaudited surface
# CONFIG_BPF_SYSCALL is not set           # if nothing in the guest needs BPF

# If you keep /dev/mem for some reason, at least keep these on:
# CONFIG_STRICT_DEVMEM=y
# CONFIG_IO_STRICT_DEVMEM=y

A minimal CONFIG surface is itself a security control

Notice what the fragment above is mostly doing: it is not configuring defenses, it is declining to build attack surface. Lockdown blocks /dev/mem at runtime; # CONFIG_DEVMEM is not set means there is no /dev/mem to block. Lockdown blocks kexec; # CONFIG_KEXEC is not set means the kexec syscalls were never compiled. Confidentiality mode blocks /proc/kcore and kprobes; not building CONFIG_PROC_KCORE and CONFIG_KPROBES achieves the same outcome with no LSM in the path at all.

This is the strongest argument for the minimal guest kernel, and it's stronger than the usual boot-speed one. A driver you compiled out cannot have an exploitable parser. A filesystem you never built cannot be mounted with a corrupted superblock crafted by a workload that noticed you left the driver in. Every CONFIG symbol you leave off deletes an entire class of bug from your guest permanently, including bugs that will be discovered in 2029 in code you never shipped. Firecracker guests need virtio over MMIO, a serial console, the KVM guest bits, and one filesystem. Everything else is a choice, and the default choice should be no.

The two controls compose rather than compete. Compile out what you can; run lockdown for the surface you had to keep. If your guest needs BPF for the workload's own observability, you keep CONFIG_BPF_SYSCALL and let lockdown constrain what BPF may touch. If it doesn't, you delete the syscall and lockdown has one less job.

The snapshot wrinkle: hardening belongs in the bake, not in boot

Here is the part that catches teams who ported their hardening from a normal fleet. A Firecracker snapshot is a serialization of a running machine — guest RAM plus VMM and vCPU state, captured mid-execution. That memory image contains a live kernel: its data structures, its LSM state, its sysctl values, its list of loaded modules. Restoring the snapshot doesn't boot anything. It resumes a machine that was already up.

Which means every per-boot hardening mechanism you're used to silently stops running. The systemd unit that sets kernel.modules_disabled=1 after early boot? It ran once, at bake time, on the machine that got snapshotted — and if it ran after the snapshot was taken, it will never run again on any restored VM. The sysctl.d drop-in, the init script that mounts securityfs and raises the lockdown level, the tool that unloads a module it no longer needs: all of them are boot-time events, and a restored guest has no boot.

The flip side is the good news, and it's genuinely good. Because kernel state is frozen into the memory image, hardening applied before the snapshot is applied to every restore, for free, forever, with zero per-create latency. Set lockdown to confidentiality at bake time and every VM restored from that snapshot comes up already locked down — not "locks down shortly after start," but already, at the instant the first guest instruction executes. A one-way ratchet inside a snapshot is the ideal case for a one-way ratchet: you pay the cost once and inherit the result a million times.

Practically: put the hardening in the template build, verify it on the built image before the bake, and treat a change to it as a template re-bake rather than a config push. A kernel rebuild invalidates existing snapshots anyway, so a lockdown or module-policy change rides along with the same rollout you already need.

Probing a guest's kernel posture from outside

None of this should be taken on faith, including from your own build pipeline. The check that matters is running a real probe against the actual template you're about to bake, and asserting on denials rather than on the presence of config files. With a sandbox API, that's a short script that boots the template, interrogates the kernel, and throws the machine away:

from pandastack import Sandbox

# Audit what a template's guest kernel actually allows root to do to itself.
# Run this against a template BEFORE you bake it, because after the bake the
# answer is frozen into every snapshot restore forever.

PROBE = r"""#!/bin/sh
mountpoint -q /sys/kernel/security 2>/dev/null || \
  mount -t securityfs none /sys/kernel/security 2>/dev/null
echo "cmdline: $(cat /proc/cmdline)"
echo "lockdown: $(cat /sys/kernel/security/lockdown 2>/dev/null || echo ABSENT)"
echo "module_loader: $([ -e /proc/modules ] && echo present || echo compiled-out)"
echo "loaded_modules: $(wc -l < /proc/modules 2>/dev/null || echo 0)"
for k in kernel.modules_disabled kernel.kexec_load_disabled \
         kernel.unprivileged_bpf_disabled kernel.kptr_restrict \
         kernel.dmesg_restrict; do
  echo "$k = $(sysctl -n $k 2>/dev/null || echo ABSENT)"
done
# Not "can I read /dev/mem" in the abstract -- actually try it.
dd if=/dev/mem of=/dev/null bs=1 count=1 2>&1 | tail -1
"""

with Sandbox.create(template="base", ttl_seconds=300) as sbx:
    sbx.filesystem.write("/tmp/kprobe.sh", PROBE)
    out = sbx.exec("sh /tmp/kprobe.sh", timeout_seconds=30)
    print(out.stdout)
    if out.exit_code != 0:
        print("probe stderr:", out.stderr[-2000:])

    # The interesting negative test: try to load a module and expect failure.
    # A zero exit code here is the finding, not the success.
    ins = sbx.exec("insmod /tmp/nonexistent.ko", timeout_seconds=15)
    print("insmod exit:", ins.exit_code, "|", ins.stderr.strip()[:200])

    # Keep the evidence. filesystem.read() returns bytes.
    sbx.filesystem.write("/tmp/report.txt", out.stdout)
    report = sbx.filesystem.read("/tmp/report.txt")
    open("kernel-posture.txt", "wb").write(report)

    sbx.kill()   # the machine, its kernel, and anything root did to it, gone

The negative test in the middle is the one people skip. Checking that /sys/kernel/security/lockdown says confidentiality tells you what the kernel claims; trying to read /dev/mem and getting refused tells you what the kernel does. Wire the second form into template CI, because a config regression that flips lockdown back to none is exactly the kind of change that passes every test you'd think to write.

Being honest: this is hardening, not isolation

Lockdown was designed to protect a Secure Boot chain from a cooperative-but-compromised root, not to contain a determined adversary who already holds ring 0 and has time. It closes documented doors. It does not make the guest kernel exploit-proof — a kernel bug reachable from an ordinary syscall doesn't care what lockdown mode you're in, and a workload that gets code execution in the kernel through such a bug is past the LSM entirely. Treat lockdown as raising the cost and narrowing the paths, which is a real and worthwhile thing, and not as a boundary.

The boundary is the hypervisor. That's the layer where a compromise of the guest kernel — total, uncontested, root-with-a-debugger compromise — still leaves the attacker inside one VM, facing KVM's hardware-enforced separation and a seccomp-confined VMM behind it. That's why it's coherent to hand a tenant root in the guest at all. Guest hardening is the layer that makes reaching the boundary harder; the boundary is what makes reaching it survivable.

And it's worth naming what a container gets from this section, because it's nothing. In a container, the "guest" kernel is the host kernel. You do not set its lockdown mode from inside — /sys/kernel/security is not yours, and raising the level would apply to the host and every co-tenant. kernel.modules_disabled is a global, non-namespaced, one-way switch: a container that could set it would be disabling module loading for the entire machine. Module signing policy, the CONFIG surface, whether kexec exists — all decided by whoever built the host kernel, which is not the tenant. A container can drop capabilities and install a seccomp filter, and those are worth doing. But the entire subject of this post — what root in your environment may do to your kernel — is a question containers can't ask, because the kernel isn't theirs. Getting your own kernel to harden is a microVM property.

How PandaStack handles it

PandaStack runs every sandbox, managed Postgres database, and git-driven app as a Firecracker microVM with its own purpose-built guest kernel, and users get real root inside it — that's the product. The guest kernel is minimal by construction: virtio over MMIO, a serial console, the KVM guest bits, and the one filesystem the rootfs needs, which means most of what lockdown would have to block was never compiled in the first place. What remains is a bake-time decision, because of the snapshot mechanics above: hardening is applied to the template before the snapshot is captured, so it's already in force at the first instruction of every restored VM rather than being applied by something that races the workload.

That's also why it costs nothing at create time. Every create restores a baked snapshot rather than cold-booting — roughly 179ms p50 and about 203ms p99, with the restore step itself around 49ms — and a cold boot near 3s happens once per template, at bake, which is exactly when the hardening runs. Forks inherit the same frozen kernel state (400–750ms same-host, 1.2–3.5s cross-host), and managed Postgres, which takes 30–90s to create, spends that time on database bootstrap rather than on kernel setup. Networking is per-sandbox virtio-net in its own namespace out of a pool of 16,384 pre-allocated /30 subnets per agent. The core is open source under Apache-2.0, so you can build your own guest kernel, set your own lockdown mode, bake it, and check the posture of a restored VM yourself — which, given everything above, is the only check that actually means anything.