all posts
blog · topic

Platform internals

Millisecond sandboxes come from a stack of kernel mechanisms working together: KVM, userfaultfd, copy-on-write storage, network namespaces and cgroups. These posts go underneath the API into memory, networking, storage and scheduling, with the measurements behind the design choices.

PandaStack benchmarks

170 posts

·11 min read

nftables vs iptables for Per-Sandbox Networking: Why I Still Write iptables

PandaStack writes `iptables` into 16,384 pre-allocated network namespaces, which is the unfashionable choice. The reason is that nftables' biggest win — not walking a long linear chain — is an argument about rule count, and our rule count per namespace is four. What actually hurts at fleet scale is the state nobody namespaced properly.

nftablesiptablesnetfilter
Ajay Kumar
·11 min read

memory.high vs memory.max: Throttle or Kill

Hard limit and soft limit is the wrong mental model. One of these files converts an overcommit mistake into a dead process; the other converts it into a latency problem. For a multi-tenant fleet, that difference is the entire design.

cgroupslinuxmemory
Ajay Kumar
·11 min read

Guest Writeback Throttling: Why dirty_ratio Is Wrong for a 2 GiB MicroVM

The guest is extremely confident about a disk that does not exist. The host is also confident, about different things. Between those two confidences live your tail latencies.

firecrackerlinux-kernelwriteback
Ajay Kumar
·11 min read

DAMON: Measuring What a Sandbox Actually Touches

Charging a host for every guest's configured RAM is what caps your density, and it is wrong by a lot — a guest handed 4 GiB to run an idle Node server touches a fraction of it. DAMON is the kernel subsystem that can tell you which fraction, at a cost bounded by region count instead of memory size. Here is how it works, and the three different wrong answers you can get instead.

linuxmemoryinternals
Ajay Kumar
·11 min read

Per-Sandbox Disk I/O Throttling and the Neighbour You Cannot See

CPU contention is visible and roughly fair. Memory contention has pressure signals. Disk contention shows up as latency in somebody else's unrelated workload, with no attribution and no ticket. Here are the three places you can actually throttle it, what each one really controls, and the two layers that quietly lie to you.

storagecgroupsfirecracker
Ajay Kumar
·11 min read

Patching the Host Kernel Under Running MicroVMs

The host kernel is the thing your isolation boundary is made of. It cannot be restarted without taking every guest on the box with it, and the clock on the patch is a security clock.

operationssecurityinfrastructure
Ajay Kumar
·11 min read

MGLRU and MicroVM Density: Reclaim When Every Page Is Someone's Guest

The host kernel picks reclaim victims using access information already filtered through a second kernel. MGLRU gives it generations instead of two buckets — which changes the shape of your bad days, not how many guests fit on the box.

linuxmemoryinternals
Ajay Kumar
·10 min read

The I/O Scheduler Inside a MicroVM Is Almost Always Wrong

Your guest's block layer is sorting requests by sector number to optimise seeks on a platter that is a file on somebody else's filesystem. Why `none` is the answer, why it still merges, and why readahead is the knob that actually moves the number.

firecrackermicrovmlinux
Ajay Kumar
·10 min read

Build, Boot, Test: Kernel CI on MicroVMs (and Where It Stops)

"Does this patch boot" should be a per-commit check, not a nightly. A microVM makes the boot step sub-second and a git bisect a coffee break — right up to the device model, the nested-virt wall, and the fact that you cannot swap our guest kernel.

kernellinuxci
Ajay Kumar

More posts in this topic