all posts
blog · topic

Platform internals

Millisecond sandboxes come from a stack of kernel mechanisms working together: KVM, userfaultfd, copy-on-write storage, network namespaces and cgroups. These posts go underneath the API into memory, networking, storage and scheduling, with the measurements behind the design choices.

PandaStack benchmarks

173 posts

·10 min read

virtio-net Offloads and Why Your microVM's Network Is Slower Than the Wire

Bulk transfers look fine. A sixty-four-byte request/response benchmark gets a third of bare metal. Both numbers are correct, and the gap between them is a virtio feature negotiation, a TAP device with its own opinion, and a per-packet tax that every sandbox on the host pays at once.

firecrackervirtio-netnetworking
Ajay Kumar
·11 min read

Flamegraphs Inside a microVM: When perf Has No Counters to Count

The flamegraph was 1,400 pixels wide and ninety-six percent of it was one box labelled [unknown]. perf runs fine inside a microVM; the recipe everybody memorised depends on three things a guest quietly does not provide, and perf will not tell you which one bit you.

perfflamegraphsprofiling
Ajay Kumar
·11 min read

Encryption at Rest for Sandbox Disks and Snapshots

"Is customer data encrypted at rest?" is one checkbox covering four very different files, three of which are not what the reviewer is picturing. One of them is a verbatim copy of someone's RAM.

encryption-at-restsecuritysnapshots
Ajay Kumar
·11 min read

nftables vs iptables for Per-Sandbox Networking: Why I Still Write iptables

PandaStack writes `iptables` into 16,384 pre-allocated network namespaces, which is the unfashionable choice. The reason is that nftables' biggest win — not walking a long linear chain — is an argument about rule count, and our rule count per namespace is four. What actually hurts at fleet scale is the state nobody namespaced properly.

nftablesiptablesnetfilter
Ajay Kumar
·11 min read

memory.high vs memory.max: Throttle or Kill

Hard limit and soft limit is the wrong mental model. One of these files converts an overcommit mistake into a dead process; the other converts it into a latency problem. For a multi-tenant fleet, that difference is the entire design.

cgroupslinuxmemory
Ajay Kumar
·11 min read

Guest Writeback Throttling: Why dirty_ratio Is Wrong for a 2 GiB MicroVM

The guest is extremely confident about a disk that does not exist. The host is also confident, about different things. Between those two confidences live your tail latencies.

firecrackerlinux-kernelwriteback
Ajay Kumar
·11 min read

DAMON: Measuring What a Sandbox Actually Touches

Charging a host for every guest's configured RAM is what caps your density, and it is wrong by a lot — a guest handed 4 GiB to run an idle Node server touches a fraction of it. DAMON is the kernel subsystem that can tell you which fraction, at a cost bounded by region count instead of memory size. Here is how it works, and the three different wrong answers you can get instead.

linuxmemoryinternals
Ajay Kumar
·11 min read

Per-Sandbox Disk I/O Throttling and the Neighbour You Cannot See

CPU contention is visible and roughly fair. Memory contention has pressure signals. Disk contention shows up as latency in somebody else's unrelated workload, with no attribution and no ticket. Here are the three places you can actually throttle it, what each one really controls, and the two layers that quietly lie to you.

storagecgroupsfirecracker
Ajay Kumar
·11 min read

Patching the Host Kernel Under Running MicroVMs

The host kernel is the thing your isolation boundary is made of. It cannot be restarted without taking every guest on the box with it, and the clock on the patch is a security clock.

operationssecurityinfrastructure
Ajay Kumar

More posts in this topic