all posts
blog · topic

Platform internals

Millisecond sandboxes come from a stack of kernel mechanisms working together: KVM, userfaultfd, copy-on-write storage, network namespaces and cgroups. These posts go underneath the API into memory, networking, storage and scheduling, with the measurements behind the design choices.

PandaStack benchmarks

160 posts

·11 min read

Giving a MicroVM Access to a Customer's Private Network

The customer's database is in their VPC and your sandbox is not. The naive answer is to hand them your egress IPs and ask them to open a hole; the answer that survives a security review is a WireGuard peer per sandbox, minted after restore, revoked on teardown, and never, ever baked into a snapshot.

networkingsecuritywireguard
Ajay Kumar
·10 min read

Rowhammer and the Attacks Below Your Hypervisor

Every isolation guarantee you buy is enforced by software running on hardware that several tenants share. Rowhammer is the clearest example of what that sentence costs: a bit flip in a DRAM row you do not own, achieved by physics rather than by a bug. Here's the honest version — what it takes to land, what ECC and TRR really buy, and the two mitigations that actually change the answer.

securityisolationhardware
Ajay Kumar
·10 min read

How to Put a CDN in Front of a Scale-to-Zero App

An origin that is allowed to sleep changes what your cache headers are for. Done properly, the wake happens in a background fetch and lands on nobody's request; done carelessly, one Vary header hands every visitor a cold boot.

cdncachingscale-to-zero
Ajay Kumar
·10 min read

Skipping DHCP: the Firecracker boot line you actually ship

Everybody ships the same eleven-token Firecracker kernel command line and nobody measures it. Ours is four tokens plus a generated ip= that skips DHCP entirely — here is how to work out which tokens are buying you anything, and why the answer changes completely once you restore snapshots instead of booting.

firecrackerkernelmicrovm
Ajay Kumar
·11 min read

virtiofs vs virtio-blk: How Files Actually Get Into a MicroVM

One gives the guest a disk it owns. The other gives it a window onto a directory the host owns. That single difference decides whether you can fork a machine in 400ms, who parses guest-controlled input, and what a multi-tenant escape looks like.

virtiofsvirtio-blkfirecracker
Ajay Kumar
·11 min read

What "persistent" actually means in a sandbox

You wrote the file. You ran cat and saw it. Neither of those facts says the bytes are on a disk. Here is every layer a write passes through inside a microVM, which of them a crash erases, and why the honest answer for an ephemeral rootfs is not "fsync harder" but "get the artifact out".

durabilitystoragesandboxes
Ajay Kumar
·11 min read

Interrupts, IRQs, and Where microVM Tail Latency Comes From

Median latency tells you the machine works. p99 tells you how the machine is built. Here is every handoff a virtio interrupt makes inside a Firecracker guest, which of those handoffs are queues, and which ones you can actually do something about.

tail-latencyfirecrackermicrovm
Ajay Kumar
·11 min read

Why the disk under your sandbox fleet decides your boot time

People pick a sandbox platform on features and then get bitten by storage hardware. A reflink clone is metadata-only and nearly free; every copy-on-write byte afterwards is a real read-modify-write against a real device. Under 50 concurrent restores, that device is either local NVMe or it is your bottleneck.

storagenvmecopy-on-write
Ajay Kumar
·11 min read

Your benchmark ran at a different clock speed than production

The same core does not run at the same speed twice. Governor, turbo bin, how many neighbours are busy, thermal headroom and ramp latency all move it — which is why the first sandbox on a quiet host looks fast and the fiftieth on a busy one gets blamed on the platform.

cpubenchmarkingperformance
Ajay Kumar

More posts in this topic