Why the disk under your sandbox fleet decides your boot time

Nobody evaluates a sandbox platform by asking what disk the hosts have. They read the docs, check the SDK, run one create, see a number they like, and sign up. Then eight weeks later the p99 doubles at a specific time of day, nothing in the application changed, and the platform's status page is green. What changed is the device underneath, and the reason it took eight weeks to show up is that a single create is a terrible test of storage.

I'm Ajay; I build PandaStack, where every sandbox create is a Firecracker snapshot restore and there is no warm pool of idle VMs waiting to absorb the work. That design makes us unusually exposed to the host disk: if the device is slow, we have nowhere to hide it. So this post is what I've learned from being on the wrong side of that exposure — how a create actually loads the device, why the free part of copy-on-write is free and the expensive part is invisible until it isn't, and why local NVMe is close to mandatory for a dense fleet even though local NVMe is the one disk that dies with the instance.

What a sandbox create actually asks of the disk

Our create path is roughly eight steps and takes about 179ms at p50: allocate a pre-built network namespace, patch the tap, clone the rootfs, fork and exec firecracker, POST the snapshot load, resume, probe TCP port 22, and write the database row asynchronously. Three of those touch storage, and they touch it in three completely different ways.

The rootfs clone is a metadata operation — no data movement at all, if the filesystem cooperates. The snapshot load maps a multi-gigabyte memory image and starts touching it. The guest's first seconds of life are a scattering of small reads against the rootfs as the resumed userspace touches files that were not in the page cache at bake time. None of that is a big sequential stream. It is small, random, and bursty, which is precisely the profile that separates devices.

Now multiply by concurrency. One create is a rounding error on any SSD. Fifty creates landing in the same second on the same host is a queue: fifty guests all faulting in memory pages and rootfs blocks at once, each fault a small read that some device has to answer before that guest can make progress. A sandbox fleet is a concurrency machine. The interesting number was never single-threaded throughput; it was how the latency distribution behaves at depth.

The clone is free. The writes are not.

Copy-on-write gets described as "instant cloning" and that description is half true in a way that hides the whole cost model. The clone genuinely is instant. On XFS or btrfs, the FICLONE ioctl points a new file's extents at the same physical blocks as the source and updates reference counts. It is O(metadata): a 10 GiB rootfs clones in about the same time as a 1 GiB one, because no data moves either way.

Here is the size of that effect in our own code, from the comment sitting above the mount setup in cloud-init: putting the agent's data directory on XFS with reflink enabled drops the per-sandbox rootfs clone from roughly 5500ms as a full copy to roughly 1ms. In the restore path we treat reflink as the fast lane at around 2–25ms, dm-snapshot as the fallback at around 80ms on filesystems without reflink, and a real copy as the last resort that scales with image size. Same code, same hardware, three orders of magnitude apart, determined entirely by how the filesystem was formatted.

# Does this filesystem actually do reflinks? Cheapest checks first.

# 1. What filesystem is it? (ext4 reports as "ext2/ext3" here -- yes, really.)
stat -f -c %T /var/lib/pandastack

# 2. On XFS, reflink is a superblock feature. If it is off, it is off forever:
#    mkfs is the only place to turn it on.
xfs_info /var/lib/pandastack | grep -o 'reflink=[01]'

# 3. The only check that cannot lie: attempt one.
truncate -s 1G /var/lib/pandastack/.reflink-probe.src
cp --reflink=always /var/lib/pandastack/.reflink-probe.src \
                    /var/lib/pandastack/.reflink-probe.dst \
  && echo "reflink OK" \
  || echo "NO reflink -- every clone is a full copy, silently"
rm -f /var/lib/pandastack/.reflink-probe.*

# 4. Reflinks do not cross filesystems: FICLONE returns EXDEV. The template
#    image and the per-sandbox rootfs MUST live on the same mount or you get
#    the copy path with no error anywhere except your latency graph.
df --output=source,target /var/lib/pandastack/template-snaps \
                          /var/lib/pandastack/vms

# For the record, this is how we create the filesystem the agent runs on --
# a 300G XFS image, reflink on, loop-mounted, because the host's own root
# filesystem is usually ext4 and cannot do this.
truncate -s 300G /opt/pandastack.img
mkfs.xfs -m reflink=1 -m crc=1 -q /opt/pandastack.img
mount -o loop /opt/pandastack.img /var/lib/pandastack

So the clone is nearly free. The bill arrives on the first write. When a guest writes one byte into a block that is still shared with its parent, the kernel has to allocate a new block, read the old contents, apply the modification, and write it out. On our dm-snapshot fallback path the chunk size is 8 sectors — 4 KB, chosen to match the ext4 block size baked into the rootfs images — so a one-byte write is a 4 KB read-modify-write against the device. On XFS reflink, the unsharing happens at extent granularity through the filesystem's own CoW machinery, with the same underlying truth: the first write to shared data is real I/O.

One more thing the CoW model buys you, which people miss: the per-sandbox backing file starts at zero bytes on disk. We allocate a sparse 1 GB upper bound per sandbox and the blocks only materialise as the guest writes. Storage cost for a freshly forked sandbox is genuinely nothing until it does something. We had to make our own billing code aware of this — it counts allocated blocks rather than apparent size, because a reflinked snapshot reports multiple gigabytes of apparent size while physically occupying almost none of it, and charging for the apparent size would have billed customers for storage that does not exist.

The number that matters is IOPS under concurrency

Local NVMe and network block storage differ in two ways, and only one of them gets talked about. The famous difference is latency: a local NVMe read is tens of microseconds, a network block read is hundreds of microseconds to low milliseconds because there is an actual network in the path, plus a storage service on the far end doing replication. Roughly an order of magnitude, sometimes more, per operation.

The difference that actually kills fleets is the cap. A network block volume has a maximum IOPS rate, either provisioned or derived from its size, and your instance has a second, separate limit on aggregate storage bandwidth and operations across all its attached volumes. Both are enforced. A local NVMe device has no such thing — it has physics, and physics degrades gradually and predictably as you push queue depth. A provisioned cap does not degrade gradually. You are fine, you are fine, you are fine, and then every I/O in the system is queued behind a throttle that does not care that fifteen of those requests belong to a customer watching a spinner.

Think about what that means for a host running fifty sandboxes. The per-volume cap is not per-sandbox; it is the ceiling for everything on that volume, which means it is a fleet-wide ceiling that fifty tenants share. Density and storage headroom become the same variable. You can be at 40% CPU and 50% memory and still be unable to start another sandbox at a usable latency, because the constraint that binds is one you never put on a dashboard.

Measure the thing that will actually break

Almost every storage benchmark I see people run is a single sequential stream, which every device on earth passes. Run the test that resembles a fleet: small blocks, random offsets, many jobs, deep queues, and long enough to exhaust any burst allowance. And read the percentiles, not the mean — the mean is what your dashboard shows and the p99.9 is what your customer emails you about.

# The test that tells you nothing: one sequential stream. Everything passes.
fio --name=useless --rw=read --bs=1M --size=8G --numjobs=1 --iodepth=1 \
    --filename=/var/lib/pandastack/fio.probe

# The test that matters: many small random reads in flight at once. This is
# what N guests faulting in memory and rootfs blocks simultaneously looks
# like to the device.
fio --name=fleet-restore \
    --filename=/var/lib/pandastack/fio.probe --size=64G \
    --rw=randread --bs=4k \
    --ioengine=io_uring --direct=1 \
    --numjobs=16 --iodepth=32 \
    --time_based --runtime=1800 \
    --percentile_list=50:95:99:99.9:99.99 \
    --group_reporting

# The copy-on-write pattern: first touch of a shared block is a
# read-modify-write, so mixed traffic is the honest test for fork behaviour.
fio --name=cow-first-write \
    --filename=/var/lib/pandastack/fio.probe --size=64G \
    --rw=randrw --rwmixwrite=30 --bs=4k \
    --ioengine=io_uring --direct=1 \
    --numjobs=16 --iodepth=32 \
    --time_based --runtime=1800 \
    --percentile_list=50:95:99:99.9 \
    --group_reporting

# Three rules for reading the output:
#   1. --size must exceed host RAM, or you are benchmarking the page cache.
#   2. --runtime must be long (30+ min). A short run hides burst exhaustion,
#      which is exactly the failure you are trying to find.
#   3. Compare clat percentiles, not IOPS. A device that averages well and
#      has a 40ms p99.9 will produce user-visible stalls under a fleet.

That third rule is the one I'd tattoo on people. Two devices can report similar average IOPS and behave completely differently under a restore storm, because the fleet's user-visible latency is set by the tail. When fifty guests each issue a few hundred faults, some of those faults land in the tail by simple arithmetic, and a guest is not ready until its slowest blocking fault returns.

The 3pm failure mode

Burst-credit storage is the single most effective trap in cloud infrastructure, because it is designed to make evaluation succeed and steady state fail. The model: the volume accrues credits while idle and spends them when busy, so a fresh volume performs above its baseline for a while and then drops to baseline once the balance is gone. Every benchmark you run on day one is spending credits. Every capacity plan you build from those numbers describes a machine that does not exist during your busy hour.

The symptom is unmistakable once you've seen it. Boot times are fine all morning. Traffic ramps through the day, credits drain, and somewhere in the afternoon creates that took a couple hundred milliseconds start taking seconds, in a pattern that correlates with nothing in your application metrics. Overnight the balance refills and the morning looks perfect again, which is why the first three investigations conclude there's no problem. You cannot find this from inside the guest, because from inside the guest a slow disk is indistinguishable from a busy one.

• Check whether the volume type is credit-based at all — the newer provisioned-IOPS types are not, and moving off a burst type is usually a cheaper fix than any tuning you could do.
• Graph the credit balance as a first-class metric alongside CPU and memory. If your provider exposes it and you are not alerting on it, you have an outage scheduled for a date you don't know.
• Check the instance-level limit separately from the volume limit. Attaching more volumes does not help once the instance aggregate is the binding constraint, and this catches people who thought they had solved it by striping.
• Benchmark for at least thirty minutes on a volume that has been busy, not a freshly created one. A fresh volume with a full credit balance will happily tell you everything is fine.
• Watch the distribution, not the average. Throttling shows up as a fat tail long before it shows up as a lower mean.

Why a dense snapshot-restore fleet effectively requires local NVMe

Our architecture has no warm pool. There is no set of idle VMs absorbing create latency on our behalf, which is a deliberate choice — idle VMs cost money whether or not anyone uses them, and a warm pool is a bet on your own traffic forecast. The consequence is that every single create does real storage work: a clone, a snapshot load, and a run of page faults. There is nowhere for a slow device to hide.

What replaces the warm pool is the page cache and a content-addressed local cache. We ask the kernel to read the memory image ahead with POSIX_FADV_WILLNEED before we need it, so the restore is served from RAM rather than from the device on a warm host. For memory streamed from object storage, chunks land in a persistent per-seed cache on local disk, keyed by the seed's bucket and object so a re-bake self-invalidates rather than serving stale bytes. The comment I wrote above that cache still says it best: the per-host warm state lives in this content-addressed cache, not in idle VMs. That trade only works if local disk is fast enough to be a credible stand-in for memory. On network block storage it is not, and the entire design collapses back to needing idle VMs.

There's a second-order effect too. Page cache is shared across guests restoring from the same template, so the twentieth concurrent create of a template mostly hits RAM. But cache is finite and a busy host evicts. Every eviction turns a future restore back into device I/O, which means your worst case is always the device, and your worst case arrives exactly when the host is busiest — which is to say, when you can least afford it.

The honest counterpoint: local NVMe dies with the instance

Everything above argues for local NVMe. Local NVMe has one property that makes it unusable as a primary store: it is ephemeral. The instance goes away — autoscaler, maintenance event, health check, a MIG deciding to recreate the instance — and the disk goes with it. We have had that exact event. A GCP maintenance window caused instance recreation that wiped boot disks, and with them the locally-baked snapshots on those hosts. So the local disk cannot be truth.

The resolution is to be explicit about which artifact is cached and which is authoritative, and to never let those roles blur. For us: baked template seeds are published to object storage per generation with a CURRENT pointer and a SHA256 manifest, and that is truth. User snapshots and forks mirror to a bucket so any agent in the fleet can serve a restore rather than only the agent that took it. What lives on local NVMe is a cache of those things, plus the per-sandbox CoW layers that are ephemeral by definition. Agents pull seeds at boot; if a pull fails, the agent cold-bakes on first use and takes a roughly 3-second boot instead of 179ms. The comment on that function is the whole philosophy in one line: seeding is an optimization, never a correctness dependency.

Customer data does not live on that disk at all. Durable volumes and managed-Postgres data directories sit on a separate attached persistent disk that survives instance recreation, deliberately mounted on top of the fast local filesystem so an autoheal detaches and reattaches the same disk instead of taking the data with the host. Two disks, two jobs. The fast ephemeral one carries latency-critical, reconstructible state; the slower durable one carries anything a customer would be upset to lose. Confusing the two is how you end up with either a slow fleet or a data-loss incident, and the mistake is much easier to make than it sounds because both are just mount points.

# The client-side version of the same measurement. One create tells you
# nothing about a fleet -- fire a burst and read the tail.
#   pip install pandastack   &&   export PANDASTACK_API_KEY=...
import time
from concurrent.futures import ThreadPoolExecutor
from pandastack import Sandbox

def timed_create():
    t0 = time.perf_counter()
    sbx = Sandbox.create(template="base")
    created = (time.perf_counter() - t0) * 1000

    # Now dirty shared blocks on purpose: this is the copy-on-write cost
    # that a clone benchmark never shows you.
    t1 = time.perf_counter()
    sbx.exec("dd if=/dev/urandom of=/tmp/dirty bs=1M count=256 conv=fsync")
    dirtied = (time.perf_counter() - t1) * 1000

    sbx.kill()
    return created, dirtied

with ThreadPoolExecutor(max_workers=32) as pool:
    rows = list(pool.map(lambda _: timed_create(), range(64)))

creates = sorted(r[0] for r in rows)
writes = sorted(r[1] for r in rows)

def pct(xs, p):
    return xs[min(len(xs) - 1, int(len(xs) * p))]

print(f"create   p50={pct(creates,0.50):.0f}ms  p99={pct(creates,0.99):.0f}ms")
print(f"256MiB   p50={pct(writes,0.50):.0f}ms  p99={pct(writes,0.99):.0f}ms")

# Read the gap between p50 and p99, not p50. On a throttled volume the
# creates stay respectable and the write column explodes -- that is the
# signature of storage being the constraint rather than the platform.

What we got wrong

I want to be specific about a mistake here, because the tidy version of this argument is more confident than I have earned. Streaming memory from object storage on demand works well: page faults become 4 MiB range requests, absent chunks are zero-filled with no fetch at all, and a shared per-host cache means only the first restore of a template generation pays the network. That removed multi-gigabyte memory downloads from the create path and it has held up.

So we tried the same trick on the rootfs — demand-page the disk from object storage over a network block device, with copy-on-write layered on top. It corrupted guest filesystems in production. The mechanism was ugly: with the streaming path enabled, sync replaced the local disk image with a sparse zero placeholder and wrote a sidecar pointing at the remote object, and any code path that cloned the placeholder instead of using the streamed device handed a guest a disk full of zeros. That flag is pinned off across the fleet now, enforced through configuration management rather than first-boot provisioning, because a knob set only at first boot means a drifted host stays drifted forever.

The lesson I took: memory and disk are not symmetric, even though both are "bytes you can fetch in ranges." Guest memory is read through a fault handler we control completely, where a slow or failed fetch is a retry. A rootfs is read by an entire guest kernel through a block layer with its own caching, ordering and error semantics, and a byte that arrives wrong is not a stall, it is corruption. Which is a long way of saying that the local disk stays load-bearing for the rootfs, and I no longer expect to design that away.

What to ask before you commit a fleet

1. What device backs the host's working directory, and is it local or network-attached? If a platform cannot answer this in one sentence, the answer is network-attached.
2. Is the filesystem reflink-capable, and are the template image and the per-sandbox disks on the same mount? A cross-mount clone falls back to a full copy and the only evidence is a latency graph.
3. What happens on the first write after a fork? Ask about chunk or extent granularity, because that number times the child's dirty-block count is the real fork cost.
4. Is the volume credit-based, and is the credit balance alerted on? If the answer is "I'd have to check," check today rather than at 3pm on a busy Thursday.
5. What is the instance-level storage limit, separate from the per-volume one? This is the cap that survives your attempt to fix things by adding volumes.
6. Which artifacts are cached on the ephemeral disk and which are authoritative in object storage? If nobody can draw that line, an instance recreation will draw it for them.

The summary

Storage hardware decides sandbox performance more than most of the software above it, and it does so in a way that is nearly invisible during evaluation. A reflink clone is metadata-only and costs about a millisecond regardless of image size — but only on a reflink-capable filesystem, only within one mount, and only until the guest writes. Every first write to a shared block is a read-modify-write against a real device, so fork cost tracks the write pattern rather than the clone.

Under concurrency the device's tail latency becomes your product's tail latency, and network block storage adds both a network hop per operation and a hard cap that fifty tenants on a host share between them. Local NVMe removes the hop and the cap, at the price of being ephemeral — so it has to be treated as a cache, with object storage as the truth it can be rebuilt from, and with anything a customer would miss living on a durable disk instead.

If you take one action from this: run fio with small random blocks, sixteen jobs, queue depth 32, for thirty minutes, on a volume that has already been busy, and look at p99.9 rather than the average. That single run tells you more about how a sandbox fleet will behave in month three than any feature comparison will.