What "persistent" actually means in a sandbox

Someone opens a support thread roughly once a month with a variant of the same sentence: "the file was definitely there, I ran cat and saw it." And they did see it. The file was there. It was there in a place that a power cut erases, which is a fact `cat` is structurally incapable of telling you, because `cat` reads back through exactly the same cache the write landed in.

This trips up good engineers because the mental model is not wrong so much as truncated. Everybody knows there is a buffer somewhere. Almost nobody has counted how many buffers, or knows which of them their platform actually honours a flush through. In a sandbox there are more layers than on a laptop, because a virtual machine adds a whole second operating system's worth of caching underneath the first one.

So this is the walk down the stack: every place a write can be sitting while your program believes it succeeded, what erases each one, and then the part that matters more than any of it — that on a throwaway rootfs, durability is the wrong goal entirely.

Six places your write can be

Trace a single `f.write(b)` inside a Python process running in a microVM. Here is where those bytes live, in order, and what makes them disappear.

1. The language runtime's buffer

Your call returned, and the bytes are in a userspace array owned by your process. No system call has happened. The kernel does not know this data exists. If the process segfaults here, or someone sends it SIGKILL, or the interpreter exits without running your `finally` block, the write never happened at all — and nothing outside the process ever had a chance to save it.

This layer is the one people accidentally rely on most, because buffering is invisible and size-dependent. A small write sits in the buffer; a large one spills through to the kernel because the buffer filled up. That is why "it works with the big file and loses the small one" is such a common and such a maddening bug report.

2. The guest page cache

`flush()` gets you here: a real `write(2)`, which returns success. The bytes are now dirty pages in the guest kernel's page cache. This is the layer where `cat` starts lying to you, and it is not really lying — reads are served from the page cache, so a read-after-write is genuinely consistent. It just tells you nothing about disks. Every process in the guest agrees the file has that content. The guest kernel will get around to writing it out within a few seconds, or when a writeback threshold trips, or when you ask.

Kill the process now and the data survives, because the kernel owns it. Kill the VM now and it is gone.

3. The guest block layer and the virtio queue

`fsync()` asks the guest kernel to push those dirty pages down through its filesystem and block layer and not return until they are handed off. Inside a microVM the "disk" underneath is a virtio-blk device: the guest queues descriptors into a shared ring, the VMM picks them up, and a FLUSH request is a distinct thing the guest can put on that ring to say "I need this durable, tell me when it is."

This is the hinge of the whole post. Everything above this line is the guest doing its job correctly. Everything below is the hypervisor deciding whether to take that FLUSH seriously.

4. The host page cache

The VMM services the write by writing into a file on the host — for us, an ext4 image sitting on the host's filesystem. So the guest's carefully-fsynced bytes land in the host kernel's page cache, which is a completely separate set of dirty pages with a completely separate writeback schedule, one level of abstraction that the guest cannot see and cannot reach.

Whether the guest's FLUSH turns into a real `fsync()` on that host file is a configuration choice made by the hypervisor. In Firecracker it is the drive's `cache_type`, and the default is `Unsafe`, which does not pass the flush through. Under that default, a guest `fsync()` returns success while the data is still only in the host page cache. The guest's durability promise is intact and meaningless at the same time.

5. The host disk — and 6. whatever is under it

Host writeback eventually issues the I/O, and the drive acknowledges it. If the drive has a volatile write cache and the stack does not issue a cache-flush command, the acknowledgement is again about a buffer. On cloud block storage there is a network and a replication protocol under that, with its own definition of committed. You reach real durability only when every layer honours the barrier, and a single layer that quietly does not is enough to make all the ones above it decorative.

The three calls, and the fourth one everybody forgets

Here is the same layering as code. Nothing exotic; the interesting part is the last block, which is the one that gets skipped in about nine out of ten codebases I have read.

import os

payload = b'{"status": "done"}'

# Layer 1 -> 2: hand the bytes to the kernel. write(2) returns success.
# They are now DIRTY PAGES in the guest page cache. `cat` will show them.
# A crash of the guest loses them; a crash of your process does not.
f = open("/data/results.json", "wb")
f.write(payload)
f.flush()          # flushes the RUNTIME buffer, not the kernel's

# Layer 2 -> 3+: push them down the block layer and wait. This is the only
# call in the file that is about durability rather than visibility.
os.fsync(f.fileno())
f.close()

# The one everybody forgets: a file's NAME lives in its DIRECTORY, and a
# directory is a file with its own dirty pages. fsync on the data does not
# make the directory entry durable. Crash in between and you can come back
# to a file that exists with the right bytes and no name, or a name with
# no file. Both are real, both are confusing at 3am.
dfd = os.open("/data", os.O_DIRECTORY | os.O_RDONLY)
os.fsync(dfd)
os.close(dfd)

The pattern that actually gives you an all-or-nothing update is write-to-temp, fsync the temp, rename over the target, fsync the directory. Rename is atomic within a filesystem, so a reader sees either the whole old file or the whole new one and never a half-written mixture. We use exactly that shape in our own WAL relay: the host agent streams each segment to a `.tmp` file, `fsync`s it, renames it into place, and only then returns 201 to the guest. Postgres treats anything other than a 2xx as "not archived" and retries, so returning 201 before the bytes were durable would be us lying to a database about its own write-ahead log.

# Atomic replace. Four steps, in this order, no shortcuts.
import os, tempfile

def atomic_write(path, data):
    d = os.path.dirname(path)
    fd, tmp = tempfile.mkstemp(dir=d)        # same filesystem, so rename is atomic
    try:
        with os.fdopen(fd, "wb") as f:
            f.write(data)
            f.flush()
            os.fsync(f.fileno())             # 1. the DATA is durable
        os.replace(tmp, path)                # 2. the NAME flips atomically
        dfd = os.open(d, os.O_DIRECTORY | os.O_RDONLY)
        try:
            os.fsync(dfd)                    # 3. the RENAME is durable
        finally:
            os.close(dfd)
    except BaseException:
        try: os.unlink(tmp)                  # 4. never leave a half file behind
        except FileNotFoundError: pass
        raise

What the hypervisor does with your flush

Firecracker's block device has a `cache_type` field with two values, and it decides whether layer 3 connects to layer 4 or dead-ends into it. `Unsafe` is the default and does not honour guest flush requests. `Writeback` does: a guest `fsync()` becomes a real host `fsync()`, and costs you write throughput to do it.

The failure mode of getting this wrong is nastier than "you lose the last few seconds". A filesystem's crash recovery is built on ordering — the journal records intent before the change lands, so replay can finish or undo it. If the layer beneath is free to reorder and to acknowledge flushes it did not perform, the ordering guarantee evaporates and the image can come back inconsistent rather than merely stale. Losing recent writes is an incident. Coming back with a corrupt filesystem is a much longer week.

We set this per drive, on purpose, in opposite directions:

• The rootfs stays Unsafe. It is a copy-on-write clone that gets deleted with the sandbox, an app rebuilds from git, and anything that matters lives elsewhere. Paying a real host fsync on every guest write there would slow down every workload to protect data that is by definition throwaway.
• Durable volumes get Writeback. A volume in our system IS the durable store — managed Postgres keeps its PGDATA on one — so a guest fsync there has to mean what it says. The throughput cost is the correct trade for storage whose entire purpose is to survive.
• The opt-out is spelled PANDASTACK_VOLUME_CACHE_UNSAFE=1 and only the exact string "1" turns it off. There is a test asserting that a typo, an empty value, or the word "true" does not silently disable durability, because that is the direction where being permissive costs somebody their data.

On an ephemeral rootfs, durability is the wrong question

Here is where I want to change the framing rather than answer the question as asked. People arrive wanting to know how to make writes durable inside a sandbox. The rootfs of a sandbox is a reflinked copy-on-write clone of a template image, created in about four milliseconds and removed when the sandbox is torn down. Its whole design intent is that it is disposable — no residue between tenants, no cleanup script that might miss something, no per-sandbox storage bill for a directory nobody will read again.

Asking how to make that disk durable is like asking how to make a scratch buffer permanent. You can succeed at the local goal — a perfectly fsynced file, guaranteed on the host's disk — and still lose the data thirty seconds later, because the sandbox hit its TTL and the entire image was deleted. `fsync()` protects against a crash. It does not protect against deletion, and deletion is the thing that is actually scheduled to happen to that disk.

fsync answers "will this survive a power cut?" On an ephemeral rootfs the question you actually have is "will this survive being thrown away on purpose?", and there is no system call for that.

So the real question is a routing question: which store should this piece of state live in? There are three destinations and they are genuinely different, not three tiers of the same thing.

The ephemeral rootfs: working space

Source checkouts, node_modules, intermediate build output, the model's scratch files, the half-finished CSV. Fast, free, isolated, and gone. Do not fsync here; you are only slowing yourself down to protect data whose life expectancy is measured against a TTL, not against a hardware failure. The correct discipline is to know, before the sandbox starts, which single artifact has to leave.

The durable volume: a disk that outlives the machine

A volume is a named ext4 image owned by your workspace, stored on the host outside the sandbox's directory, attached at create time and appearing in the guest as /dev/vdb, then /dev/vdc. You mount it yourself, which surprises people who expect it at a fixed path — that is deliberate, because you choose where it belongs in your filesystem. It survives the sandbox being deleted, it is billed on provisioned size rather than bytes written, and it is the drive that runs in Writeback mode, so an `fsync()` on a volume is a real one.

The honest limitation: a volume is a file on one host's disk. It outlives the sandbox, not the host. That makes it right for a warm cache you would hate to rebuild, a model checkpoint, an agent's accumulated notes — and wrong as the only copy of anything you would be upset to lose.

The managed database: durability as a product, not a flag

This is what the word "durable" is actually supposed to mean, and it is a stack rather than a setting. Postgres keeps PGDATA on a volume drive in Writeback mode, so its own WAL fsync is a real fsync. Its `archive_command` ships each completed WAL segment out of the guest — guests hold no cloud credentials, so segments are POSTed over the host-internal veth to a relay on the agent, which spools them to disk with an fsync and a rename before returning 201. `archive_timeout` is 60 seconds, so a quiet database still rolls a segment every minute rather than holding recent commits hostage until 16 MiB accumulate. A daily `pg_basebackup` streams a full tarball out the same way. Base backup plus archived WAL is what makes a rebuild on a different host possible at all.

Notice what that buys and what it does not. It buys off-host durability and point-in-time recovery. It does not make a single commit's acknowledgement wait for an object store — the WAL segment for your most recent transaction is durable on the host, and off-host within the archive interval. Every durability story has a number like that in it. A vendor who cannot tell you theirs has not measured it.

What this looks like in code

The pattern for the common case is unglamorous, which is the point: compute in the disposable thing, then move the one artifact that matters into a store whose job is keeping it.

import os
from pandastack import Sandbox   # pip install pandastack; PANDASTACK_API_KEY in env

JOB = """
import json, os, pathlib
pathlib.Path("/work/out").mkdir(parents=True, exist_ok=True)
# No fsync in here. This disk is going to be deleted on purpose in 600s,
# and fsync has nothing to say about deletion.
with open("/work/out/report.json", "w") as f:
    json.dump({"rows": 41892, "status": "ok"}, f)
"""

sbx = Sandbox.create(template="code-interpreter", ttl_seconds=600)
try:
    sbx.filesystem.write("/work/job.py", JOB)
    r = sbx.exec("cd /work && python3 job.py")
    if r.exit_code != 0:
        raise RuntimeError(r.stderr)

    # THIS is the durability step. Everything before it is scratch work.
    blob = sbx.filesystem.read("/work/out/report.json")
finally:
    sbx.kill()

# Land it somewhere whose entire product is not losing things.
s3.put_object(Bucket="reports", Key="2026-09-08/report.json", Body=blob)

When the state is too big to shuttle out on every run, or it is a cache rather than an artifact, attach a volume instead — and here the fsync discipline is worth having, because this drive is not scheduled for deletion.

from pandastack import Client, Sandbox

client = Client()
client.volumes.create(name="agent-notes", size_mb=4096)

sbx = Sandbox.create(
    template="agent",
    volumes=[{"name": "agent-notes"}],   # first extra volume -> /dev/vdb
)

# Volumes attach as raw block devices; YOU choose the mount point. First use
# needs a filesystem on it -- after that, just mount.
sbx.exec("mkdir -p /mnt/notes && (mount /dev/vdb /mnt/notes || "
         "(mkfs.ext4 -F /dev/vdb && mount /dev/vdb /mnt/notes))")

# On this drive the guest's flush IS honoured on the host (Writeback), so a
# sync here is a real durability barrier rather than a formality.
sbx.exec("python3 -c \"import os;f=open('/mnt/notes/state.json','w');"
         "f.write('{}');f.flush();os.fsync(f.fileno());f.close()\"")
sbx.exec("sync")
sbx.kill()   # the sandbox dies; /mnt/notes' contents do not

Snapshots are a different durability model, and it confuses people in both directions

Now the part that inverts the whole discussion. A microVM snapshot captures guest RAM verbatim. Guest RAM includes the guest page cache. So the dirty pages you never flushed — the ones a crash would have erased — are captured by a snapshot and come back on restore, still dirty, still unflushed, exactly as they were.

That surprises people in both directions at once. Coming from the crash-safety world, it is surprising that unflushed data survives: a snapshot is not a power cut, it is a pause, and a pause preserves the volatile state that a power cut destroys. Coming from the backup world, it is surprising in the other direction: a snapshot is not a backup. It is one file on one host (mirrored to object storage when you have opted into cross-region replication), and it captures a moment, not a history. You cannot recover to five minutes before it.

The subtlety underneath is that memory and disk must be captured at the same instant or the pair is inconsistent. Our snapshot path pauses the VM, captures the memory and device state, and reflinks the rootfs inside the same pause window. It did not always. For a while, restores ran the snapshot's memory image over a fresh template disk, so every file written after template bake vanished on restore while the guest's page cache still cheerfully believed those files existed. That is a very confusing bug to be on the receiving end of, and an E2E run found it rather than a customer, which is the only reason I am comfortable writing it down.

The rules I actually apply

1. Decide the destination before you write the code. "Which store does this belong in?" is answerable in five seconds and prevents the whole class of problem. "How do I make my rootfs durable?" is not really answerable at all.
2. On an ephemeral rootfs, do not fsync. You are paying for a guarantee against the wrong failure. The failure that will actually happen is a scheduled deletion.
3. For anything that must survive, get it out during the run, not at the end. A sandbox that dies at 90% through a job took your artifact with it; one that streamed results out as it went did not.
4. On a durable volume, fsync properly: data, then rename, then the directory. And verify that the platform passes the flush through — ask which cache mode the drive runs in, and be suspicious of a vendor who does not know.
5. Treat a snapshot as a fast-start artifact, not as a backup. It restores a machine; it does not give you a history to recover into.
6. If it is relational state you would be genuinely upset to lose, put it in a database that does WAL archiving off-host, and learn the archive interval. That number is your worst-case data loss window, and it exists whether or not you know it.

The summary

A write passes through a userspace buffer, the guest page cache, the guest block layer, a virtio queue, the host page cache, and finally a disk with its own opinions about caching. `write()` gets you through the first hop. `fsync()` gets you the rest of the way only if every layer beneath honours the barrier — and the hypervisor's default in Firecracker is not to.

Which is fine, because on a sandbox rootfs the answer was never fsync. That disk is meant to be thrown away, and "persistent" is not a property you can add to it. Pick the destination that matches the state: object storage for artifacts, a durable volume for caches and checkpoints on a drive that honours flushes, a managed database with off-host WAL archiving for anything relational and important.

The good news is that this is a five-second decision made once, at design time, rather than a subtle bug found at 3am. The bad news is that if you skip it, `cat` will keep telling you everything is fine right up until the moment it is not.