Sparse Files and Hole Punching: Why Your Snapshot Lies About Its Size

A Firecracker memory snapshot for a 4 GiB guest is a 4 GiB file. Ask ls and it will tell you 4294967296 bytes, every time, with total confidence. Ask du about the same file and you may get a number that is a small fraction of that. Neither tool is lying. They are answering two different questions — 'how long is this file?' versus 'how much disk is this file consuming?' — and on a memory snapshot those two questions have wildly different answers, because most of a freshly booted guest's RAM is zeros that were never written down. The gap between the two numbers is one of the most useful properties of snapshot storage: it is why you can keep a lot of templates on a modest disk. It is also a fact that will betray you the first time it crosses a tool that does not understand it, and the failure mode is always the same shape — your storage bill, your transfer time, or your capacity dashboard suddenly reflects the apparent size instead of the real one. This post is the mechanics: what a hole is, how holes get into a memory file, how to see them, how to put them back, and the five ways sparseness quietly stops being true.

Two sizes: apparent and allocated

Every regular file on Linux carries two independent size numbers in its inode. st_size is the apparent size — the logical length of the file, the offset one past the last byte you could read. That is what ls -l prints and what stat calls Size. st_blocks is the allocated size — the number of 512-byte units the filesystem has actually assigned to this file. That is what du reports (after multiplying by 512 and rounding to whatever unit you asked for) and what stat calls Blocks. For an ordinary file the two track each other: write 100 MiB of real data and the filesystem allocates roughly 100 MiB of blocks, plus a little metadata. A sparse file is one where they diverge, because some byte ranges within the file's logical length have no blocks backing them at all. Those ranges are holes. They are not zeros stored on disk; they are the absence of storage, plus a promise from the filesystem that if you read there, you get zeros.

# The same file, two honest answers. Sizes below are illustrative -- run this
# against your own snapshots, the ratio depends entirely on the workload.

$ ls -l /var/lib/pandastack/seeds/base/vm.mem
-rw-r--r-- 1 root root 4294967296 Aug 26 09:14 vm.mem
#                       ^ apparent size: 4 GiB, always, for a 4 GiB guest

$ du -h --apparent-size /var/lib/pandastack/seeds/base/vm.mem
4.0G    vm.mem          # st_size again -- what the guest thinks its RAM is

$ du -h /var/lib/pandastack/seeds/base/vm.mem
<less>  vm.mem          # st_blocks * 512 -- what the filesystem is paying for

$ stat /var/lib/pandastack/seeds/base/vm.mem
  File: vm.mem
  Size: 4294967296   Blocks: <N>    IO Block: 4096   regular file
#       ^ apparent bytes    ^ 512-byte units actually allocated
#                                    ^ the filesystem block size: hole granularity

$ ls -ls /var/lib/pandastack/seeds/base/vm.mem
#  ^ the leading column is allocated 1K blocks, not apparent size --
#    the one place ls will tell you the truth about disk usage

# Where the bytes actually live, extent by extent. Gaps between logical
# offsets are holes; look for the 'unwritten' flag too (see below).
$ filefrag -v /var/lib/pandastack/seeds/base/vm.mem | head -20

Why is a memory file sparse in the first place? Because guest RAM mostly is not used. A guest with 4 GiB of RAM that has booted a kernel, started an init system and a couple of services has touched a modest slice of its address space; the rest of guest-physical memory has never been written by anything and reads back as zeros. When that address space is serialized to a file, those never-touched regions are long runs of zero bytes — and long runs of zero bytes are exactly what a hole represents for free.

How the holes get there

There are two distinct mechanisms, and it is worth keeping them separate because they apply at different times. The first is skipping. If you seek past a region and write beyond it, the filesystem never allocates blocks for the skipped range — that range becomes a hole, and st_size grows to cover it. This is the classic way to create a sparse file: open, lseek to 4 GiB, write one byte, close. You now have a 4 GiB file occupying essentially nothing. A writer that knows a region is all zeros can simply not write it, and the hole appears by omission. Whether a given snapshot's memory dump does this depends on how it writes the region: a full dump that streams every byte of guest RAM will allocate blocks for those bytes unless something downstream elides them, while a differential snapshot that writes only the pages dirtied since the base naturally leaves everything else as holes. Check the behaviour of your Firecracker version against its documentation rather than assuming — and then just measure the resulting file, which takes one du.

The second is punching, and this is the interesting one because it works after the fact on a file that is already fully allocated. fallocate(2) with the flags FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE takes an offset and a length in an existing file and deallocates the blocks in that range. The blocks go back to the filesystem's free pool. The file's apparent size does not change — that is what KEEP_SIZE means, and the kernel requires it to be set alongside PUNCH_HOLE. Reads of the punched range afterwards return zeros. So you can write out a memory file the simple way, scan it for block-aligned runs of zeros, punch them out, and end up with a sparse file that is byte-for-byte identical on read to the dense one you started with. The read side is the payoff and deserves being said plainly: reading a hole costs no device I/O. There is nothing on the platter or in the NAND to fetch. The filesystem sees a logical offset with no extent mapping, and the kernel hands your buffer zeros directly. For a memory-mapped snapshot this is even better — a fault on a page inside a hole resolves to the kernel's shared zero page or a freshly zeroed anonymous page, without a storage round trip. Zeros are the one kind of data you never have to move.

Filesystem support, and the alignment trap

Sparseness is a filesystem feature, not a universal one, and hole punching is a separate capability from merely storing sparse files. The mainstream Linux filesystems you would actually run a microVM host on handle both, but the details differ enough to matter:

• ext4 — sparse files and FALLOC_FL_PUNCH_HOLE are supported. Standard 4 KiB blocks on typical configurations.
• XFS — sparse files and punch hole supported; XFS is also where reflink-based copy-on-write cloning lives, which interacts with sparseness in ways covered further down.
• btrfs — sparse files and punch hole supported, with its own compression and reflink layers on top that change what 'allocated' even means.
• tmpfs — supports punch hole, which is how you release page-cache-backed pages of a file living in RAM.
• Network and pseudo filesystems — assume nothing. NFS behaviour depends on version and server; various FUSE and overlay layers pass punch requests through, emulate them, or return EOPNOTSUPP. Always handle the failure, and verify against the man page for the stack you are actually on.

The trap underneath all of this is granularity. A hole is the absence of an allocated filesystem block, so the smallest hole you can have is one block. On a 4 KiB-block filesystem, punching a 1 KiB run of zeros inside a 4 KiB block deallocates nothing — the kernel will accept the call and simply zero those bytes in place, because it cannot free a fraction of a block. Alignment matters just as much: a punch request from offset 3 KiB to 9 KiB can only free the one whole block in the middle; the partial blocks at either end get zeroed, not deallocated.

One more subtlety that will make you distrust du: a range can be allocated but unwritten. fallocate without PUNCH_HOLE preallocates blocks — it reserves extents and marks them unwritten, so reads return zeros but the space is committed and counted. Those extents show up in du and in filefrag (flagged unwritten on ext4/XFS) and look, from the outside, exactly like real data. Preallocated-but-unwritten is the opposite of a hole: zeros you are paying for.

Betrayal 1: the tool that reads zeros and writes zeros

This is the big one, and it is entirely mundane. A hole is invisible through the ordinary read path. If a program opens the file and reads it start to finish, it sees zeros — real, ordinary zero bytes, indistinguishable from zeros that were genuinely stored. If that program then writes what it read to a new file, it writes those zeros, and the new file is fully allocated. Your sparse snapshot has been reconstituted at full apparent size, and nothing warned you.

• cp — Preserves holes: usually, on a local copy. GNU cp defaults to --sparse=auto, a deliberately crude heuristic on the source file. Gotcha: it is a heuristic, and it is GNU-specific. Pass --sparse=always when you care, and do not assume a non-GNU cp behaves the same; check your coreutils man page.
• cp --reflink=always — Preserves holes: yes, and shares the allocated extents too. Gotcha: requires a reflink-capable filesystem (XFS, btrfs) and both files on the same filesystem; the clone's du now double-counts shared extents.
• tar — Preserves holes: only with -S / --sparse. Gotcha: without it, every zero byte is faithfully written into the archive, so your tarball is apparent-size before compression. GNU tar and bsdtar differ in detection and encoding — verify the flag on the tar you actually ship with.
• rsync — Preserves holes: with -S / --sparse. Gotcha: historically --sparse conflicted with --inplace, and combinations with delta transfer have changed across versions; read the man page for your rsync before relying on it in a sync loop.
• dd — Preserves holes: with conv=sparse. Gotcha: it detects zero-filled output blocks and seeks over them; a partially zero block still gets written. Pick a bs that matches or exceeds the filesystem block size.
• scp — Preserves holes: no. Gotcha: it streams the file as bytes over the wire, so you transfer and store every zero. There is no sparse mode to turn on.
• cat / shell redirection / a naive read-write loop in any language — Preserves holes: no. Gotcha: this is the default in every language's 'copy a file' snippet. If your uploader is a for-loop over 1 MiB reads, it is inflating.
• A container image layer or an OCI push — Preserves holes: no, in general. Gotcha: layer tarballs go through the tar path above and then get compressed; compression hides the cost on the wire but the extraction on the other end is fully allocated.

The practical rule I have landed on: treat sparseness as a property that survives only where you have explicitly arranged for it to survive. Every hop — copy, archive, upload, download, extract, restore — is a place it can be lost, and losing it is silent. If a snapshot pipeline matters, put a du check at both ends and alert on the ratio changing.

Betrayal 2: object storage has no holes

S3, GCS, and every other object store deal in opaque byte sequences. An object of length N is N bytes; there is no metadata anywhere in the API that says 'bytes 1 GiB through 3 GiB of this object are zeros, do not bother storing them.' The moment your sparse 4 GiB memory file becomes an object, it is a 4 GiB object. You are billed for 4 GiB of storage, you pay 4 GiB of egress every time it is fetched, and the upload takes as long as 4 GiB takes. You have three honest ways out, and they compose:

1. Compress. Zeros compress to essentially nothing, so a gzip or zstd stream of a sparse-shaped file is small on the wire and in the bucket. The cost is that you must decompress the whole thing to use any of it — which kills random access, and random access is exactly what demand-paged restore needs.
2. Chunk plus an index. Split the file into fixed-size chunks, record which chunks are entirely zero, and upload only the non-zero ones. The index is tiny. A reader that wants a zero chunk does not fetch anything; it fills zeros locally. This keeps random access, which is the point.
3. Do not move it at all. Reflink-clone locally where you can, and only publish to object storage on the paths that genuinely cross hosts.

Option 2 is what we run on PandaStack. A baked memory snapshot ships with a small sidecar header that records which fixed-size chunks of the memory image are non-zero. On restore, the userfaultfd handler serving guest page faults consults that index first: a fault landing in a chunk the index marks absent is satisfied by installing a zero page directly — no HTTP Range GET, no network, no bytes billed. Only chunks that actually contain data are ever fetched. The sparseness of the file, which object storage threw away, is reconstructed as one bit per chunk in a sidecar and used at exactly the moment it pays.

Betrayal 3: du says one thing, df says another

Sooner or later someone walks a snapshot directory with du, sums it, compares it to df, and files a bug about missing disk. There are several legitimate reasons the two disagree and they push in opposite directions, which is what makes the debugging confusing.

• du sums per-file allocated blocks; df asks the filesystem how much space is free. Filesystem metadata, journals, and reserved-for-root blocks are counted by df and not by du, so df-used normally exceeds the du sum a little.
• Deleted-but-open files are gone from du's walk (no directory entry) but still hold their blocks until the last file descriptor closes. A leaked handle on an old snapshot is invisible to du and very visible to df. lsof +L1 finds them.
• Reflinked clones make the du sum exceed real usage, because shared extents are counted once per file — see the next section.
• du deduplicates hard links within a single invocation but has no idea about links outside the tree it walked, so walking two subtrees separately can double-count.
• Mounts hidden underneath other mounts, and bind mounts, will be counted twice or not at all depending on which flags you passed.

The monitoring failure this produces is specific and worth naming. If your capacity alert is built on summing file sizes — apparent sizes, from a directory listing or an object inventory — it will fire long before the disk is anywhere near full, and everyone will learn to ignore it. If it is built on du, it will miss deleted-but-open files entirely and let you hit ENOSPC with a green dashboard. Alert on df for 'am I about to run out', and use du and apparent size as diagnostics after the alert fires, not as the alert.

Betrayal 4: reflinks make per-file accounting meaningless

Copy-on-write cloning is the other half of a snapshot platform. On XFS and btrfs, cp --reflink creates a second file that points at the same physical extents as the first; nothing is copied, and divergence happens per-block on write. That is what makes cloning a multi-gigabyte rootfs an O(metadata) operation instead of an O(bytes) one. The accounting consequence is that st_blocks is a per-file view of a fundamentally shared world. Both the original and the clone report the full allocated size, because both really do reference those blocks — but the blocks exist once. Sum du across a directory of reflinked clones and you will get a number that can exceed the capacity of the disk they are sitting on, which is a good sign your methodology is wrong rather than a good sign about your compression ratio. Combine this with sparseness — where the same tools also undercount relative to apparent size — and per-file arithmetic tells you nothing reliable in either direction.

The fix is to stop asking files and start asking the filesystem. df gives you the ground truth for 'how much room is left'. On btrfs, btrfs filesystem du distinguishes exclusive from shared bytes per file, which is the number you actually want when deciding what deleting something would recover. On XFS, the honest approach is df plus knowing your clone topology. And when you delete one of two reflinked clones, you free only the extents that clone had diverged into — the shared ones stay, because the sibling still needs them.

Betrayal 5 (which is actually a gift): restore and streaming

Everything above frames holes as something you lose. On the restore path the polarity flips: knowing where the holes are is precisely the information that makes demand-paged restore cheap, and losing it makes it expensive. Consider a restore that pages guest memory in on demand — whether from a local file through mmap and ordinary page faults, or across the network through a userfaultfd handler pulling ranges from object storage. Every fault has to be answered with 4 KiB (or 2 MiB, on a hugepage-backed guest) of content. If the handler knows the faulting offset falls in a region that is all zeros, answering is free: install a zero page and return. If it does not know, it has to go get the bytes — a read, or worse, an HTTP round trip — to discover that they were zeros all along. Same result, orders of magnitude more expensive, repeated for every zero page the guest happens to touch.

A large share of the faults in a freshly restored guest are zero-fills, because that is what an operating system does with fresh memory: the allocator hands out a page and the kernel zeroes it. So the zero-region index is not a marginal optimisation, it is most of the traffic. Measure the split on your own snapshots — instrument your handler to count zero-fill faults versus data faults and the ratio will tell you where your restore time is actually going.

Measuring it: SEEK_DATA and SEEK_HOLE

du and stat give you a scalar. To find out where the holes are, lseek(2) has two specialised whence values that most people never touch. SEEK_DATA moves the file offset to the start of the next range containing data, at or after the given offset; if there is no data left, it fails with ENXIO. SEEK_HOLE moves to the start of the next hole, at or after the offset — and end-of-file always counts as a hole, so it always terminates. Alternating between them walks the file's allocated extents exactly. One caveat on the semantics, because it bites people writing backup tools: the filesystem is permitted to be conservative. It may report a range as data when parts of it are zeros — a preallocated-but-unwritten extent, say, or a region it has not bothered to analyse. It is not permitted to report real data as a hole. So this is safe to use for 'what can I skip copying', and is not a reliable oracle for 'what is definitely zero'. If you need the latter, read the bytes and check them.

package main

import (
	"errors"
	"fmt"
	"os"

	"golang.org/x/sys/unix"
)

// extents enumerates the [start, end) byte ranges of a file that are backed by
// data, using lseek(2) with SEEK_DATA / SEEK_HOLE. Everything not returned is a
// hole: it reads back as zeros and costs no device I/O.
//
// Note the filesystem is allowed to be conservative here -- it may report a
// range as data that happens to contain zeros. It must never report data as a
// hole, which is the direction that matters for correctness.
func extents(f *os.File) ([][2]int64, error) {
	fd := int(f.Fd())
	fi, err := f.Stat()
	if err != nil {
		return nil, err
	}
	size := fi.Size()

	var out [][2]int64
	var off int64
	for off < size {
		start, err := unix.Seek(fd, off, unix.SEEK_DATA)
		if errors.Is(err, unix.ENXIO) {
			break // no data at or after off: the rest of the file is a hole
		}
		if err != nil {
			return nil, fmt.Errorf("SEEK_DATA at %d: %w", off, err)
		}
		end, err := unix.Seek(fd, start, unix.SEEK_HOLE)
		if err != nil {
			return nil, fmt.Errorf("SEEK_HOLE at %d: %w", start, err)
		}
		out = append(out, [2]int64{start, end})
		off = end
	}
	return out, nil
}

func main() {
	f, err := os.Open("/var/lib/pandastack/seeds/base/vm.mem")
	if err != nil {
		panic(err)
	}
	defer f.Close()

	ex, err := extents(f)
	if err != nil {
		panic(err)
	}

	var data int64
	for _, e := range ex {
		data += e[1] - e[0]
	}
	fi, _ := f.Stat()
	fmt.Printf("apparent=%d bytes  extents=%d  data=%d bytes  holes=%d bytes\n",
		fi.Size(), len(ex), data, fi.Size()-data)
}

Putting the holes back

When a file has been inflated — it came off an object store, or through scp, or out of a tar without -S — you can re-sparsify it in place. Scan it in filesystem-block units, find the block-aligned runs that are entirely zero, and punch them. This is exactly the operation fstrim performs at the block layer and that qemu-img and virt-sparsify do for disk images; there is nothing exotic about it, and it is short enough to write yourself when you want it inside an existing pipeline.

#define _GNU_SOURCE
#include <fcntl.h>
#include <linux/falloc.h>   /* FALLOC_FL_PUNCH_HOLE, FALLOC_FL_KEEP_SIZE */
#include <stdlib.h>
#include <sys/stat.h>
#include <unistd.h>

static int all_zero(const unsigned char *p, size_t n)
{
    for (size_t i = 0; i < n; i++)
        if (p[i])
            return 0;
    return 1;
}

/*
 * Deallocate every block-aligned run of zeros in fd, in place.
 *
 * FALLOC_FL_PUNCH_HOLE frees the blocks; FALLOC_FL_KEEP_SIZE is mandatory
 * alongside it and keeps st_size unchanged, so the file keeps its apparent
 * length and simply stops paying for the zeros. Reads of a punched range
 * return zeros with no device I/O.
 *
 * blk MUST be the filesystem block size (stat's st_blksize / statvfs's
 * f_bsize) or a multiple of it. Punching a sub-block run frees nothing --
 * the kernel just zeroes those bytes in place, and you will happily report
 * having reclaimed gigabytes while df does not move.
 *
 * Returns 0 on success, -1 with errno set. EOPNOTSUPP means this filesystem
 * cannot punch holes; that is a normal answer, not a bug -- handle it.
 */
int resparsify(int fd, off_t size, size_t blk)
{
    unsigned char *buf = malloc(blk);
    if (!buf)
        return -1;

    off_t run = -1;   /* start of the current all-zero run, or -1 for none */
    int rc = 0;

    for (off_t off = 0; off < size; off += (off_t)blk) {
        size_t want = (size - off < (off_t)blk) ? (size_t)(size - off) : blk;
        ssize_t n = pread(fd, buf, want, off);
        if (n < 0) { rc = -1; goto out; }

        /* Only a whole block can become a hole, so a short trailing read is
         * never eligible -- it closes any run in progress instead. */
        int zero = (n == (ssize_t)blk) && all_zero(buf, (size_t)n);

        if (zero) {
            if (run < 0)
                run = off;
        } else if (run >= 0) {
            if (fallocate(fd, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE,
                          run, off - run) < 0) { rc = -1; goto out; }
            run = -1;
        }
    }

    if (run >= 0 &&
        fallocate(fd, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE,
                  run, size - run) < 0)
        rc = -1;

out:
    free(buf);
    return rc;
}

/* Open O_RDWR (fallocate needs write access), then:
 *
 *   struct stat st;
 *   fstat(fd, &st);
 *   resparsify(fd, st.st_size, (size_t)st.st_blksize);
 *
 * Coalescing adjacent zero blocks into one punch call, as above, matters:
 * one fallocate per 4 KiB block over a multi-gigabyte file is a great deal
 * of syscall overhead for the same result.
 */

Two operational notes. First, punching holes in a file that something has mapped MAP_SHARED is a way to make that process see zeros where it expected data — do this to snapshot artifacts at rest, not to a memory file a running VM is mapped onto. Second, if the file lives on a reflinked clone, punching frees blocks only for that file's references; the sibling keeps what it still points at, which is correct and also means the reclaimed number will not match your arithmetic.

The design point: one bit instead of 4096 bytes

Step back from the tooling and there is a single idea running through all of this. A page of zeros carries no information. Storing it costs 4096 bytes, transferring it costs 4096 bytes, and both are pure waste, because the fact 'this page is zero' fits in one bit. Sparse files are the filesystem's expression of that idea. Compression is another expression of it, one that trades random access away. A non-zero-chunk index is a third, and it is the one that keeps random access, which is the property demand-paged restore lives on. The scheduling of the work is what makes it a good trade. A snapshot is written once and restored many times. Determining which regions are zero is a linear scan of the image — you pay it once, at bake time, off the critical path, and you get a bitmap that is small enough to keep in memory and ship alongside the artifact forever. Re-deriving that information on every restore, or re-decompressing the zeros on every restore, means paying repeatedly for an answer that never changes. Compute-once-consume-many is the whole reason a snapshot exists in the first place; the zero map is just the same principle applied one level down.

The summary

A file has an apparent size (st_size, what ls shows) and an allocated size (st_blocks, what du shows), and for a Firecracker memory snapshot they can differ enormously, because unused guest RAM is zeros and zeros can be holes. Holes arrive either by omission — a writer skipping a range — or by fallocate with FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE after the fact, and reading one returns zeros with no device I/O. ext4, XFS and btrfs all support this; the granularity is the filesystem block size, so unaligned or sub-block punches free nothing. Sparseness then quietly dies at every hop that reads-and-writes bytes: use cp --sparse=always or --reflink, tar -S, rsync -S, dd conv=sparse, and accept that scp and object storage will inflate you. Watch df for capacity, not summed file sizes, and remember that reflinked clones make per-file accounting double-count. And on the restore path, invert the framing: a recorded map of which regions are zero is the single most valuable piece of metadata you can attach to a memory image, because it turns a fetch into a zero-fill. On PandaStack this shows up in the boot path. Every sandbox, managed database and git-driven app starts by restoring a baked Firecracker snapshot — around 179ms p50 and 203ms p99 end to end, with the memory-load step landing near 49ms — and when the memory image is being streamed from object storage rather than read locally, the non-zero-chunk index is what keeps the fault handler from fetching megabytes of nothing. The core is open source under Apache-2.0, so you can point du, filefrag and the two snippets above at your own snapshots and see what your workloads actually leave untouched. For the artifact layout, start with /blog/firecracker-snapshot-file-format-explained; for how the restore consumes it, /blog/firecracker-memory-file-mmap-explained and /blog/userfaultfd-explained.