virtio-blk Discard and TRIM in Firecracker, Explained

Here is the bug report you will eventually file against your own platform. A sandbox pulls a 6 GB dataset, processes it, writes a 200 KB summary, deletes the dataset, and exits. Inside the guest, `df` says the disk is nearly empty — the filesystem is telling the truth. On the host, the backing image file is still 6 GB larger than it was this morning, and it will stay that way. Multiply by a fleet, add copy-on-write clones that were supposed to be cheap, and you get a disk-usage graph with exactly one direction of travel and no obvious culprit, because nobody is leaking anything. Everybody deleted their files.

I'm Ajay; I built PandaStack, a Firecracker microVM platform where every sandbox create is a snapshot restore onto a copy-on-write rootfs clone, which means disk divergence is not an accounting curiosity — it is the thing that determines whether a host holds fifty sandboxes or five. This post is about discard: what actually has to happen for a `rm` in a guest to give a block back to the host, every link in that chain and how each one silently no-ops, why the `discard` mount option is usually the wrong answer despite being the obvious one, what it does to reflinked CoW clones, the non-obvious security consequence of snapshotting a guest that deleted secrets but never trimmed, and the pre-snapshot hygiene sequence you should be automating.

The chain from rm to a reclaimed block

For a deleted file inside a microVM to reduce the number of blocks the host has allocated to that VM's disk image, five distinct things have to happen, in order, in five different pieces of software. It is worth spelling them out individually because when the disk doesn't shrink, your job is to find which link is missing, and the failure looks identical from the outside in all five cases.

1. The guest filesystem frees the blocks in its own allocator. This is what `unlink` does: it drops the directory entry, decrements the inode link count, and marks the data blocks free in the block bitmap or extent tree. The data is still there. Nothing has been written to those blocks. The filesystem has simply stopped believing in them.
2. The filesystem tells the block layer. This is the step everyone skips, because nothing in the POSIX contract requires it. The filesystem has to explicitly issue a `REQ_OP_DISCARD` for the freed range — either inline at transaction-commit time (the `discard` mount option) or in a batch when something calls the `FITRIM` ioctl (which is all `fstrim` is).
3. The block layer forwards it to the device. It will only do this if the device's queue limits say the device can handle it. `discard_max_bytes` of 0 means the block layer drops the request on the floor and returns not-supported to the caller. No error is logged, no counter increments, and the filesystem shrugs.
4. The virtio-blk device carries it. The guest driver only issues discards if the device negotiated `VIRTIO_BLK_F_DISCARD` during feature negotiation, at which point discards travel as `VIRTIO_BLK_T_DISCARD` requests on the virtqueue with a range descriptor, bounded by config fields the device advertises for maximum sectors and segment count.
5. The VMM punches a hole. On the host side, a discard against a file-backed disk image becomes `fallocate` with `FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE` over the corresponding byte range. The file's apparent size is unchanged; its allocated block count drops; reads of that range now return zeros. This is the only step that actually gives you disk back.

Five links, five independent owners: your filesystem, your mount options or your cron, the kernel's block layer, the VMM's virtio-blk implementation, and the host filesystem's support for hole punching. Break any one and the other four still work perfectly, which is the problem.

Measure it before you theorise about it

Almost everyone who reasons about this problem reasons about the wrong number, and it is not their fault: the tools are actively unhelpful here. A disk image is a sparse file, and a sparse file has two sizes that disagree by design.

• `ls -l` shows `st_size` — the apparent size. For a sparse image this is a statement about the addressable range you promised the guest, not about disk you are paying for. A freshly created 20 GiB image that has never been written shows 20 GiB here, forever, and tells you nothing.
• `du` shows allocated blocks — derived from `st_blocks`, which counts 512-byte units actually backed by storage. This is the number your capacity planning cares about. Use `du --block-size=1` so you get bytes rather than a rounded human-readable figure that hides a gigabyte of drift.
• `stat` shows both at once, which makes it the honest tool: `%s` for apparent size, `%b` for allocated 512-byte blocks. If those two disagree by a lot, you have a sparse file; if they stop disagreeing over time, your sparseness is being eaten.
• `df` inside the guest shows the guest filesystem's own free-space accounting. It is correct, and it is about a completely different thing. A guest reporting 95% free while its host image is fully allocated is not a contradiction — it is the exact signature of a missing discard chain.
• `filefrag -v` on the host image shows the extent map, including holes. When you want to prove that a hole-punch actually happened rather than believing a delta in `du`, this is where you look.

# Measure from the HOST, on the backing image. Anything the guest tells you is
# about the guest's own allocator, not about disk you are being billed for.
IMG=/var/lib/pandastack/vms/$SBX/rootfs.ext4

ls -l "$IMG"                  # apparent size: 21474836480 (20 GiB) -- always
du --block-size=1 "$IMG"      # allocated:     1908736000  (~1.8 GiB) -- real
stat -c 'apparent=%s allocated=%b x %B' "$IMG"

# --- inside the guest -------------------------------------------------------
dd if=/dev/urandom of=/work/blob bs=1M count=6144   # 6 GiB of honest work
sync
#   host: du is now ~6 GiB higher. Nothing surprising has happened yet.

rm /work/blob
sync
df -h /work                   # guest: 6 GiB free again. The guest is correct.
#   host: du reports THE SAME NUMBER AS BEFORE THE rm. This is the whole post.

# Before blaming anything, ask whether the guest even has a way to tell the
# host. Zero here means the block layer will drop every discard request, and
# no mount option and no cron job will change that.
cat /sys/block/vda/queue/discard_max_bytes    # 0 -> chain is cut at link 3
cat /sys/block/vda/queue/discard_granularity  # smallest range worth discarding

# If it is non-zero: walk the free-space map and emit discards for every free
# extent. That is all fstrim does -- one FITRIM ioctl, on demand.
fstrim -v /work
#   /work: 6.1 GiB (6553600000 bytes) trimmed
#   ...or, on a device with no discard support:
#   fstrim: /work: the discard operation is not supported

# host again: allocated blocks drop back. Nobody rewrote anything; the VMM
# punched holes in the backing file over the ranges the guest gave up.
du --block-size=1 "$IMG"
filefrag -v "$IMG" | head -20   # proof: the extent map now contains holes

Run that once on your own platform before reading further. The result tells you which half of this post applies to you, and it takes ninety seconds.

Where the chain breaks, link by link

The device never advertises discard, so the guest never tries

This is the first thing to check and the one people check last, because it is invisible from the guest's application layer and there is no obvious place to look. If the virtio-blk device does not negotiate `VIRTIO_BLK_F_DISCARD`, the guest's block layer sets `discard_max_bytes` to 0, and every discard the filesystem generates is discarded — in the other sense of the word. You can mount with `discard`, you can run `fstrim` hourly, you can tune `discard_granularity` to your heart's content, and nothing will leave the guest.

Firecracker's block device is configured through `PUT /drives` on the API socket, and the shape of that request is instructive mostly for what is not in it.

# The full drive surface. Read it for what is ABSENT as much as what is present.
curl --unix-socket /run/firecracker.sock -i \
  -X PUT 'http://localhost/drives/rootfs' \
  -H 'Content-Type: application/json' \
  -d '{
        "drive_id": "rootfs",
        "path_on_host": "/var/lib/pandastack/vms/abc123/clone.ext4",
        "is_root_device": true,
        "is_read_only": false,
        "cache_type": "Unsafe",
        "io_engine": "Async",
        "rate_limiter": {
          "bandwidth": { "size": 209715200, "refill_time": 1000 }
        }
      }'

# There is no discard key here and no typo above. The drive model covers
# identity, host path, root-ness, read-only-ness, cache policy, the I/O engine
# and a rate limiter. Whether the emulated device offers VIRTIO_BLK_F_DISCARD
# to the guest is a property of the VMM's block implementation and its version,
# not a per-drive switch you flip. Check the release notes for the exact build
# you run before you design around it -- this surface moves between versions.

# So do not reason about it. Measure it, from inside the guest, on the device:
#   discard_max_bytes == 0  -> the guest will never emit a discard, and fstrim
#                              exits non-zero with "operation is not supported"
#   discard_max_bytes >  0  -> the chain is live; fstrim is now load-bearing
cat /sys/block/vda/queue/discard_max_bytes
cat /sys/block/vda/queue/discard_max_hw_bytes

If it reads 0, the guest-side story is over and your reclaim has to happen on the host against a stopped image — the section on pre-snapshot hygiene below covers exactly that. It is also worth knowing that Firecracker supports a vhost-user block backend, which moves the device implementation into a separate process entirely; in that architecture the feature set the guest sees is whatever that backend chooses to offer, not whatever the VMM does. I wrote about the shape of that split in /blog/firecracker-vhost-user-block-explained. Either way, verify against your build rather than against a blog post, including this one.

The device supports it, but nothing ever asks

The happier failure. `discard_max_bytes` is a healthy number, the chain is intact end to end, and the disk still grows — because no filesystem ever issued a discard. Linux filesystems do not trim on delete by default. ext4 and XFS both default to `nodiscard`; freeing blocks updates their own metadata and stops. The block layer is waiting for a request that is never made.

There are exactly two ways to make the request: mount with `discard` so it happens inline on every free, or call `FITRIM` periodically via `fstrim`. Most distributions ship a `fstrim.timer` systemd unit that runs weekly and most people never think about it again. In a microVM whose entire lifespan is eleven minutes, a weekly timer is decoration. If your guests are ephemeral, the timer will never fire in the life of any VM you ever create, and you will have all the configuration of a working trim strategy and none of the effect.

The discard mount option, and why distros moved away from it

So mount with `discard` and be done? This is the genuinely useful piece of advice in the post, so here is the full argument. With `discard` on ext4, freed extents are discarded as part of the journal commit that frees them. That makes reclaim immediate and removes an entire class of operational forgetting. It also puts a device round trip on the critical path of every `unlink`, inside a transaction commit, and discard is not a fast operation on most hardware — on physical SSDs it can stall the queue while the controller does bookkeeping.

The pathological case is a build directory. `rm -rf node_modules` frees tens of thousands of small extents, and inline discard turns that into tens of thousands of small discard requests, most of them below `discard_granularity` and therefore ignored by the device anyway — you paid for the round trip and got nothing. This is why the mainstream advice moved from the mount option to a periodic `fstrim`: batching lets the filesystem coalesce free extents into a small number of large ranges, which is the shape devices actually handle well, and it moves the cost off the latency path of every delete and onto a schedule you control. Btrfs took a third route and made asynchronous discard its default, queueing and batching trims in the background; ext4 and XFS have no equivalent, so for them the choice really is inline-or-scheduled.

For microVMs the calculus tilts further toward scheduled trims, but for a different reason: your guests are short-lived and your unit of work is a job, not a week. The right schedule is not weekly and not per-unlink — it is once, at the end of the job, before the machine is snapshotted or destroyed. That is a place in your orchestration you already have code in.

The thin layer underneath eats it

One more link, and it lives below the VMM. If the disk image is not a plain file on a normal filesystem but sits on a thin-provisioning target, the discard has to survive one more translation. Device-mapper thin pools can be configured to ignore discards entirely, or to accept them and update their own mapping without passing them down to the underlying device. Both are legitimate configurations with real reasons behind them, and both mean your carefully plumbed `fstrim` frees space in a layer you were not measuring and not in the one you were. If you run LVM thin, a SAN LUN, or any storage that presents thin volumes, confirm the passdown behaviour of your specific target before concluding the guest is at fault.

Copy-on-write is where this stops being cosmetic

On a single fat VM with a dedicated disk, a non-shrinking image is untidy. On a platform that creates disks by cloning, it changes the economics. PandaStack clones a template rootfs with reflink — an O(metadata) operation that produces a new file sharing every extent with the original — and can use dm-snapshot as an alternative CoW path. A clone starts costing essentially nothing and accrues cost only where it diverges from the template. That is the entire reason a create is 179ms p50 and roughly 203ms p99 rather than a multi-gigabyte copy; the mechanics are in /blog/copy-on-write-rootfs and the two CoW paths are compared in /blog/dm-snapshot-vs-reflink-cow.

Now put a job in that clone that downloads 6 GB and deletes it. The download wrote 6 GB of blocks that the clone did not share with the template, so the clone has diverged by 6 GB — it now has its own private extents, allocated on the host, referenced by nothing else. The delete freed them in the guest's allocator and changed nothing on the host. That clone is permanently 6 GB, for the rest of its life, and it will stay 6 GB in any snapshot you take of it. Its divergence is now proportional to total I/O rather than to the size of the result, which is the opposite of the property you bought CoW for.

Be precise about what a discard does in this world, because it is easy to get wrong in a way that produces false confidence. Punching a hole in a range of a reflinked file removes that file's reference to the shared extent. It does not touch the original, and it does not free the underlying blocks unless that reference was the last one. The practical reading: trimming a clone reclaims blocks the clone allocated privately by writing, and reclaims nothing at all over ranges it still shares with the template — because those blocks are still in use by the template and by every sibling clone. Which is exactly right. The shared base is not waste; the divergence is.

Snapshots remember what you deleted

This is the non-obvious one, and it is a security property rather than a capacity one. Because deleting a file does not erase its contents, a disk image taken from a guest that deleted sensitive data still contains that data verbatim in unreferenced blocks. A snapshot captured after the delete but before a trim carries those bytes with it — into your snapshot store, into object storage, into every clone restored from that snapshot, into whatever cross-region replication you have. The filesystem metadata says the file is gone. A hex dump of the image says otherwise, and forensic recovery of unlinked-but-not-overwritten files is a solved undergraduate exercise.

So a trim before capture is not merely a size optimisation; it is the step that makes deleted mean deleted. When the discard becomes a hole-punch on the backing file, the extents are removed from the file's map and subsequent reads of that range return zeros. On a file-backed virtual disk that is a genuine erasure of the bytes from the artifact, which is a stronger guarantee than TRIM gives you on physical SSDs, where the command is a hint about garbage collection and the flash may hold the old contents for a while yet. The broader problem of what ends up baked into a snapshot is in /blog/firecracker-snapshot-secrets-security — a build cache full of credentials and a deleted-but-not-trimmed key file are the same class of mistake.

The honest counterweight, since it belongs in the same section: discard is also a small information disclosure. Telling the host which ranges are free tells the host something about the guest's allocation pattern, and a host that watches hole-punch traffic learns roughly when and how much a guest freed. In most threat models this is uninteresting — the host already runs your VMM and can read your entire memory image at will. In a model where the host is genuinely untrusted, you were not going to hand it a plaintext disk image in the first place, and encrypted volumes change this calculation entirely. Weigh it, do not agonise over it.

Trim before you bake: the cheapest snapshot optimisation there is

Template rootfs images are built by a pipeline that installs toolchains, populates package caches, compiles things, and then deletes the intermediates. Every one of those deletes is a metadata edit, so the baked image ships with the full weight of the build even though the build artifacts are gone. Running one `fstrim` at the end of the bake, before the image is frozen and published, is usually the single largest reduction available and it costs one line in a script.

That reduction compounds where artifact movement dominates latency. A same-host fork lands in 400–750ms because the rootfs and memory are already local; a cross-host fork is 1.2–3.5s, and the difference is almost entirely bytes going over a network. Shrinking the rootfs artifact shrinks the part of that budget you can actually control. Be precise about scope, though: trimming affects the disk artifact only. It does nothing to `vm.mem`, which is a separate file governed by how much guest memory was touched — dropping the guest's page cache before capture is the lever there, and the memory side has its own streaming story in /blog/snapshot-restore-boot-path.

The blunter alternative you will see recommended is zero-filling free space: write a huge file of zeros until the disk is full, delete it, then run `fallocate --dig-holes` on the host to punch out every range that reads as zero. It works, in the narrow sense that you end up with a smaller image. It is also the wrong tool almost everywhere, and it is a genuinely nasty trap on a CoW clone.

The pre-snapshot hygiene sequence to automate

Here is the shape worth wiring into your orchestration once, so nobody has to remember it. It runs at the end of a job or at the end of a template bake, and it deliberately fails loudly at the trim step rather than swallowing the error, because a silent trim failure is the thing this entire post exists to prevent.

#!/usr/bin/env bash
# Pre-snapshot hygiene. Automate it -- a human forgets, and the cost of
# forgetting is paid on every restore of that template, forever.
set -euo pipefail
SOCK=/run/firecracker.sock

# 1) Quiesce writers. Snapshotting a filesystem mid-write means every restored
#    clone inherits the same dirty state and the same recovery on first boot.
guest_exec "systemctl stop app.service 2>/dev/null || true"

# 2) Flush. The page cache is the guest's opinion; the block device is the
#    fact. A snapshot captures both, disagreeing, and keeps the disagreement.
guest_exec "sync"

# 3) Drop caches. This does NOT shrink the rootfs -- it shrinks vm.mem, by not
#    freezing gigabytes of page cache for files nobody will ever read again.
guest_exec "echo 3 > /proc/sys/vm/drop_caches"

# 4) Return the free blocks. If the device advertises discard this is the whole
#    job. If it does not, fstrim exits non-zero HERE -- which is precisely why
#    this line is not decorated with `|| true`. A silent no-op is the bug.
if guest_exec "fstrim -av"; then
  echo "guest-side trim ok"
else
  echo "guest has no discard path; deferring to host-side reclaim" >&2
  NEEDS_HOST_RECLAIM=1
fi

# 5) Pause, then capture. Pausing stops the vCPUs so nothing races the capture;
#    step 4 already told the host which blocks are dead.
curl --unix-socket "$SOCK" -X PUT 'http://localhost/vm' \
  -H 'Content-Type: application/json' -d '{"state":"Paused"}'
curl --unix-socket "$SOCK" -X PUT 'http://localhost/snapshot/create' \
  -H 'Content-Type: application/json' \
  -d '{"snapshot_type":"Full","snapshot_path":"/seed/vm.state","mem_file_path":"/seed/vm.mem"}'

# 6) Host-side reclaim, only when the guest had no way to discard. The VM must
#    be STOPPED. The loop driver does implement discard, and translates it into
#    FALLOC_FL_PUNCH_HOLE against the backing file -- exactly the operation the
#    guest could not reach through its own virtio device.
if [ -n "${NEEDS_HOST_RECLAIM:-}" ]; then
  LOOP=$(losetup --find --show /seed/rootfs.ext4)
  mount -o discard "$LOOP" /mnt/reclaim
  fstrim -v /mnt/reclaim
  umount /mnt/reclaim
  losetup -d "$LOOP"
fi

# The only number that matters: what every cross-host restore has to move.
du --block-size=1 /seed/rootfs.ext4

Step 6 is the escape hatch worth internalising, because it removes the VMM from the equation. Mounting the stopped image through a loop device gives you a block device that does support discard, and a `fstrim` against that mount performs the hole-punches the guest was never able to request. It is offline-only and it is slower than doing it in the guest, but it is deterministic and it works regardless of what your virtio device advertises. The one thing it cannot do is help you with a running VM, which is why an ephemeral fleet with no guest discard path should be doing this at bake time rather than pretending it can do it per-job.

Five strategies, honestly compared

• When space comes back — `discard` mount option: immediately, at the transaction commit that frees the blocks. Periodic `fstrim`: on the timer's schedule, which for an ephemeral microVM usually means never. Manual pre-snapshot trim: exactly once, at the moment you care, before capture. Zero-fill + sparsify: when the offline sparsify pass runs, after the guest has already inflated the image to full. Do nothing: never, and the image grows monotonically toward its apparent size.
• Runtime cost — `discard` mount option: a device round trip inside every unlink's commit, worst on `rm -rf` of many small files where most requests fall below `discard_granularity` and are ignored anyway. Periodic `fstrim`: one batched I/O burst on a schedule, coalesced into large ranges. Manual pre-snapshot trim: the same burst, at a point where nothing is latency-sensitive. Zero-fill + sparsify: writes the entire free space of the disk, then reads it back — by far the most expensive option. Do nothing: free at runtime, expensive on the balance sheet.
• CoW clone divergence — `discard` mount option: kept minimal continuously, so divergence tracks live data. Periodic `fstrim`: sawtooths between trims; the peak is what your capacity planning has to survive. Manual pre-snapshot trim: divergence peaks during the job and collapses before capture, which is the shape you want. Zero-fill + sparsify: actively harmful — writing zeros over shared extents breaks CoW and diverges the clone before reclaiming anything. Do nothing: divergence grows with total I/O rather than with kept results.
• Snapshot and artifact size — `discard` mount option: consistently small, whenever you capture. Periodic `fstrim`: depends entirely on whether a trim happened to run before capture, which is a coin flip. Manual pre-snapshot trim: minimal by construction, which is the point. Zero-fill + sparsify: minimal, eventually, after doing enormous work to get there. Do nothing: carries every intermediate the build ever created, on every restore, in every region.
• Data remnants — `discard` mount option: deleted content is hole-punched out of the backing file promptly, so a snapshot rarely contains it. Periodic `fstrim`: a window exists between delete and trim in which the bytes are still in the artifact. Manual pre-snapshot trim: closes that window at exactly the moment it matters. Zero-fill + sparsify: overwrites remnants with zeros, which is thorough, if brutal. Do nothing: every deleted secret, cache entry and credential the guest ever touched ships inside the image.
• Behaviour when the device offers no discard — `discard` mount option: silently does nothing; mount succeeds and you believe you are covered. Periodic `fstrim`: fails loudly with "the discard operation is not supported", which is the most useful signal in this whole table. Manual pre-snapshot trim: the same loud failure, at a point in your pipeline where you can branch to a host-side loop-mount reclaim. Zero-fill + sparsify: the only option that still works, which is its entire justification. Do nothing: indistinguishable from every other option, which is how this problem survives so long.
• Operational complexity — `discard` mount option: one mount flag, no scheduling, no orchestration hook. Periodic `fstrim`: a timer unit that already ships with your distro, plus the realisation that its weekly cadence is meaningless for VMs measured in minutes. Manual pre-snapshot trim: a few lines in the code path that already snapshots, plus a decision about what to do when it fails. Zero-fill + sparsify: a whole offline pipeline stage with real failure modes. Do nothing: zero complexity and an unbounded disk bill.

The default I would recommend for anything microVM-shaped: do not mount with `discard`, do run an explicit trim as the last step of every job and every template bake, treat a non-zero exit from `fstrim` as a real signal rather than noise, and keep a host-side loop-mount reclaim path for the case where the guest's device offers you nothing. Reserve zero-filling for images you cannot loop-mount and cannot trim, and understand what it does to a clone before you reach for it.

What this looks like from inside a sandbox platform

The reason I care about this at all is density. A host holds sandboxes until it runs out of memory or disk, and on a CoW platform the disk term is entirely divergence. Each PandaStack agent pre-allocates 16,384 network slots, so networking is never the binding constraint; memory and disk are, and disk is the one where a single untrimmed job can quietly cost you a hundred times what the job's actual output was worth. The economics of that tradeoff are worked through in /blog/microvm-density-economics. In code, the fix is one line that most people never write.

import os
from pandastack import Sandbox

# A disk-heavy job: pull a large corpus, chew on it, keep a small answer.
# The interesting part is not the work. It is what the machine's DISK looks
# like on the way out, because that is what a snapshot would preserve and
# what the clone's divergence will be billed against.

corpus_url = os.environ["CORPUS_URL"]

sbx = Sandbox.create(
    template="code-interpreter",
    ttl_seconds=1800,
    metadata={"job": "nightly-corpus-rollup"},
)
try:
    out = sbx.exec(
        f"cd /work && curl -sSfL '{corpus_url}' -o corpus.tar.zst "
        "&& zstd -d --stdout corpus.tar.zst | tar -x "
        "&& python3 -m rollup --in corpus --out summary.json",
        timeout_seconds=1500,
    )
    if out.exit_code != 0:
        raise RuntimeError(f"rollup failed ({out.exit_code}): {out.stderr[-2000:]}")

    # ~40 GiB went through the disk. This is the 200 KB we actually wanted.
    result = sbx.filesystem.read("/work/summary.json")

    # Delete the intermediates -- and then TELL THE BLOCK LAYER. The rm alone
    # is a metadata edit in the guest's allocator: without the trim, this
    # clone has permanently diverged by 40 GiB, and every snapshot taken from
    # it carries those blocks along with their contents.
    sbx.exec("rm -rf /work/corpus /work/corpus.tar.zst && sync", timeout_seconds=120)

    trim = sbx.exec("fstrim -v /work", timeout_seconds=300)
    if trim.exit_code != 0:
        # Not fatal to the job, but it IS the difference between a clone that
        # costs what the job produced and one that costs what the job touched.
        # Do not swallow this: it means the device never advertised discard,
        # and no amount of retrying in the guest will change that.
        print(f"WARN no guest discard path: {trim.stderr.strip()}")
finally:
    sbx.kill()

For an ephemeral sandbox that gets destroyed immediately, the trim buys you nothing — the clone is deleted and its private extents go with it. The trim earns its keep in three specific situations, and it is worth knowing which one you are in: long-lived persistent sandboxes where divergence accumulates across many jobs, any sandbox you intend to snapshot or fork, and template bakes where the reduction is inherited by every restore for as long as that generation is current.

The summary

Deleting a file inside a guest is an edit to a bitmap that only the guest can see. For a host block to come back, the filesystem has to issue a discard, the block layer has to accept it, the virtio device has to have negotiated support for it, and the VMM has to turn it into a hole-punch on the backing file. All four links are optional, all four fail without an error message, and the resulting symptom — a disk that grows and never shrinks while every process behaves impeccably — looks the same regardless of which one is missing. Check `discard_max_bytes` first; it decides which half of the problem you have.

In a copy-on-write world the stakes are higher than tidiness. A clone is cheap because it shares extents with its template and only pays for divergence, and a job that writes six gigabytes and deletes them has diverged six gigabytes unless something punched those blocks back out. Discard is what keeps divergence proportional to what a job kept rather than to everything it ever touched. It is also what makes a delete real: a snapshot taken after an `rm` but before a trim still contains the bytes, which matters rather a lot if the deleted thing was a credential.

The practical version fits on a sticky note. Do not mount with `discard` — you will pay a device round trip on every unlink to solve a problem that batches beautifully. Do run `fstrim` explicitly at the end of every job and, more importantly, at the end of every template bake, where one line shrinks an artifact that every future restore has to move. Treat a failing `fstrim` as information rather than noise, and keep a host-side loop-mount reclaim path for guests whose device offers no discard at all. And do not zero-fill a copy-on-write clone to reclaim space; you will break sharing on every block you touch on your way to freeing it, which is a memorable way to make a disk problem worse.

If you want the surrounding machinery: the clone mechanics are in /blog/copy-on-write-rootfs, the two CoW backends are compared in /blog/dm-snapshot-vs-reflink-cow, the durability implications of the drive's cache policy are in /blog/firecracker-block-device-cache-modes, and what else ends up baked into an image you publish is in /blog/firecracker-snapshot-secrets-security.