Where Sleeping Workloads Live: Storage Tiering Explained

Scale-to-zero gets sold as one sentence — "you stop paying when it's idle" — and everyone hears the CPU half of it. The CPU half is genuinely the easy half. You stop the workload, the cores go back in the pool, the meter stops. Done. The half nobody puts on the pricing page is that the workload's bytes did not go anywhere. A sleeping app still has a memory image and a disk image sitting on an SSD, and if 90% of your fleet is asleep at any given moment, most of your fastest and most expensive storage is holding data nobody has touched in three weeks.

That's the thing tiering fixes. It's unglamorous, it's mostly bookkeeping, and it's the difference between scale-to-zero being a real cost saving and being a partial one you quietly subsidise out of the storage budget.

I'm Ajay, I build PandaStack. We run sleeping apps as Firecracker microVM snapshots, and this is what we learned about where those snapshots should live, how to decide when to move them, and — the part that actually determines whether tiering is viable — how to wake something that isn't on local disk anymore without the user noticing.

What a sleeping workload actually consists of

Before you can tier something you have to be precise about what "it" is. A sleeping microVM is three artifacts with wildly different sizes and wildly different tiering economics:

• A memory image. The guest's RAM, frozen. Nominally this is the whole configured size — our base template is baked at 4 GiB — but the overwhelming majority of a guest's pages are zeros it never touched, so a zero-eliding capture writes a small fraction of that. This is usually the biggest artifact and the one worth moving.
• A disk image. Almost always a copy-on-write delta against a shared template rootfs, not a standalone disk. The apparent size is tens of gigabytes; the actual allocated blocks are whatever this workload wrote since it was created — node_modules, a build output, some logs.
• Metadata. The VMM state file, the network identity the guest was baked with, a manifest, checksums. Kilobytes. Free to move, and the thing you must never lose, because without it the other two are inert.

The copy-on-write point is the one that decides the economics, so it's worth stating bluntly: you are tiering the delta, not the image. If a thousand sleeping apps all derive from one template rootfs, the template stays hot on every host forever — it's shared, it's read constantly, moving it would be insane. What you move is the private per-app divergence. A freshly deployed app that ran an install and a build might have a delta of a few hundred megabytes against a shared base that's an order of magnitude larger. Tier the delta and you free real disk. Tier by apparent size and you'll conclude tiering is a huge win, build it, and discover you've spent a quarter shuffling sparse holes around.

The tiers are a curve, not a switch

The mental model people arrive with is binary: it's either on disk or it's in object storage. That's not wrong so much as it's missing the middle, and the middle is where most of your fleet should live. Think of it as a curve trading wake latency against storage price, with at least three points on it:

• Wake latency — Local NVMe: low hundreds of milliseconds, because there is no network in the path at all. Cheaper local or network-attached tier: adds a fetch over a link you control, so tens of milliseconds of extra latency per access, predictable. Object storage: dominated by object-store request latency and however much of the image you actually need before the guest can run — slower, and variable in a way local disk never is.
• Storage price — Local NVMe: the most expensive byte in your fleet, and you pay for it whether or not anything reads it. Cheaper local tier: meaningfully less per byte, usually at the cost of IOPS you don't need for a sleeper. Object storage: roughly an order of magnitude cheaper per byte, which is why the whole exercise exists.
• Access cost — Local NVMe: none, it's just I/O. Cheaper local tier: none or negligible. Object storage: you pay per request and, depending on your provider and topology, per byte egressed. This is the term people forget, and it's the one that can invert the whole calculation.
• Failure mode — Local NVMe: the host dies, the sleeper dies with it unless you replicated it. Cheaper local tier: same blast radius, usually. Object storage: survives the host entirely, which is a durability upgrade you get for free alongside the cost saving.
• Right for — Local NVMe: anything that woke in the last few days. Cheaper local tier: the medium-cold middle, if your infrastructure has one worth using. Object storage: the long tail that has been asleep for weeks and statistically will stay that way.

That last row is the real argument for tiering. Sleeper age distributions are brutally long-tailed. A large fraction of what's asleep on your hosts right now was last touched a month ago and will next be touched either never or in another month. Keeping that on NVMe next to the workloads that wake every hour is paying premium prices for archival storage.

The durability line is worth pausing on too. Moving a cold sleeper to object storage isn't only cheaper — it's the difference between "this app exists on exactly one machine" and "this app exists in a bucket." We learned that one the direct way: a maintenance event recreated instances in a managed group and took their boot disks with them, and the workloads that survived cleanly were the ones whose state already lived somewhere other than that disk.

The demotion policy: age, size, and enough hysteresis to stop thrashing

A demotion sweep is a background job that walks everything asleep on local disk and asks whether it should still be there. The naive version is a single age threshold — asleep 7 days, ship it. That works, and it's worse than it needs to be in two directions at once. It demotes tiny things whose disk you didn't need back, and it keeps enormous things around for a week when you're at 95% disk.

So weight by size, and let disk pressure move the threshold. You are not tidying; you are buying back bytes, and a demotion that frees 40 MiB is not worth a wake penalty later:

// tiering/sweep.go -- decide which sleepers leave local NVMe.
package tiering

import "time"

type Sleeper struct {
	ID      string
	Tier    Tier      // TierLocal | TierRemote
	Bytes   int64     // ALLOCATED blocks: CoW delta + memory image. Not file size.
	Touched time.Time // explicit stamp, written on wake and on publish
	Wakes7d int       // how often this thing is actually used
}

// pressure is diskUsed/diskTotal, so a full host demotes earlier than an
// empty one and the policy self-tunes instead of needing a magic constant
// per hardware generation.
func shouldDemote(s Sleeper, now time.Time, pressure float64) bool {
	if s.Tier != TierLocal {
		return false
	}

	// Hysteresis. Anything that woke more than twice this week stays hot no
	// matter what its age says. A workload that wakes daily will otherwise
	// ping-pong across the tier boundary forever, paying an upload and a
	// cold fetch every round trip to save disk it needs again tomorrow.
	if s.Wakes7d > 2 {
		return false
	}

	age := now.Sub(s.Touched)
	if age < 72*time.Hour { // hard floor: never demote something recent
		return false
	}

	gib := float64(s.Bytes) / (1 << 30)
	// score > 1 means "worth the round trip now". Old and large wins;
	// old and tiny loses; large and recent waits.
	score := (age.Hours() / 168.0) * (gib + 0.25) * pressure
	return score > 1.0
}

Two details in there matter more than the formula. The first is the hysteresis, and I'd argue it's the single most important line: without a wake-frequency guard, a workload that wakes every day at 9am will be demoted every night and promoted every morning, and you will have built a machine that converts your storage budget into egress charges. Tier boundaries need to be sticky in both directions — a minimum residency after promotion as well as a wake-count guard before demotion.

The second is `Touched`. Do not use filesystem atime for this. atime is one of the most reliably misleading fields in Unix: it's `relatime` on most modern mounts, so it only updates if it's already a day stale; it's `noatime` on plenty of production hosts because someone quite reasonably turned it off for performance; and it can be bumped by your own housekeeping — a backup, a checksum verification, an antivirus scan — none of which mean the workload was used. Write an explicit timestamp when the workload is actually woken or published, or key off the mtime of a piece of metadata you control and only advance deliberately, like the presence bitmap of its chunk cache. Then the sweep is reading a fact you asserted rather than a side effect you hoped meant something.

Promotion: streaming beats downloading, and it isn't close

Here is the assumption that kills most tiering designs before they ship: that waking a demoted workload means downloading it back. If your wake path is "fetch the whole multi-gigabyte memory image from object storage, write it to local disk, then start the VM," your cold wakes are bounded below by however long it takes to move gigabytes, and no policy tuning saves you. You will build the thing, discover cold wakes take tens of seconds, and either turn tiering off or set the demotion threshold so conservative it never fires.

You don't have to do that. Guest memory is demand-paged by construction — that's what an MMU is for — so you can start the VM before the memory arrives and fetch pages as the guest touches them. On Linux the mechanism is userfaultfd: you register the guest's memory region with the kernel, hand the file descriptor to the VMM, and when the guest touches a page that isn't there yet, the kernel hands you a fault event instead of a segfault. You resolve it — fetch the containing chunk over an HTTP range request, install it with UFFDIO_COPY — and the guest continues, having no idea anything unusual happened.

Three things make that practical rather than merely clever. Zero-elision: a small header baked next to the image records which chunks are non-zero, and absent chunks are answered with a zero page and no network call at all. Since guest memory is overwhelmingly zeros, this removes most of the fetches before you make them. Prefetch: record the hot chunk set at capture time and replay it in the background the moment the restore starts, so by the time the guest gets around to faulting on those pages they're already local. And chunk sizing: fetch in units big enough that the per-request overhead of object storage amortises, small enough that you're not pulling megabytes to satisfy one page fault. A few megabytes per chunk is a reasonable place to start arguing.

For calibration on the good case: a locally warm snapshot restore on our stack is p50 179ms and p99 around 203ms, and that includes the whole create path, not just the memory load. A cold wake where the image genuinely lives in a bucket is slower than that, and I'm not going to quote you a number for it, because it depends on your object store's latency, your chunk size, your prefetch coverage and how much of the guest's working set the app touches before it can answer a request. What I can tell you is the shape: it's dominated by object-storage round trips, and the entire engineering job is reducing the count of those round trips on the critical path, not making each one faster.

The shared chunk cache, and the crash-safety rule that isn't optional

There's a large win available if your sleepers share a common ancestor. Restores from the same template pull the same chunks — same kernel, same booted userland, same warm runtime — so a per-host cache keyed on the source object means the first wake of that image pays remote latency and every subsequent wake on that host reads from local disk. First one's slow, the rest are fast.

Be precise about what this dedupes, though, because it's easy to oversell. The cache is keyed on the object, so it shares chunks between wakes of the same image. It does not magically share chunks between two different apps' private memory images, even if those images are 80% identical, unless you've built content-addressed chunking on top — which is a different and much larger project. Template seeds benefit enormously. Per-app private snapshots benefit only on repeat wakes of that same app.

Now the part that will bite you. A chunk cache has a presence bitmap saying which chunks are valid locally, and the ordering of "write the data" and "set the bit" is a correctness boundary, not a performance detail:

// Fetch one chunk of a memory image over HTTP Range, then make it durable
// BEFORE anything is allowed to believe it is present.
func (c *ChunkCache) fetch(ctx context.Context, idx int64) ([]byte, error) {
	start := idx * chunkSize
	end := start + chunkSize - 1 // Range end is inclusive

	req, err := http.NewRequestWithContext(ctx, http.MethodGet, c.objectURL, nil)
	if err != nil {
		return nil, err
	}
	req.Header.Set("Range", fmt.Sprintf("bytes=%d-%d", start, end))
	req.Header.Set("Authorization", "Bearer "+c.token)

	resp, err := c.client.Do(req)
	if err != nil {
		return nil, err
	}
	defer resp.Body.Close()

	// A 200 here means the server ignored Range and is about to stream you
	// the entire multi-gigabyte object. That is a failure, not a bonus.
	if resp.StatusCode != http.StatusPartialContent {
		return nil, fmt.Errorf("range GET %s: want 206, got %s", c.objectURL, resp.Status)
	}

	buf := make([]byte, chunkSize)
	if _, err := io.ReadFull(resp.Body, buf); err != nil && err != io.ErrUnexpectedEOF {
		return nil, err // short final chunk is fine; a truncated middle one is not
	}

	// Ordering is the whole game:
	//   1. write into the sparse local cache file
	//   2. fdatasync it
	//   3. only THEN publish the present bit
	// Reverse 2 and 3 and a power loss leaves a bitmap promising data the
	// disk never received. The next wake then serves those bytes to a guest
	// as memory -- and a guest handed garbage memory does not politely
	// error, it computes wrong answers until something eventually panics.
	if _, err := c.file.WriteAt(buf, start); err != nil {
		return nil, err
	}
	if err := c.file.Sync(); err != nil {
		return nil, err
	}
	c.publishPresent(idx) // atomic tmp-write + rename of the bitmap

	return buf, nil
}

I want to be dramatic about this for one paragraph because I think it's under-appreciated. Corrupt a file and the reader gets a parse error. Corrupt a database page and you get a checksum failure and a loud complaint. Corrupt guest memory and you get nothing — no error, no signal, no log line. The guest reads whatever bytes you handed it and treats them as instructions and data, and the failure surfaces minutes later somewhere completely unrelated, if it surfaces at all. There is no defensive coding inside the guest that helps, because from the guest's point of view nothing went wrong. So the rule is absolute: a present bit is a durability claim, and you only make claims you've already fsynced.

One useful side effect of doing it this way: that bitmap's mtime, advanced only when you flush new chunks, is an honest last-used signal. Which is exactly what the demotion sweep wanted and atime refused to give you.

Two correctness rules I'd treat as non-negotiable

The first: never demote something you haven't proven you can restore. Upload, verify a manifest of per-chunk checksums against what actually landed in the bucket, and — for anything you're about to delete the only local copy of — do a real restore test, not a HEAD request. "The upload returned 200" is a statement about an HTTP conversation, not about whether the bytes in that bucket boot. We publish seeds with a SHA256 manifest for exactly this reason, and I'd generalise it: your tiering system's most important test is the one that pulls a random cold sleeper back and boots it, on a schedule, forever.

The second: treat "deleted from the hot tier" as two phases, never one. Mark the local copy as evictable, keep it, verify the remote, and only then reclaim the space — with the metadata flip to "lives remotely" as a separate, atomic, durable step. The failure you're guarding against is the one where the local delete succeeds, the metadata write doesn't, and now you have a workload the control plane thinks is on a disk where it isn't. Every tiering system that has hurt anyone hurt them in that window.

When not to tier at all

Two cases, and both of them are real.

If a workload has a strict wake SLO, don't tier it. Not "tier it with a shorter threshold" — don't. A cold wake's latency is a function of somebody else's object storage, which means its tail is a distribution you don't own and can't fix at 3am. If the product promise is a hard number, the storage has to be somewhere you control. Charge for that tier if you need to; pin it either way.

And if the egress cost of the wakes exceeds the disk you saved, tiering is a machine for converting cheap storage into expensive bandwidth. This is arithmetic, not judgement, so do the arithmetic before you build anything:

# Is tiering this workload actually cheaper? Put YOUR provider's numbers in.
LOCAL_NVME_GB_MONTH = 0.20   # premium local SSD
OBJECT_GB_MONTH     = 0.02   # standard object storage
EGRESS_GB           = 0.01   # per GB pulled back on a wake

def monthly_saving(delta_gib, wakes_per_month, working_set_fraction=0.25):
    """Positive = tiering wins. working_set_fraction is the share of the image
    a streaming wake actually pulls -- measure it, don't guess it. If your
    wake path downloads the whole image, this is 1.0 and the maths gets ugly
    fast."""
    hot_cost  = delta_gib * LOCAL_NVME_GB_MONTH
    cold_cost = delta_gib * OBJECT_GB_MONTH
    pulled    = delta_gib * working_set_fraction * wakes_per_month
    return hot_cost - (cold_cost + pulled * EGRESS_GB)

for wakes in (1, 10, 100, 500):
    print(wakes, "wakes/mo ->", round(monthly_saving(3.0, wakes), 3))

# The shape you'll see: hugely positive for the long tail, and negative
# somewhere past a few hundred wakes a month. That crossover point IS your
# hysteresis threshold -- you don't have to invent one, you can compute it.

The nice thing about writing it out is that the crossover falls out of the model instead of being a number somebody picked in a design doc. Feed your real prices in, find where the saving goes negative, and set the wake-frequency guard just below it. Now your policy has a reason rather than a vibe.

The takeaway

Scale-to-zero that only frees CPU is half a feature. The other half: tier the copy-on-write delta rather than the apparent size, treat the tiers as a latency-versus-cost curve with a middle, demote on an explicit touch timestamp weighted by size and damped by hysteresis, promote by streaming over range requests rather than downloading, cache chunks per-host so only the first wake pays the network, and never publish a presence bit before the data is durable. Then verify restores continuously, delete in two phases, and pin anything with a hard wake SLO to local disk where it belongs. Do that and the storage half stops being the part you hope nobody asks about.