Snapshot Restore vs Cold Boot: The Tradeoffs Nobody Lists

Every writeup about microVM snapshots opens the same way: restore is faster than boot, here is a graph, please buy our product. That's true and it's also the least interesting thing about snapshots. The interesting part is that a restored machine is not a fresh machine that happened to arrive early. It is a specific machine, from a specific moment in the past, resurrected — and it brings that moment with it. Its clock, its random number generator, its DNS cache, its open sockets, its memory of what year it is. You didn't skip the boot. You skipped time.

I'm Ajay; I built PandaStack, which creates every sandbox by restoring a baked Firecracker snapshot rather than cold-booting. I have therefore spent an unreasonable amount of my life debugging the consequences, and I want to write the post I wanted three years ago: not "snapshots are fast" but "here is the complete bill, itemized, including the lines you only discover at 2am when TLS starts failing on a machine that thinks it is May." The speed is real. So is the bill.

What a cold boot actually does

To understand what a restore skips, you have to be specific about what booting is. It is not one thing; it's a pipeline of distinct phases, each of which leaves the machine in a more useful state than it found it.

1. Firmware and kernel handoff: the VMM loads a kernel image, sets up initial memory and vCPU state, and jumps into it. Under a microVM there's no BIOS to enumerate, so you're already skipping most of what makes real hardware slow.
2. Kernel init: memory management comes up, schedulers initialize, and the kernel probes virtual devices — block, network, entropy, console — building the device model everything else will talk to.
3. Root filesystem mount and pivot: the kernel finds the root device, mounts it, and hands control to PID 1.
4. PID 1 and unit ordering: your init system reads its unit graph and starts things in dependency order. A lot of real-world boot time hides here — not in the kernel, but in one unit waiting on another that's waiting on the network.
5. Network configuration: interfaces come up, addresses are assigned, routes and DNS are configured, and anything needing to resolve a name now can.
6. Service warm-up: the application starts. A JIT compiles hot paths from cold, connection pools dial out and complete handshakes, caches populate, module graphs load into memory.

The last phase is often the most expensive one, and it is entirely your code's fault rather than the kernel's. A machine that boots in a second and then spends eight seconds JIT-warming and filling connection pools has an eight-second problem that no kernel tuning fixes. This is why snapshot restore is more powerful than it first sounds: it doesn't just skip the kernel, it skips your warm-up. A snapshot taken after your app is warm restores a warm app.

That's the whole trick, stated honestly: a restore starts from a machine that already did every one of those six phases. Which is exactly why it's fast, and exactly why it's weird. Everything on that list produced state, and every piece of that state is now frozen in a file and about to be handed to you again — possibly months later, possibly a thousand times concurrently.

The gains, with real numbers

Here's what PandaStack actually measures. There is no warm pool of idle VMs sitting around — every single create restores a baked template snapshot on demand. The restore step itself lands around 49ms. End-to-end create, including allocating a pre-built network namespace, reflinking the rootfs, forking the VMM process, loading the snapshot, resuming, and probing for readiness, is p50 179ms with p99 around 203ms. The first-ever cold boot of a template — the one that produces the snapshot in the first place — is about 3 seconds. So the arithmetic is: pay 3 seconds once, then pay ~179ms forever.

The second gain gets discussed less: restores are cheap in more than time. Guest memory restores copy-on-write, so the Nth restore doesn't copy gigabytes of RAM — pages are shared until something writes to them, and most of a warm machine's memory never gets written. The rootfs is a reflink clone, O(metadata) rather than O(disk). Forks work the same way: same-host 400–750ms, cross-host 1.2–3.5s. The network side is pre-allocated too — 16,384 /30 subnets per agent, already built as namespaces and taps, because doing that cold costs more than the rest of the create combined.

The bill

Now the part nobody puts on the graph. Seven things change when you stop booting machines. None of them are dealbreakers. All of them are things you have to know, because every one of them is a bug that will otherwise find you on its own schedule.

1. Configuration is frozen

A Firecracker snapshot captures the machine configuration alongside the memory image. vCPU count and RAM size are fixed at bake time and cannot be changed at restore. This is a property of the hypervisor and it isn't arbitrary: the memory image is literally a dump of a specific-sized address space, and the vCPU state is a dump of a specific number of vCPUs. You cannot restore a 4 GiB memory image into an 8 GiB machine any more than you can restore a photograph at a different aspect ratio and call it the same photograph.

This surprises people coming from containers, where memory is a flag you pass at run time. Here, sizing is a template-level decision made ahead of time, and resizing means re-baking. On PandaStack the `base` apps template is baked at 2 vCPU and 4 GiB, and the agent deliberately overrides an incoming create request's cpu/memory to match the snapshot rather than quietly producing a machine that disagrees with its own memory image. Treat guest size like a schema: version it, change it on purpose, expect a rebuild rather than a hot edit.

2. The clock is frozen

This is my favorite one, in the way that a scar is your favorite one. The guest's notion of time is part of its state. When you restore a snapshot baked in May and resume it in July, the guest wakes up believing it is May. Not approximately May. It is extremely confident about May. It has a wall clock, it has a monotonic clock that was counting when the machine was suspended, and neither of them noticed the intervening two months.

Then it tries to do something normal and everything goes sideways in a way that looks like a network problem:

• TLS validation breaks the moment an upstream rotates a certificate whose `notBefore` is after your bake date. The guest sees a certificate from the future and correctly refuses it. Your logs say "certificate is not yet valid," which is a sentence that will cost you an hour if you don't know why.
• JWTs and signed URLs misjudge expiry in both directions — the guest happily accepts tokens that expired weeks ago, and rejects freshly minted ones as too far in the future if there's a `nbf` skew check.
• Cron and any interval scheduler either fire a two-month backlog at once when time snaps forward, or sit silently doing nothing, depending on the implementation.
• Cache TTLs, rate-limit windows, and retry backoffs computed from monotonic time are all measured against a clock that skipped.
• Every log line is timestamped in the past, which makes correlating a guest incident with host metrics an exercise in creative archaeology.

The fix is unglamorous and non-optional: the platform must resync the guest clock on every restore, resume, and wake — not "eventually via NTP," but as part of the restore path, before the guest is handed back to the user. PandaStack does this on all three transitions. If you're building on raw Firecracker, this is one of the first things you should wire up, and you should test it by baking a snapshot, waiting a week, restoring it, and running `date`. If your platform doesn't do it, you will find out when a CA rotates and a fleet of machines simultaneously decides the internet is untrustworthy.

# The "what actually changed across a restore?" diagnostic.
# Run this immediately after a sandbox comes up. Every line is a
# question about whether the machine knows what happened to it.

# 1. Does the guest know what day it is?
date -u +'%Y-%m-%dT%H:%M:%SZ'

# 2. Uptime is measured from BOOT, not from restore. A machine that
#    came up 40ms ago reporting 6 days of uptime has been restored.
cat /proc/uptime

# 3. Boot id: stable across a restore, regenerated on a real boot.
#    Two "different" VMs with the same boot_id are clones.
cat /proc/sys/kernel/random/boot_id

# 4. Entropy: is the pool the same one the snapshot was baked with?
#    Identical output across two fresh sandboxes = cloned RNG state.
head -c 16 /dev/urandom | xxd -p

# 5. TLS sanity — the canary for a frozen clock. This fails with
#    "certificate is not yet valid" long before anything else breaks.
curl -sS -o /dev/null -w '%{http_code}\n' https://example.com

# 6. Did anything reseed us? On a restored guest the kernel logs a
#    VM generation change here if the VMM signalled one.
dmesg 2>/dev/null | grep -iE 'vmgenid|random: .*(crng|reseed)' | tail -5

3. Entropy is cloned

This is the one with actual security consequences, and it follows inevitably from the definition. A snapshot is a memory image. The kernel's random number generator state lives in memory. Therefore every restore of the same snapshot starts with the same RNG state. Restore it a thousand times in parallel and you have a thousand machines that will produce the same "random" bytes in the same order, unless something intervenes.

Think about what that touches. Session identifiers. CSRF tokens. Nonces. Ephemeral keys for a TLS handshake. UUIDs, if they're v4 and drawn from the same pool. If your workload generates a keypair at startup and startup was captured in the snapshot, then every clone has the same key — and worse, it isn't obvious, because everything still functions. Nothing errors. Two users just quietly get the same session token.

It's not only the RNG. Anything in memory at bake time is in every clone: an in-flight session ID, a cached signing key, a connection's negotiated state. The mechanism that exists to solve this is VMGenID — a virtual device whose entire job is to tell the guest "you were restored; the identifier you knew has changed." A modern Linux guest treats that change as a signal to reseed its CRNG. The plumbing has to work end to end: the VMM exposes the device, the guest kernel is new enough to act on it, and — critically — your userspace re-derives anything it cached before the snapshot. A kernel reseed does nothing for a Python process holding a `SystemRandom` it initialized at import time.

4. Snapshots capture secrets

A memory image is a memory image. Whatever was in guest RAM at bake time is in the file: environment variables, API tokens you exported to test something, a decrypted private key, the contents of a `.env` you meant to delete, the plaintext of everything your process had open. Encryption at rest doesn't help, because the state that matters was decrypted and resident when you pressed pause.

And then that file gets replicated. In any serious multi-host setup, snapshots go to object storage so other machines can restore them — PandaStack streams memory chunks out of GCS on demand for exactly this reason. So a snapshot is, among other things, an extremely fast and very reliable way to distribute your old secrets to every host in the fleet, plus a bucket, plus whatever your bucket's lifecycle policy considers an appropriate retention period. The convenience is the risk. It works flawlessly.

• Bake from a clean machine. The bake environment should have exactly the credentials it needs to build, and none of the ones the workload will use at run time.
• Inject runtime secrets after restore, through the same path you'd use for any ephemeral machine — not baked in, not in the template's environment.
• Treat the snapshot bucket with the same access controls as a secrets store, because functionally it is one, just an accidental one.
• A secret that ever entered a baked snapshot is compromised for the life of that snapshot. Rotation means rotating the credential *and* re-baking; doing only the first leaves the old value sitting in every replica.

5. Staleness

A snapshot is a frozen filesystem plus a frozen memory image, and freezing is a thing that ages badly. The machine you baked in May has May's packages, May's CA bundle, May's pinned dependency versions, and May's understanding of which credentials are still valid. Restore it in July and it's a machine that skipped two months of the world and is, as established, extremely confident.

Container images have this problem too, which is why nobody who has run a registry finds it surprising. But snapshots make it slightly worse in one respect: an image is instructions, so rebuilding it re-runs those instructions and you get whatever `apt-get update` gets today. A snapshot rebuilds nothing. It restores exactly the bytes. There is no path by which a snapshot drifts forward accidentally; drift only ever happens because you deliberately re-baked.

So re-bake cadence is now an operational responsibility you didn't have before, and it needs an owner and a schedule rather than a vibe. What ages, roughly in the order it'll bite you: CA bundles and anything with an expiry (see: the frozen clock, compounding); OS security patches; runtime patch versions; pinned dependency trees with published CVEs; embedded credentials. Put a re-bake on a cron, put the current generation's age on a dashboard, and alert when it crosses whatever number you'd be embarrassed to admit in a security review.

6. Version and host coupling

A snapshot is a serialization of internal VMM state, and serialization formats are versioned. Restoring a snapshot on a VMM version different from the one that took it is not guaranteed to work, and failure modes range from a clean refusal (good) to something subtler (bad). Firecracker documents its snapshot compatibility policy and version-mapping rules; read those rather than my summary, since the details change per release. The operational shape is the same everywhere: upgrading your VMM is now coupled to your snapshot inventory, and "roll the fleet to a new version" quietly means "and re-bake, in the right order, with a rollback plan."

The subtler coupling is CPU features. A snapshot captures a guest that already probed its CPU and made decisions — which instruction set extensions to use, which paths a JIT compiled, which crypto implementation a library selected. Restore that guest onto a host lacking a feature it decided to use at bake time and it doesn't gracefully degrade. It executes an illegal instruction and dies, producing a confusing crash rather than a helpful error, and only on some of your hosts — the worst possible distribution of a bug.

Two ways out; pick one deliberately. Keep the pool homogeneous — same CPU model everywhere a snapshot can land, enforced by scheduling constraints rather than hope — or use CPU feature masking (CPU templates) to present a reduced, uniform CPU to every guest, so what the guest probes at bake time exists on every host it might restore onto. Masking costs you the newest instructions on your newest hardware; homogeneity costs you capacity-planning flexibility. Both are fine. Choosing neither means the bug lands on whoever is on call the week you add an instance type.

7. Debuggability

"It works on a cold boot but not on a restore" is a genuinely new bug class, and it's disorienting the first several times because the two machines are supposed to be the same machine. Your instinct is to diff the filesystem, which is identical, and then to check the config, which is identical, and then to consider a career change.

The good news is that the search space is small. In my experience the cause is essentially always one of the six things above. Work the list in order of how often it's the answer: is the clock right? Is the entropy fresh, or did every clone get the same bytes? Is the machine the size the app expects? Is something in there stale — an expired credential, an outdated CA bundle? Did this land on a host with a different CPU than the bake host? Is the VMM version the same? Six questions, and the fifth one only matters if you have heterogeneous hosts.

The other thing that helps enormously is making "was I restored?" a fact your application observes rather than infers. `/proc/uptime` against wall-clock age is the cheapest signal available: a machine reporting six days of uptime that started answering requests forty milliseconds ago has obviously been restored. Log it at startup. Future you, staring at a line saying the process has been up a week on a sandbox created this morning, saves an hour.

from pandastack import Sandbox

# Create from a baked snapshot. This is the ~179ms p50 path: no cold
# boot, no warm pool — a machine from the past, resumed.
sbx = Sandbox.create(template="base", ttl_seconds=900)

PROBE = r"""
import os, ssl, time, socket, datetime

with open("/proc/uptime") as f:
    uptime = float(f.read().split()[0])

# Guest wall clock vs. how long the guest thinks it has been running.
now = datetime.datetime.now(datetime.timezone.utc)
boot = now - datetime.timedelta(seconds=uptime)

print(f"guest_now   = {now.isoformat()}")
print(f"guest_boot  = {boot.isoformat()}")
print(f"uptime_s    = {uptime:.1f}")

# A fresh sandbox reporting minutes of uptime was RESTORED, not booted.
print(f"restored    = {uptime > 60}")
print(f"entropy     = {os.urandom(8).hex()}")  # differ across clones?
print(f"boot_id     = {open('/proc/sys/kernel/random/boot_id').read().strip()}")

# The clock canary: TLS is the first thing a frozen clock breaks.
try:
    ctx = ssl.create_default_context()
    with socket.create_connection(("example.com", 443), timeout=10) as raw:
        with ctx.wrap_socket(raw, server_hostname="example.com") as tls:
            print("tls         = ok", tls.version())
except ssl.SSLCertVerificationError as e:
    print("tls         = FAILED (check the guest clock):", e.verify_message)
"""

sbx.filesystem.write("/tmp/probe.py", PROBE)
r = sbx.exec("python3 /tmp/probe.py", timeout_seconds=60)
print(r.stdout)
if r.exit_code != 0:
    print("probe failed:", r.stderr[-2000:])

# Run it twice and diff the entropy line. Identical bytes across two
# separate sandboxes means nothing reseeded the CRNG after restore.
sbx.kill()

The two models, side by side

• Time to usable machine — Cold boot: kernel init, device probe, PID 1 unit ordering, network config, then your app's own warm-up, which is usually the biggest slice. Snapshot restore: ~49ms restore step, ~179ms p50 end-to-end create, ~203ms p99, with the app already warm.
• First run of a template — Cold boot: identical to every subsequent run, which is honest but never improves. Snapshot restore: ~3s for the first cold boot that produces the snapshot, then ~179ms forever after.
• Machine sizing — Cold boot: set vCPU and RAM per instance at launch, freely. Snapshot restore: frozen at bake time; resizing requires a re-bake, because the memory image is a specific-sized address space.
• Clock — Cold boot: correct on arrival; the guest syncs like any freshly-started machine. Snapshot restore: wakes believing it is bake time, and needs an explicit resync on restore, resume, and wake or TLS eventually breaks.
• Entropy and identity — Cold boot: fresh RNG state, fresh boot id, fresh everything, every time. Snapshot restore: cloned from the image; needs VMGenID plus userspace that actually re-derives what it cached.
• Secrets — Cold boot: only what you inject at run time is ever resident. Snapshot restore: whatever was in RAM at bake time is in the memory image, and that image gets replicated to object storage.
• Freshness — Cold boot: picks up whatever the image or provisioning produces today. Snapshot restore: exactly the bytes you froze; drift only happens when you deliberately re-bake, so cadence becomes an owned responsibility.
• Host and VMM coupling — Cold boot: the guest probes the CPU it actually landed on and adapts. Snapshot restore: the guest already decided which CPU features to use; needs homogeneous pools or CPU feature masking, plus VMM version discipline.
• Marginal cost of the Nth machine — Cold boot: a full boot and a full warm-up, every time. Snapshot restore: copy-on-write memory and a reflink rootfs, so no gigabytes get copied per clone.

So when is cold boot actually right?

It would be a bad post that spent two thousand words on the bill and then told you to pay it unconditionally. Cold boot is the correct choice more often than snapshot vendors like to admit, and the cases are clean.

• One-off, long-lived machines. If a machine boots once and runs for six months, three seconds of boot is a rounding error. You'd be paying real operational cost — bake pipelines, staleness monitoring, entropy hygiene — to optimize something that rounds to zero. Just boot it.
• Development and debugging loops. When you're changing the machine itself, a snapshot is an obstacle: every change means a re-bake, and a re-bake is slower than the boot you were avoiding. Boot cold while iterating; snapshot once it's stable.
• Trivially cheap startup. A static binary that starts in milliseconds with no JIT, no pools, and no caches has already given up most of the snapshot's advantage — you'd take on all seven bill items to save kernel time you weren't spending much of.
• Per-instance sizing matters more than speed. If tenants genuinely need different memory footprints and you can't enumerate them into a small set of baked sizes, frozen config is the wrong tradeoff. Boot, and set size at launch.
• Per-instance uniqueness where you don't control the guest software. If you can't guarantee the workload re-derives identity after a VMGenID change — because it's someone else's binary — cloning its memory is a security decision you're making on its behalf.
• Low creation frequency. Snapshot machinery pays for itself in aggregate. Ten machines a day is not aggregate.

Snapshot restore is a bet that the machine you froze is still the machine you want. That bet is excellent when you create machines constantly and terrible when you create them rarely — and the bet's odds decay with every day since the last bake.

The decision checklist

If you're weighing this for a real system, here's the list I'd work through. Answer them before you build, not after the first weird incident.

1. How often do you create machines? Hundreds or thousands a day makes the amortization obvious. A handful makes it a hobby.
2. How much of your startup cost is warm-up rather than boot? If most of it is your app filling pools and JIT-warming, snapshots buy far more than the boot numbers suggest — and that's the strongest case for them.
3. Can you enumerate your machine sizes into a small set? If yes, frozen config is a non-issue: bake one snapshot per size. If sizing is genuinely continuous, this model fights you.
4. Does something resync the guest clock on restore, resume, and wake? If you can't point at the code that does it, assume it doesn't, and expect TLS failures on a CA rotation.
5. Does the guest reseed after restore — kernel *and* userspace? Verify empirically: restore twice, compare 16 bytes of `/dev/urandom`. Identical output is a finding, not a curiosity.
6. Is the bake environment clean of run-time secrets? Assume the memory image is a public artifact within your fleet and design backwards from there.
7. Who owns re-bake cadence, and is snapshot age on a dashboard? An unowned re-bake schedule is one that stopped six months ago and nobody noticed.
8. Are your hosts homogeneous, or do you mask CPU features? Pick deliberately. Discovering this via an illegal-instruction crash on one instance type is an expensive way to make the decision.
9. Is your VMM upgrade plan coupled to your snapshot inventory? Read your VMM's actual compatibility policy — mine is a summary, theirs is the contract.
10. When something works cold and fails restored, does your team know to check these six things first? That knowledge is the difference between a twenty-minute fix and a lost afternoon.

I'd make the same choice again — every PandaStack create is a restore, and the 179ms p50 is the whole product. But be clear about what the choice is. You are not buying a faster boot; you are buying out of booting entirely, and in exchange you accept a machine that is a specific moment in the past, with all of that moment's assumptions intact and confident. Sync its clock, reseed its randomness, keep secrets out of its memory, re-bake it on a schedule, and pin down what CPU it thinks it's running on. Do those five things and the weirdness stays theoretical. Skip them and you'll rediscover each one individually, in production, in an order chosen by chance.