Snapshot Restore vs Container Image Pull: Two Ways to Start Fast

There are two fundamentally different ways to make a workload start fast, and the industry spends most of its energy on the one that helps less. The container answer is: get the bytes to the machine faster. Pull layers (or hit a local cache), unpack them, stack an overlayfs, exec PID 1, and let your application initialize itself. The snapshot answer is: don't initialize anything. Map a memory image of a process that already finished initializing, and resume it mid-instruction. I'm Ajay, I built PandaStack on the second model — and the reason isn't that pulling images is slow. It's that pulling images was never the expensive part.

What actually happens when a container starts

Lay out the sequence honestly, because the interesting bit is at the end. A container start resolves an image reference, pulls whatever layers aren't already on the host, unpacks each compressed layer to disk, stacks them into an overlayfs mount, sets up namespaces and cgroups, and execs PID 1. At that moment your orchestrator says the container is running, your dashboard turns green, and a metric somewhere records a very respectable number.

Then the actual work starts. PID 1 is your runtime, and it begins from absolutely nothing. A Python service imports its dependency tree — hundreds of modules, each one a stat, an open, a read, a compile-or-unpickle, a chunk of module-level code. A JVM or Node service starts interpreting cold and won't reach steady-state throughput until the JIT has seen enough of your hot paths to compile them. Then come the connection pools, the TLS handshakes to your database and cache, the config fetch, the schema reflection, the model weights or embedding index someone decided to load at boot. None of this is in the image-pull path. All of it happens after the container is, officially, running.

"Container started" and "application ready" are two different events, and every optimization aimed at the first one is aimed at the smaller number.

This is why the interesting measurement isn't the one your runtime reports. Time the gap yourself: from process launch to the first successful health check. In a typical Python service with a heavy scientific or ML dependency tree, that gap can dwarf everything that came before it — your container started in a couple hundred milliseconds and then spent the next several seconds importing libraries, which is a kind of performance.

#!/usr/bin/env bash
# Measure the number that matters: not "container running",
# but "app answered /healthz". These are NOT the same event.
set -euo pipefail

start=$(date +%s.%N)

# Whatever your runtime is. The point is what comes AFTER this line.
docker run -d --rm --name svc -p 8080:8080 acme/api:latest >/dev/null
running=$(date +%s.%N)

# Poll until the application itself says it is ready.
until curl -fsS http://localhost:8080/healthz >/dev/null 2>&1; do
  sleep 0.05
done
ready=$(date +%s.%N)

echo "container reported running : $(echo "$running - $start" | bc)s"
echo "app answered /healthz     : $(echo "$ready   - $start" | bc)s"
echo "---"
echo "the gap between those two lines is your application init."
echo "no registry mirror, no lazy pull, and no faster disk shrinks it."

docker rm -f svc >/dev/null

Run that against your own service before you believe anything else in this post. The two numbers it prints are the entire argument. If they're close together, your startup is genuinely dominated by image distribution and the container ecosystem has excellent tools for you. If the second is a multiple of the first — which it very often is for anything with a real dependency tree — then every hour spent on pull latency is an hour spent on the wrong half.

Why lazy pulling attacks the wrong half

The container world has built genuinely excellent machinery for the byte-delivery problem, and I want to be precise that it works. Lazy-pulling snapshotters like stargz and nydus restructure images so the container can start before the whole thing has landed, fetching file ranges on demand instead of unpacking every layer up front. Registry mirrors and pull-through caches move the bytes closer. Pre-pulled node caches and DaemonSet warmers put popular images on the disk before anything schedules onto it. Peer-to-peer distribution stops a thousand nodes from stampeding one registry. All of it is real engineering and all of it helps.

But look at what every single one of those optimizes: the interval between "we need this image" and "exec PID 1". They compress the prologue. Not one of them touches the part where your interpreter imports four hundred modules, your JIT starts cold, and your connection pool dials out. Drive image pull all the way to zero — imagine the layers were already unpacked and sitting in page cache, the theoretical best case — and application init is still there, unchanged, in full.

The honest framing: image pull optimization has a hard floor at "the process starts from scratch, instantly." That floor is not zero. For a service whose init dominates, the floor is most of your startup time.

What a snapshot captures that an image cannot

A container image is a filesystem. A stack of tarballs, some metadata, an entrypoint. It describes what your program will read when it starts — which is a genuinely useful thing to describe, and it's why images are portable, cacheable, and content-addressed. But a filesystem has no notion of a running process, so an image can only ever hand you a starting line.

A microVM snapshot captures the machine instead: the guest's physical RAM and the VMM's device and CPU register state, frozen at an instant. That's a categorically different object, because RAM is where all the expensive init went. Restoring one doesn't start a process. It resumes one, mid-instruction, with everything it had already computed still sitting there.

• An initialized heap — every object your framework allocated during boot: the parsed config, the routing table, the dependency-injection graph, the ORM's reflected schema. An image contains the code that builds those; a snapshot contains the result.
• An interpreter with its modules already imported — a Python process with hundreds of modules resident in memory, past every stat, read, compile, and module-level side effect. Restore doesn't import them again. They never got unimported.
• A warmed JIT and warmed caches — compiled hot paths in a JIT-based runtime, populated in-process caches, template caches, the CPU-and-memory work that only happens after real code has run once.
• Open, restored state — file descriptors, mapped regions, and device state restored as part of the VM's saved state, rather than re-established from zero by init code.
• Loaded weights and indexes — a model, an embedding index, or a large lookup table already in RAM. An image ships the file; the snapshot ships it already parsed and resident.
• Kernel-side warmth — a guest kernel already booted with devices probed, plus a page cache reflecting what the workload actually touched. Even a perfectly cached image still boots or execs into a cold context.

This is the whole asymmetry in one sentence: an image is the input to initialization, and a snapshot is its output. If you find yourself carefully optimizing how fast the input arrives, it's worth asking whether you could just cache the output instead.

The head-to-head

Neither model is universally better. They're optimizing different segments of the same timeline, and they fail in different places.

• What the artifact is — Container image pull: a content-addressed stack of filesystem layers describing what your program will read. Snapshot restore: a memory image plus device/CPU state describing a machine that has already run.
• What starting costs you — Container image pull: fetch (or cache-hit) + unpack + overlay mount + exec, then the full application init, every single time. Snapshot restore: map the memory image, load device state, unpause the vCPUs. Application init already happened, once, at bake time.
• The floor on latency — Container image pull: bounded below by how fast your app initializes from scratch, even with a perfectly warm local cache. Snapshot restore: bounded by how fast you can map memory and resume — on PandaStack about 49ms for the snapshot-load step, 179ms p50 and ~203ms p99 end to end for a create.
• Artifact size and handling — Container image pull: compressed layers, deduplicated across images, incrementally updatable, excellent registry tooling. Snapshot restore: a memory file roughly the size of the guest's RAM — bigger, less dedupable, and it wants streaming or CoW tricks to stay practical.
• Portability — Container image pull: runs anywhere the runtime and architecture match; the ecosystem's crown jewel. Snapshot restore: far pickier — the CPU features visible at bake time must be available at restore time, so heterogeneous fleets need CPU templates to mask a common feature set.
• Resource shape — Container image pull: CPU and memory limits are set at run time and can often be changed on restart. Snapshot restore: vCPU count and RAM size are baked into the snapshot; changing them means re-baking, not passing a flag.
• Freshness and patching — Container image pull: rebuild the image, roll the deployment; the layer model makes patch propagation routine. Snapshot restore: a snapshot is a frozen machine, and frozen machines drift — an unpatched library sitting in a memory image will keep being restored until someone re-bakes.
• Failure mode when things go wrong — Container image pull: registry outage, rate limit, or a slow cold pull on a fresh node. Snapshot restore: a stale or invalidated snapshot forces a real cold boot (~3s on PandaStack for a template's first spawn), plus clock and entropy weirdness on resume if you don't handle it.
• Operational surface — Container image pull: enormous, mature, well-understood; everyone on your team already knows it. Snapshot restore: needs a real substrate — copy-on-write memory, a CoW disk layer, and demand paging — and far fewer engineers have debugged it at 3am.

The costs of snapshots, stated plainly

I'd rather you take the trade-off seriously than take my word for it, so here are the parts that hurt. Snapshots are big: you're persisting the guest's RAM, so a multi-gigabyte guest produces a multi-gigabyte artifact, and unlike layered images it doesn't dedupe nicely across variants. Naively, that means a create can't start until a large file has landed on the host — which would reintroduce exactly the latency you were trying to kill.

Snapshots also go stale in a way images don't. A container image with a vulnerable library is fixed by rebuilding and rolling. A snapshot with a vulnerable library is a frozen machine that will keep waking up with that library resident until someone re-bakes it — so bakes have to be a first-class, automated part of your build pipeline, with the same cadence discipline you'd apply to base-image updates. Snapshots also freeze machine shape: vCPU count and RAM are properties of the saved state, not runtime flags, so "give this one 8 GiB instead" means a re-bake, not a parameter.

Then there's portability. A restored guest expects the CPU it was frozen on. Restore it onto a host with a different microarchitecture and code that was compiled or dispatched against features present at bake time can meet a host that lacks them, with results ranging from an illegal instruction to something subtler and worse. The mitigation is CPU templates: mask the guest's visible feature set down to a common baseline across your fleet so a snapshot restores anywhere in it. That's real work, and it costs you the newest instructions on your newest hardware.

That clock problem in particular bites in production and looks like magic when it does: a guest restored from a snapshot baked weeks ago wakes up believing it's still bake day, and then every TLS handshake starts failing because the certificate it's validating hasn't been issued yet, as far as it's concerned. The fix is a deliberate clock sync on every restore and resume. It's not hard — it's just not optional, and you only learn it once.

Streaming memory: lazy pulling, but for RAM

The artifact-size problem has the same shape as the image-size problem, so it has the same shape of answer — and this is where the two models converge more than people expect. Lazy pulling works because a container reads a small fraction of the files in its image. Memory streaming works for the same reason: a restored guest touches a small fraction of its RAM before it's usefully serving, and downloading pages it will never touch is pure waste.

So PandaStack doesn't download the memory file. The agent backs guest memory with userfaultfd: it registers the guest's memory regions with the kernel, and when the guest touches a page that isn't there yet, the fault is delivered to a userspace handler instead of the kernel resolving it. The handler maps that fault to an offset in the snapshot, fetches the surrounding 4 MiB chunk from object storage with an HTTP Range GET, installs it with UFFDIO_COPY, and wakes the parked vCPU. All-zero chunks are recorded at bake time and filled in locally with no fetch at all. A bake-time prefetch trace replays the hot chunk set in the background so most faults become cache hits, and fetched chunks land in a shared per-host cache, so the first restore of a template on a host pays network latency and every later one is local-disk fast.

The parallel to stargz and nydus is exact, one layer down: demand-fault the working set instead of transferring the whole artifact. The difference is what's being demand-faulted. Lazy image pulling streams the files a cold process is about to read on its way to initializing itself. Memory streaming streams the pages of a process that already finished initializing. Same technique, applied to the half of the problem that costs more.

When image pull is genuinely the right answer

Snapshots are not a universal upgrade, and pretending otherwise would be the kind of vendor argument I'd roll my eyes at. There's a large and boring class of workloads where container image pull is simply correct.

• Long-running services. If a process starts once and serves for weeks, a 12-second init amortizes to nothing. Optimizing it is a rounding error dressed up as engineering.
• Heterogeneous fleets. Mixed CPU generations, mixed vendors, multiple clouds — images run anywhere the architecture matches, while snapshots need CPU templates and careful masking to survive the same spread.
• Rapid patch cadence. If you rebuild and roll constantly, the layered model's incremental updates and content addressing are exactly what you want, and a snapshot's frozen-machine staleness is pure friction.
• Small, fast-starting programs. A static Go binary that's serving in milliseconds has no init worth caching. The snapshot would be all cost and no benefit.
• Operational simplicity. Your team already knows registries, layers, and rollbacks. Snapshot substrates bring CoW memory, demand paging, and bake pipelines — a real budget line, not a footnote.

The pattern where snapshots dominate is the mirror image of that list: short-lived work, started constantly, with expensive init. AI agent sandboxes, code interpreters, per-run CI environments, per-request isolated execution, branch-and-explore fan-outs. Anywhere you're paying the same initialization over and over and throwing the result away, caching that result is the obvious move — and a snapshot is what "caching that result" actually looks like.

The hybrid: build an image, boot it once, snapshot the warm state

You don't have to pick. The model that works in practice uses both, each for the half it's good at. Build a container image the normal way — it's the best artifact format we have for describing a filesystem, and you keep your whole existing build, scan, and provenance pipeline. Then, once, boot that image into a microVM, let it fully initialize, wait until it's genuinely ready to serve, and snapshot the warm machine. From then on, restore the snapshot instead of starting from scratch. The image is your build artifact; the snapshot is your start artifact.

That's exactly how PandaStack templates work. A template is defined by a Dockerfile, built like any image, and converted into a root filesystem. The first spawn of a new template is a real cold boot — roughly 3 seconds to reach ready — and at the end of it the agent captures a snapshot automatically. Every create after that restores instead of boots: about 49ms for the snapshot-load step, 179ms p50 and ~203ms p99 end to end. You cold-boot once per template and restore forever after, which puts the expensive part squarely in your build pipeline where it belongs.

from pandastack import Sandbox

# The template was built from a Dockerfile like any image.
# The difference is what create() does with it: it does NOT pull,
# unpack, and exec a cold PID 1. It restores a machine that already
# booted and already finished initializing.
# ~49ms snapshot-load; 179ms p50 / ~203ms p99 end to end.
sbx = Sandbox.create(template="code-interpreter", ttl_seconds=600)

# No readiness loop needed — the probe is part of create.
print(sbx.exec("python -c \"import sys; print(len(sys.modules))\"").stdout)

# Do the expensive init ONCE, then freeze the result.
# This is the whole idea: cache the OUTPUT of initialization,
# not just the input to it.
sbx.exec("pip install -q numpy pandas scikit-learn", timeout_seconds=300)
sbx.exec("python -c 'import numpy, pandas, sklearn'")  # warm the imports
snap = sbx.snapshot()   # guest RAM + device state, imports already resident
sbx.kill()

# Each fork resumes a machine with those modules ALREADY imported.
# Nothing re-imports anything. Same-host fork: 400-750ms
# (cross-host 1.2-3.5s, since the artifacts move over the network first).
workers = [snap.fork(ttl_seconds=300) for _ in range(8)]
for w in workers:
    print(w.exec("python -c 'import pandas; print(pandas.__version__)'").stdout)
    w.kill()

The forking behaviour at the bottom is the part that has no container analogue at all. Because a fork is "restore this specific running machine" rather than "start a fresh copy of this image", the children share the parent's memory pages copy-on-write until they write to them. Eight branches off one warmed-up interpreter cost far less RAM than eight cold starts, and each one begins with the imports, the heap, and the caches already populated. You can't express that with a filesystem artifact, because the thing you want to copy isn't on the filesystem.

The summary

Two ways to start fast, aimed at two different halves of the same timeline. Container image pull optimizes byte delivery — lazy pulling, registry mirrors, pre-pulled caches, peer distribution — and the ecosystem has gotten genuinely excellent at it. But it has a hard floor: after the bytes arrive, your program still starts from nothing and pays full application init, every time. Snapshot restore moves that init off the request path entirely by freezing a machine that already did it — initialized heap, imported modules, warmed JIT, loaded weights — and resuming it mid-instruction. The costs are real and specific: bigger artifacts, staleness that demands a disciplined re-bake cadence, baked-in vCPU and RAM, CPU-model portability that needs templates, and clock and entropy problems on clone that you must handle deliberately rather than discover in production.

If your process starts once and runs for a month, keep pulling images and go work on something that matters. If you start the same expensive environment thousands of times a day and throw it away each time, you're re-running the same initialization on a loop and paying for it every iteration. Build the image, boot it once, snapshot the warm state, restore from then on — and stream the memory on demand so the restore doesn't wait on a multi-gigabyte download, which is just lazy pulling applied to RAM. Measure your own gap between "container running" and "app ready" first. That number decides which half of the problem you actually have.

Every sandbox, managed database, and git-driven app on PandaStack runs on this restore path, and the control-plane API and per-host agent are open source under Apache-2.0 — so you can run them on your own Linux KVM hosts and time the restores yourself. For the create pipeline step by step, see /blog/snapshot-restore-vs-warm-pools; for the memory mechanics, /blog/microvm-snapshotting-warm-starts.