Golden Images vs Snapshot Baking

Two techniques get sold with the same three words — fast, identical, every time — and people reach for whichever one their tooling already has. A golden image is a disk you built once and stamp out forever. A snapshot is a machine you booted once and resume forever. They overlap enough that the confusion is reasonable, and they differ in exactly the places that produce a strange bug six months later.

I'm Ajay; I build PandaStack, a Firecracker microVM platform where every sandbox create is a snapshot restore, so read this as opinionated. It is also confession-shaped: most of the sharp edges below are things I learned by shipping them. The short version is that a golden image removes install time and a snapshot removes boot time, those are different costs, and the correct architecture for most people is both, layered — with the image as your supply chain and the snapshot as your latency.

What is actually inside each artifact

Start with the contents, because every difference downstream falls out of them. A golden image is a filesystem: packages installed, files laid down, a kernel and an init system sitting there waiting. That is all it is. When you launch it, a real boot happens — firmware or a direct kernel load, kernel init, device probing, mounting the root filesystem, then userspace: systemd or your init walking a dependency graph, starting sshd, starting your runtime, running whatever first-boot logic you wrote. The image removed the apt-get. It did not remove one microsecond of any of that.

A snapshot contains the filesystem too, and then it contains the parts the image never had: the full contents of guest RAM, the vCPU register state of every core, and the state of every emulated device — the virtio queues, the interrupt controller, the block and network device configuration. It is a machine caught mid-sentence. The boot already happened, once, at bake time, and what you are storing is the outcome of that boot rather than the ingredients for another one.

Which means restoring is not starting. Nothing initializes. The kernel does not probe devices; it already probed them and remembers the result. Your Python interpreter does not import its standard library; the objects are already on the heap, at the same addresses, with the same reference counts. The JIT does not re-warm. You are not booting a machine that resembles the one you baked — you are continuing the one you baked.

# Layer 1: the golden image. Boring, reproducible, auditable -- this is the
# artifact your security team should have to reason about, and no other.
cat > Dockerfile <<'EOF'
FROM ubuntu:24.04
RUN apt-get update && apt-get install -y --no-install-recommends \
      ca-certificates curl git openssh-server python3.12 && \
    rm -rf /var/lib/apt/lists/*
COPY pandastack-init /usr/local/bin/pandastack-init
EOF

docker buildx build --platform linux/amd64 -t base:2026-09-08 --load .

# Flatten the image to a filesystem, then to an ext4 disk. This is the
# "Packer output" equivalent: a disk, and nothing whatsoever but a disk.
cid=$(docker create base:2026-09-08)
mkdir -p rootfs && docker export "$cid" | tar -C rootfs -xf -
docker rm "$cid"

truncate -s 2G rootfs.ext4
mkfs.ext4 -d rootfs -F rootfs.ext4
sha256sum rootfs.ext4 | tee rootfs.ext4.sha256   # your supply chain lives here

# Note what you now have and what you do not. You have every package you will
# ever need, pinned and hashed. You have not saved a single millisecond of
# kernel init, device probe, systemd dependency graph, or sshd startup.

Install time and boot time are different costs

This is the whole argument, so it is worth being blunt. Install time is minutes and you pay it at build time, once, for everybody. Boot time is seconds and you pay it at run time, every time, per instance. A golden image is a fantastic answer to the first and a complete non-answer to the second, which is fine if you launch instances hourly and irrelevant if you launch them per request.

Concretely, on our own numbers: a PandaStack sandbox created from a baked snapshot lands at 179ms p50 and around 203ms p99, end to end — allocate networking, reflink the rootfs, exec Firecracker, load the snapshot, resume, probe the port. The first spawn of a template, before a snapshot exists, is a cold boot at roughly 3 seconds. Same image. Same packages. Same kernel. The only variable is whether somebody already performed the boot on your behalf.

A golden image means you never install anything twice. A snapshot means you never boot anything twice. If your workload is a request, the second one is the one your users can feel.

The 3 seconds versus 179ms gap is not a benchmark trick; it is the same work, moved. Somebody still paid the kernel init and the systemd graph — it was me, once, at bake time, and the result is a file. That reframing is the entire technique: boot is a computation, its output is a memory image, and computations whose inputs never change should be cached.

# Layer 2: bake a snapshot from a booted instance of that image.
# Firecracker's API is a Unix socket that speaks JSON. That is all of it.
API=/run/firecracker.sock
fc() { curl -s --unix-socket "$API" -H 'Content-Type: application/json' "$@"; }

# --- BAKE: pause the running guest, then freeze it whole ---
fc -X PATCH 'http://localhost/vm' -d '{"state": "Paused"}'

fc -X PUT 'http://localhost/snapshot/create' -d '{
  "snapshot_type": "Full",
  "snapshot_path": "/seeds/base/vm.state",
  "mem_file_path":  "/seeds/base/vm.mem"
}'
# vm.state is small: device state and vCPU registers.
# vm.mem is the guest's RAM, byte for byte, including whatever happened to be
# in it at this instant. Hold that thought until the "secrets" section.

# --- RESTORE: a different firecracker process, minutes or weeks later ---
fc -X PUT 'http://localhost/snapshot/load' -d '{
  "snapshot_path": "/seeds/base/vm.state",
  "mem_backend": { "backend_type": "File", "backend_path": "/seeds/base/vm.mem" },
  "enable_diff_snapshots": false,
  "resume_vm": false
}'

fc -X PATCH 'http://localhost/vm' -d '{"state": "Resumed"}'
# The guest did not boot. It continued, from the instruction after the pause,
# convinced that no time has passed. It is correct about the instruction and
# wrong about very nearly everything else.

The things a snapshot freezes that a disk image does not

Here is the section I would want if I were evaluating this. A golden image freezes files, and files are inert; the interesting state gets created fresh on every boot. A snapshot freezes that interesting state too, and hands the same copy of it to every machine you ever restore. Five categories, roughly in order of how badly they bite.

Entropy: every restore resumes with the same RNG

The guest kernel's random pool is memory. It is in the snapshot. So every guest restored from one snapshot resumes with a byte-identical CSPRNG state, and until something reseeds it, identical reads produce identical bytes. Restore that snapshot a thousand times and you have a thousand machines that agree on what random means.

Follow that into your application layer and it stops being a curiosity. Anything that generates a key pair, a session token, a CSRF token, a UUID4, a password salt, a nonce, or a TLS ephemeral key on a freshly restored guest is drawing from a pool that another tenant's guest also has. Two sandboxes minting the "same" session token is not a collision you can hash your way out of; it is the same secret, issued twice, by design.

The fix is that the guest must get fresh entropy at resume, before userspace does anything cryptographic. Firecracker exposes a virtio-rng device for exactly this, and there is a VM generation ID mechanism whose whole job is to tell a guest kernel that it has been restored so it can reseed — but guest-side support for that is comparatively recent, and our templates run the 5.10 kernel, so I do not rely on it. Reseed explicitly on resume and treat it as part of the restore path, not as a nice-to-have.

The clock: a restored guest wakes believing it is bake time

This one is a genuine production lesson and I will state it as one: we shipped it, it broke, we fixed it. A resumed guest continues its timekeeping from the instant of the pause. Wall clock, monotonic clock, every timer — all of it picks up where it left off. Bake a template on Monday, restore it on Friday, and the guest sincerely believes it is Monday.

The failure mode is not a wrong log timestamp, which is what everybody expects and which nobody would page for. It is TLS. Certificate validation is a time comparison, so a guest whose clock is days behind starts rejecting perfectly good certificates as not yet valid, and a guest whose clock is far enough off in the other direction sees expired ones. What reaches you is an application that cannot talk to an API it talked to yesterday, with a certificate error naming a certificate that is fine. You will check the certificate first. Everyone does.

The fix is an explicit clock resync on every restore, resume and wake — push the host's real time into the guest before userspace is allowed to open a TLS connection. It is a small piece of code in the restore path and it belongs there rather than in a boot-time service, because on this path there is no boot for a boot-time service to hook.

Baked identity: the network the guest thinks it has

The guest's IP address, MAC address, routing table and ARP cache are also memory. They freeze. A restored guest does not ask for an address; it already has one, the one it had at bake time, and it will happily emit packets from that address on a host that has never heard of it.

So the restore path has to make the host's networking match what the guest already believes, rather than the other way round. PandaStack pre-allocates per-sandbox network namespaces — 16,384 /30 subnets per agent, each with a veth pair and a TAP device — and on restore patches the TAP's MAC and the host-side routes to match the identity frozen in the template's metadata. The guest is never told it moved. It is a small conspiracy, maintained on its behalf, and the alternative is renegotiating an address on a machine that thinks it did that days ago.

Baked sizing: RAM and vCPU are properties of the snapshot

A golden image is indifferent to machine size: the same AMI boots on a small instance or a huge one, because sizing is decided at boot. In a snapshot, the memory image is the memory — a 4 GiB snapshot is a description of 4 GiB of RAM, and there is no coherent way to hand the restored guest 8 GiB, because the guest's page tables, its kernel's memory maps and its allocator's idea of the world were all built for the size it had. vCPU count is the same story: the register state you saved is per-core.

The practical consequence is worth internalising because it reshapes your API: memory becomes a template decision, not a per-instance knob. On PandaStack, the base template is baked at 4 GiB, and the agent overrides a create request's CPU and memory to match the baked snapshot rather than pretending to honour them. If you want a different size, you bake a different template. I would rather say that plainly than accept a parameter and quietly ignore it, which is what the honest version of this constraint looks like from the outside.

Secrets: RAM at bake time is RAM forever

The one that should make you careful. Whatever is in guest memory when you take the snapshot is in the snapshot: environment variables of running processes, a token an agent fetched during startup, a private key an sshd loaded, a database password sitting in a connection pool, the decrypted contents of a config file. That memory image is then copied to every host that pulls the template and mapped into every guest restored from it, including other tenants'.

There is no rotation story for this, because there is no place to rotate. The secret is not in a file you can rewrite; it is at some offset in a multi-gigabyte memory image replicated across your fleet, and the only real remediation is revoking the credential and re-baking. Bake from a pristine state, inject secrets at run time, and treat "did anything privileged touch this VM before the snapshot" as a checklist item on your bake pipeline rather than a thing you remember.

Rebuild cadence and invalidation

Both artifacts go stale; they go stale on different clocks, and the second clock is the one people forget to wind. A golden image is rebuilt when the contents should change: a CVE in a base package, a runtime version bump, a new agent binary. That cadence is well understood — it is the whole reason Packer pipelines and weekly image builds exist.

A snapshot must be re-baked whenever the template changes, and the old snapshots must be actively invalidated, because a stale snapshot is not merely out of date — it is a running machine from a previous version of your world, and it will restore happily and serve traffic. This is the failure I would watch for hardest in a home-grown setup: you patch the image, redeploy, and the fast path keeps restoring last month's memory. On PandaStack, re-baking a template invalidates the snapshots derived from it and the next spawn falls back to a cold boot plus an auto-bake — you eat about 3 seconds once, then you are back at 179ms on the new contents.

The honest, slightly annoying corollary: anything that is a bake-time property requires a re-bake to change, and existing snapshots never retroactively acquire it. Guest RAM is the obvious one. So is the memory backing — we back cold-booted guest memory with 2 MiB hugepages when it is enabled, which cuts page faults on restore by a large factor, and snapshots taken from a 4 KiB-paged guest simply stay 4 KiB-paged forever. Flipping the flag changes nothing until you re-bake. Plan template changes as builds with a rollout, not as configuration you toggle.

Four ways to start a machine, compared

Cost here means what you pay to produce and store the artifact, not dollars. The only latency figures are PandaStack's own.

• Golden image only — Cost: one build per change, cheap storage, mature tooling. Cold-start: a full boot every launch, seconds, every time, forever. Reproducibility: excellent and auditable — a recipe plus a hash, easy to diff and to sign. Freezes: files only. Kernel state, RNG, clock and network identity are created fresh on each boot, which is exactly why none of the snapshot hazards apply.
• Snapshot only — Cost: you need a machine to snapshot, and if there is no reproducible recipe behind it you are hand-crafting a pet and freezing it. Cold-start: the fast one — 179ms p50 and about 203ms p99 for us. Reproducibility: identical outputs, terrible provenance; a multi-gigabyte memory image tells you nothing about what is in it. Freezes: everything, including the five hazards above.
• Image plus snapshot, layered — Cost: two pipelines, and the second one is genuinely more work to operate. Cold-start: the same 179ms p50, with roughly 3 seconds paid once at bake. Reproducibility: the best available — provenance from the recipe, determinism from the snapshot, and a re-bake regenerates the snapshot from an artifact you can hash. Freezes: everything a snapshot does, but from a known, reviewable starting point. This is the answer for most people.
• Neither, boot from scratch each time — Cost: nothing to build or store, which is its real and underrated appeal. Cold-start: install time plus boot time on every single launch, so tens of seconds to minutes. Reproducibility: whatever your package mirror felt like this morning; the classic "worked last Tuesday" build. Freezes: nothing, which is the one honest advantage — no stale artifact can betray you if there is no artifact.

The layered answer, and what it looks like to use

Build the image the boring reproducible way — Dockerfile or Packer, pinned versions, hashed output, checked into something. Then boot exactly one instance of that image, let it finish coming up, and snapshot it. Publish the pair. The image is your supply chain: it is what you audit, scan, sign and roll back. The snapshot is your latency: it is a derived artifact, regenerated by a pipeline, and nobody should ever hand-edit it or treat it as a source of truth.

The division of labour is the point. When a CVE lands you fix the recipe, not the snapshot. When boot is slow you fix the snapshot, not the recipe. When someone asks what is in production you answer from the recipe, because the memory image cannot answer that question and never could.

from pandastack import Sandbox

# All of the above collapses into this. "template" names the layered pair:
# a reproducibly built rootfs, plus a snapshot baked from one boot of it.
# Every create is a restore -- ~179ms p50, ~203ms p99. There is no warm pool
# behind this; nothing was sitting idle waiting for you.
sbx = Sandbox.create(template="base", ttl_seconds=300)

# The guest's uptime is inherited from bake time -- it never booted for you.
print(sbx.exec("uptime -p").stdout)

# The clock is NOT inherited, because the restore path resyncs it. That is a
# deliberate correction, not a property of snapshots. Verify it in your own
# stack before you let a restored guest do TLS.
print(sbx.exec("date -u").stdout)

sbx.filesystem.write("/app/main.py", "print('a machine that never booted')")
r = sbx.exec("python3 /app/main.py")
print(r.stdout, r.exit_code)

sbx.destroy()

# Want more RAM than the template was baked with? You cannot have it here,
# and any API that says otherwise is either cold-booting you or lying. Bake a
# different template -- sizing is a build-time decision in this model.

What we do not do, so the tradeoff is visible: we do not let you resize a restored guest, we do not pretend a snapshot is auditable, and we do not claim the hazards above are solved rather than handled. Entropy and clock are fixed in the restore path because they must be. Secrets are a discipline, enforced by how you bake, and no platform can enforce it for you.

The summary

A golden image freezes a disk and removes install time. A snapshot freezes a running machine — disk plus RAM plus vCPU plus device state — and removes boot time. Different costs, different artifacts, and the reason people conflate them is that both make launches faster and more identical, just at different points on the timeline.

The price of the snapshot is that identical goes further than you wanted. The RNG is identical. The clock is identical, and days behind. The IP and MAC are identical. The size is fixed. And anything in RAM at bake time ships to every host that pulls the template. Handle all five in the restore path and in your bake discipline, and you get 179ms instead of 3 seconds. Ignore them and you get a system that is fast, reproducible, and occasionally issues two customers the same session token.

So: both, layered. Build the image the boring way, bake the snapshot from a booted instance of it, and keep straight which one you fix when. The image is what you can explain to an auditor. The snapshot is what your users experience as the product being instant.