Firecracker Snapshot-Restore vs AWS Lambda SnapStart

AWS Lambda SnapStart and Firecracker snapshot-restore attack the same enemy — the cold start — with the same core idea: instead of paying to initialize from scratch every time, capture a pre-initialized snapshot once and restore from it. But they capture different things at different layers of the stack, and that difference decides what you get back, what stays fast, and what quietly breaks. SnapStart snapshots a fully initialized function execution environment and restores it per scaling event so a request skips the expensive init phase. Firecracker snapshots an entire booted microVM — guest kernel, all of guest RAM, and every device's state — and restores the whole machine via copy-on-write memory. This post compares the two honestly: what each actually captures, the shared 'uniqueness' footgun that AWS itself warns about (restored snapshots reuse state, randomness, and seeds), the frozen-clock and entropy caveats that fall out of a whole-VM restore, the copy-on-write and on-demand page-in that keep restore cheap, and where PandaStack sits — restoring on every create, not just on cold starts. Lambda specifics are described qualitatively; verify them against the current AWS docs, since SnapStart's language coverage and behavior evolve.

Same idea, different layer

Both systems answer 'why re-do expensive setup every time?' with 'don't — freeze the result once and thaw it.' The layer they freeze at is the whole difference. SnapStart operates at the function-runtime layer: it runs your function's initialization once, snapshots the initialized execution environment (the process and its warmed-up runtime), and restores from that snapshot when Lambda needs a fresh environment to serve traffic — so the init code doesn't run again on the restore path. Firecracker operates at the hypervisor layer: it freezes a running microVM at one instant — the guest kernel, its physical RAM byte for byte, and its emulated device state — and restore resumes that machine mid-instruction. One thaws a warmed-up function; the other thaws a warmed-up computer.

What SnapStart captures: an initialized function environment

The expensive part of a cold start is usually initialization: for a JVM-based function that's classloading, JIT warm-up, framework bootstrapping, dependency wiring — work that can take seconds before the first request is ever handled. SnapStart was introduced first for Java/JVM functions precisely because that init cost is so pronounced there. The idea is to run that initialization phase once at deploy/publish time, snapshot the fully initialized execution environment, cache the snapshot, and then restore from the snapshot when a new environment is needed to serve requests — so the per-scaling-event path resumes an already-initialized function instead of building one from cold. The mechanics and current language/runtime coverage are AWS's to define and change; verify against the AWS docs. The shape that matters here: capture happens at the function-runtime level, and restore skips re-running your init.

Because the snapshot is of an initialized environment rather than a whole machine, AWS exposes runtime hooks so your code can react to the snapshot lifecycle — do work before the environment is snapshotted, and do work after it's restored. Those hooks exist for a reason, and that reason is the footgun below: some state simply must not be captured-and-reused, and must be regenerated after each restore.

What Firecracker captures: the whole microVM

A Firecracker snapshot is a running microVM frozen at a single instant, serialized to a small fixed set of files: vm.mem is the guest's entire physical RAM byte for byte; vm.state is the VMM's serialized device and CPU state — vCPU registers, the in-kernel interrupt controller, the clock, and every virtio device's queues and configuration; and the rootfs is the disk the guest was running against. Restore is not a boot. The host launches a fresh Firecracker process, maps vm.mem, loads vm.state, and resumes the vCPUs — the guest picks up mid-instruction with its page cache warm, processes still running, files still open. No init, no device re-probe, no systemd handshake. Where SnapStart resumes an initialized function, Firecracker resumes an entire initialized operating system with your workload already running on it.

The consequence is that Firecracker restore is workload-agnostic. SnapStart's benefit is proportional to how heavy your init phase is — a function with trivial init gets little from it. Firecracker doesn't care what's running inside the guest; it maps memory and resumes the machine regardless, so a Python REPL, a Node build server, and a Postgres instance all restore through the identical path. The mechanics of how that memory map stays cheap live in /blog/firecracker-memory-snapshots.

The uniqueness footgun both systems share

Here's the comedy-gold part, and it's the same joke told at two different layers. When you restore many environments from one snapshot, every restored copy starts byte-identical — including things a program assumed were unique because it generated them once at startup. AWS explicitly warns about this for SnapStart: state established during initialization is captured in the snapshot and shared by every restored environment, so anything that must be unique per environment can't be created during init and left alone. The canonical example is randomness. If your init phase seeds a pseudo-random generator, that seed is now baked into the snapshot, and every restored environment inherits the identical seed — so they all produce the same 'random' sequence. Restore a thousand environments and you get a thousand copies rolling the same dice. It is funny right up until it's your session tokens, your nonces, or your UUIDs colliding across every instance in production.

SnapStart's answer is the runtime hooks: reseed your RNG and regenerate anything unique in the after-restore hook, and pull anything sensitive out before the snapshot. AWS also documents that its supported runtimes take steps to keep the built-in secure-randomness source safe across restore — but application-level randomness and any state your own init code cached are on you. Whole-VM restore has the exact same hazard, just relocated. A Firecracker snapshot freezes the guest kernel's entropy pool and every process's in-memory state, so every restore of that snapshot inherits identical starting entropy and identical secrets-in-RAM unless the guest reseeds after resume. Same footgun, one layer down: instead of one function's RNG, it's the whole guest's.

And the clock is frozen too

The uniqueness problem has a time-shaped sibling. A snapshot freezes not just entropy but the clock. A restored environment wakes up believing it is the exact instant it was captured — which is fine for the first restore moments after baking and increasingly wrong the longer the snapshot sits. In a Firecracker guest this bites hard: the guest clock is frozen at bake time, so a restored guest that skips a clock resync will happily reject or mint TLS certificates against the wrong wall-clock, expire tokens early or late, and fire timers on a stale sense of 'now.' The fix on the VM side is to resync the guest clock on resume; on the function side the same class of issue applies to anything that cached the current time during init. The general law: a snapshot captures a moment, and every restore re-lives that moment until you tell it otherwise.

The comparison, dimension by dimension

Side by side, the trade-offs follow from the layer each one snapshots at.

• Unit captured — Firecracker snapshot-restore: an entire microVM — guest kernel, all of guest RAM (vm.mem), and every emulated device's state. Lambda SnapStart: a fully initialized function execution environment (originally the JVM story), so restore skips the init phase.
• What restore skips — Firecracker snapshot-restore: the entire boot — kernel, device probe, userspace init — because the guest was already running when frozen. Lambda SnapStart: your function's initialization work, which is where JVM-class cold-start cost concentrates.
• Isolation boundary — Firecracker snapshot-restore: the guest resumes behind its own kernel and KVM hardware virtualization — a VM-strength boundary, restored intact on every create. Lambda SnapStart: the managed Lambda execution-environment boundary (itself Firecracker-based under the hood), operated by AWS.
• Who runs it — Firecracker snapshot-restore: you, on your own Linux KVM hosts (PandaStack's core is open source under Apache-2.0). Lambda SnapStart: a fully managed AWS feature you enable on a function; the snapshot lifecycle is AWS's to run.
• The uniqueness footgun — Firecracker snapshot-restore: every restore inherits identical guest entropy, in-RAM secrets, and a frozen clock; the guest must reseed and resync on resume. Lambda SnapStart: state captured during init is shared across restores; AWS provides runtime hooks to reseed/regenerate and warns explicitly about duplicated randomness.
• Lazy / on-demand memory — Firecracker snapshot-restore: maps vm.mem copy-on-write (MAP_PRIVATE) and can page it in on demand via userfaultfd, even streamed from object storage. Lambda SnapStart: memory restore is managed internally by AWS; the developer-visible contract is 'init is skipped,' not the paging mechanism.
• When it triggers — Firecracker snapshot-restore: on every create, by design — there is no warm pool. Lambda SnapStart: on the cold-start / scale-up path, to remove init latency when a new environment is needed.

Why the restore stays cheap: copy-on-write memory

The naive fear about restoring a whole VM is 'you have to read all of guest RAM off disk first.' You don't. When Firecracker loads the memory file, it maps vm.mem with MAP_PRIVATE — a lazy, copy-on-write mapping at page granularity. Nothing is eagerly copied at load time; the mapping just declares that the guest's physical RAM is backed by the file. A read fault on an untouched page is resolved by mapping that 4 KiB page straight from the file with no copy, and multiple guests restoring the same snapshot share those identical pages in the page cache. A write fault triggers the copy-on-write: the kernel makes a private 4 KiB copy of just that page, and the original stays pristine for everyone else. So a create pays, in time and RAM, only for the working set the guest actually touches between resume and ready — not for the declared memory size. It's the same trick that makes Unix fork() cheap, applied to an entire VM's RAM.

When the memory file isn't even local — a fresh agent, or one that's never served a template — Firecracker can be handed a userfaultfd instead of a local file, and the agent backs guest RAM itself. The guest faults on a page, the kernel parks the vCPU and posts the fault to the agent, which fetches the surrounding 4 MiB chunk from object storage over an HTTP Range GET and installs it with UFFDIO_COPY. All-zero chunks are elided with no fetch, and a prefetch trace baked at snapshot time warms the hot set into a shared per-host cache. The guest only ever pulls the pages it touches, and never learns a page came over the network. Scope matters: userfaultfd streams memory, not disk — the rootfs stays a local file because copy-on-write disk cloning needs a local block device. The full mechanics are in /blog/userfaultfd-explained.

Where PandaStack sits: restore on every create

SnapStart uses snapshot-restore to remove the init tax on cold starts — a targeted optimization on a specific slow path. PandaStack makes snapshot-restore the only path. There is no warm pool of idle VMs; every single create restores a baked Firecracker snapshot on demand. That's viable precisely because the copy-on-write memory map above makes restore cheap enough to do constantly rather than occasionally. The numbers: a create lands the snapshot-load step at roughly 49ms, with the whole pipeline at a 179ms p50 and about 203ms p99. Only the first-ever spawn of a template on an agent pays a real cold boot — on the order of 3 seconds — to produce the snapshot in the first place, after which every create restores. Fork rides the same primitive: same-host forks land in 400–750ms, cross-host forks in 1.2–3.5s once artifacts move over the network. A managed Postgres database, which blocks on real bootstrap, takes 30–90s. Networking is pre-provisioned as 16,384 /30 subnets per agent so the network slot is warm before restore begins. The full create pipeline is in /blog/snapshot-restore-boot-path.

# The agent drives Firecracker's HTTP API over its Unix socket to restore
# a baked microVM snapshot into a fresh VMM. This is the core of every create.

# Load the snapshot: map vm.mem copy-on-write and load device/CPU state.
curl --unix-socket /run/fc.sock -X PUT 'http://localhost/snapshot/load' \
  -H 'Content-Type: application/json' \
  -d '{
        "snapshot_path": "/seed/base/vm.state",
        "mem_backend": { "backend_type": "File",
                         "backend_path": "/seed/base/vm.mem" },
        "resume_vm": false
      }'

# Resume the vCPUs: the guest picks up mid-instruction, no boot, no init.
curl --unix-socket /run/fc.sock -X PATCH 'http://localhost/vm' \
  -H 'Content-Type: application/json' \
  -d '{ "state": "Resumed" }'

From the SDK, all of that is one call, and the same snapshot primitive powers snapshot and fork — so you get the fast-restore win and the duplicate-state caveat in the same place. Because every fork is a fresh copy-on-write clone of a frozen machine, the uniqueness footgun applies: if you fork ten sandboxes off one snapshot and each expects a unique random seed, reseed after the fork or they'll all roll the same dice — the exact SnapStart warning, one layer down.

from pandastack import Sandbox
import secrets

# create() restores a baked microVM snapshot on demand: no init, no warm pool.
# p50 ~179ms (~49ms of that is the snapshot-load step), ~203ms p99.
# First-ever spawn of a template cold-boots (~3s) then bakes; every create after
# that restores.
box = Sandbox.create(template="base", ttl_seconds=3600)

# fork() restores a copy-on-write clone of this frozen machine.
# ~400-750ms same-host; 1.2-3.5s cross-host (artifacts move over the network).
branch = box.fork()

# The uniqueness footgun, PandaStack edition: the clone inherits the parent's
# frozen entropy and in-RAM state. If each branch needs its own randomness,
# reseed AFTER the fork -- don't trust a seed baked into the snapshot.
branch.exec(f"echo {secrets.token_hex(16)} > /run/fresh-seed")
print(branch.id, "reseeded after fork")

None of this makes SnapStart the wrong tool — for a Lambda function with a heavy init phase running on AWS's managed platform, enabling SnapStart to skip that init is exactly right, and the runtime hooks give you a clean place to handle the uniqueness and clock caveats. The point is that both systems are the same bet — restore a pre-initialized snapshot instead of initializing from cold — placed at different layers, and both inherit the same duplicate-state, frozen-clock hazards that come with restoring the same frozen thing many times. PandaStack pushes the bet all the way down to the whole VM and all the way out to every create, because it's running untrusted, arbitrary code and needs a VM-strength boundary on every restore, not just a faster init.

The summary

AWS Lambda SnapStart snapshots a fully initialized function execution environment (originally the JVM cold-start story) and restores from it so a scaling event skips your init phase — a managed optimization on the cold-start path, with runtime hooks to handle the caveats (verify specifics against the AWS docs). Firecracker snapshot-restore snapshots an entire booted microVM — guest kernel, all of guest RAM, device state — and restores the whole machine via copy-on-write memory, workload-agnostic and behind a fresh guest kernel and KVM. Both share the same footgun, told one layer apart: every restore of a snapshot starts byte-identical, so duplicated random seeds, reused in-memory secrets, and a frozen clock will bite unless you reseed and resync after thaw — the joke AWS warns about for SnapStart is exactly the joke a Firecracker guest tells with its entropy pool. Both can lazily materialize memory: Firecracker maps vm.mem copy-on-write and can stream it via userfaultfd. PandaStack makes restore the only path — 179ms p50, ~203ms p99, no warm pool — and the core is open source under Apache-2.0, so you can run the control-plane API and per-host agent on your own Linux KVM hosts and watch the restore path yourself. For the memory mechanics start with /blog/firecracker-memory-snapshots; for the create pipeline, /blog/snapshot-restore-boot-path; for streaming, /blog/userfaultfd-explained.