The Security Gotchas of Firecracker Snapshots (Secrets Frozen in RAM)

A Firecracker memory snapshot (vm.mem) is a byte-for-byte freeze of the guest's physical RAM at one instant. That is exactly what makes restore fast — but it is also a security fact people learn the hard way: everything that was in memory at snapshot time is now sitting on disk, and it gets faithfully restored into every clone. TLS session keys, an API token you exported into a process, the entropy pool feeding /dev/urandom, decrypted database rows, the ASLR layout the kernel randomized at boot — if it lived in RAM when you hit the button, it's in the snapshot. Bake a secret into a template snapshot and you've handed it to every sandbox that restores from it. This post is the security model of that: what leaks, the especially nasty CRNG entropy-reuse problem, why snapshot files need protecting at rest, and how a well-built platform injects per-restore identity after restore instead of freezing it in.

Your snapshot is a photograph of your VM's soul, credentials included. The good news is that once you understand which parts of the soul are dangerous to photograph, the mitigations are mechanical.

A snapshot is a literal freeze of RAM

A Firecracker snapshot is two artifacts (plus the rootfs): vm.state, the small serialized VMM and vCPU state, and vm.mem, the guest's entire physical RAM copied byte for byte. A 2 GiB guest produces a 2 GiB memory file. On restore, Firecracker maps that file with MAP_PRIVATE so pages are faulted in lazily and copy-on-write — the guest resumes mid-instruction with its page cache warm and its processes exactly where they were. That resume-not-reboot behavior is the whole performance story (a create lands around 179ms p50, versus roughly 3s for a genuine cold boot).

The security consequence is a direct corollary of the performance one. If restore brings the machine back exactly as it was — processes mid-execution, memory intact — then any secret those processes were holding in memory comes back too. A snapshot is not a sanitized image; it's a memory dump that happens to also run. Whatever a debugger could have read out of that RAM, the restored guest inherits verbatim.

What actually leaks into a shared snapshot

When a snapshot is used as a template — baked once and restored many times, which is exactly how a fast sandbox platform works — anything secret captured at bake time is now shared by every descendant. The categories worth naming explicitly:

• API tokens and credentials — anything a process exported into its environment or read into memory before the snapshot: cloud keys, database passwords, a bearer token you curled with. Env vars live in the process image, which lives in vm.mem.
• TLS session keys and live connection state — session keys, sequence numbers, and buffers for any TLS connection open at snapshot time. Every restore resumes the same session material, and forks that both use it can collide.
• CRNG / entropy state — the Linux kernel's random-number generator internal state and the seed behind /dev/urandom. This is the subtle one, covered in its own section below.
• Decrypted data in memory — rows a database had cached in cleartext, a secrets file you read into a buffer, an in-memory cache of decrypted user data. Encryption at rest doesn't help once it's decrypted into RAM.
• ASLR layout and kernel randomness — the address-space randomization the kernel rolled at boot is frozen. Every restore has identical memory layout, which weakens ASLR as a mitigation across all clones.
• SSH host keys and machine identity — if generated before the snapshot, every restored VM presents the same host key and machine-id, so they're no longer unique per instance.

The unifying rule: the snapshot boundary is a trust boundary. Anything secret or unique that crosses it (by being in RAM at bake time) is no longer secret and no longer unique. The fix is to keep those things out of memory until after the boundary — inject per-restore, don't bake in.

The CRNG entropy-reuse problem

The nastiest gotcha isn't a token you can rotate — it's randomness, because it fails silently. The Linux kernel's cryptographic RNG has internal state. /dev/urandom, getrandom(), and every library that draws from them ultimately pull from that state. A snapshot freezes it. So when you restore the same snapshot N times, all N guests resume with the identical CRNG state — and if nothing reseeds them, they will produce the same sequence of 'random' numbers.

Think about what depends on 'random' being unpredictable and unique: TLS handshake nonces and ephemeral keys, session IDs, CSRF tokens, UUIDs, password salts, JWT signing nonces, cookie secrets. If two forks of the same snapshot each generate a key pair or a nonce from the same starting entropy, they can generate the same key, or reuse a nonce that was only ever supposed to be used once. Nonce reuse in particular breaks entire cipher constructions — it's not a theoretical weakening, it's a full break for schemes like GCM. This is not a PandaStack quirk; it's a documented Firecracker caveat that any snapshot-and-restore system has to handle.

The practical mitigation has two halves. First, give the guest a virtio-rng device so the host can push fresh, host-sourced entropy into the guest after restore. Second, actually reseed on resume — a small early-boot or on-restore hook that pulls fresh bytes and reseeds the kernel pool, so the frozen CRNG state is discarded before any process draws a nonce. Modern kernels also reseed the CRNG when they observe the VM-generation-ID change that Firecracker bumps on restore, which is the belt to the virtio-rng suspenders. The point is: don't let application code draw randomness from a just-restored, un-reseeded pool.

# After a restore, the guest's CRNG state is whatever was frozen in vm.mem.
# Confirm entropy is being refreshed, then force a reseed before any
# process draws a nonce or generates a key.

# 1. A virtio-rng device must be present so the host can feed entropy in.
ls -l /sys/class/misc/hw_random/rng_available   # virtio-rng exposed?
cat /proc/sys/kernel/random/entropy_avail        # pool fill level

# 2. Danger signal: two restores of the SAME snapshot that never reseed
#    will return the SAME bytes here. That must never happen in prod.
head -c 16 /dev/urandom | xxd

# 3. Force a reseed on resume — pull fresh host entropy via the virtio-rng
#    device and stir it into the kernel CRNG (RNDADDENTROPY under the hood).
rngd --rng-device=/dev/hwrng --fill-watermark=0 --once

# 4. Belt-and-suspenders: kernels that honor the VM Generation ID reseed
#    automatically on restore. Verify it changed across the restore.
dmesg | grep -i 'random: .*crng.*reseed'

Snapshot files are unencrypted at rest

vm.mem is a plain file. Firecracker does not encrypt it, and there is no reason it would — it's a raw memory image. That means a snapshot at rest is a cleartext copy of everything that was in guest RAM, sitting on a local disk or in an object-storage bucket. Anyone who can read that file — a misconfigured bucket, an over-broad IAM role, a backup that got copied somewhere, a curious operator — can grep it for tokens, extract key material, or reconstruct decrypted data. Treat every snapshot artifact as a secret of the highest sensitivity of anything the guest ever held.

Concretely: encrypt snapshot storage at rest (bucket-level or envelope encryption), lock down access with least-privilege IAM so only the agents that restore them can read them, keep them off shared or world-readable paths, and expire them on a schedule so a stale snapshot isn't quietly leaking last quarter's credentials. On PandaStack, seed and snapshot artifacts live in access-controlled GCS buckets keyed per generation, and the streaming path fetches chunks over authenticated Range GETs rather than exposing the raw file — but the principle holds for any deployment: the snapshot is only as protected as the bucket it sits in.

The anti-pattern: fork-shares-secrets

Fork makes all of this sharper, because a fork is a copy-on-write clone of a running machine — memory included. If the parent had a secret or a live authenticated session in RAM, every fork inherits the exact same bytes. That's wonderful for sharing a warm page cache and a cloned repo; it's a liability for anything that was supposed to be per-instance unique.

The failure mode is seductive because it looks efficient: authenticate once in the parent, fork ten workers, and now all ten are 'already logged in.' What you've actually built is ten machines sharing one identity, one session key, and — until reseed — one random stream. If those workers each mint a token or open an outbound TLS connection, they can collide in ways that are miserable to debug and occasionally a real vulnerability. The rule is simple: fork shares the machine, so fork shares the machine's secrets. Anything that must be per-fork unique has to be (re)generated after the fork, from freshly reseeded entropy — not inherited.

The fix: inject identity after restore, not before

The whole class of problems dissolves if you invert the order. Don't bake secrets into the template you snapshot; bake a generic, credential-free machine, and inject per-restore identity after the snapshot is restored. The snapshot stays a clean, shareable baseline that's safe to store and safe to clone, and each live sandbox gets its own token, its own keys, its own reseeded entropy — none of which ever touched vm.mem.

In practice that means the template bake does everything that's safe to share — the OS, the runtime, warm caches, installed dependencies — and stops short of anything unique or secret. Then, on each create/restore, the platform writes the per-sandbox secret into the running guest via the guest agent (a file write or an exec) after it's up. The secret is generated fresh per sandbox and lands only in that one live machine's memory, never in the shared snapshot on disk.

from pandastack import PandaStack
import secrets

ps = PandaStack()

# The 'base' template snapshot is a clean, credential-free baseline. It was
# baked WITHOUT any tokens in RAM, so vm.mem on disk holds no secrets and is
# safe to store and clone. Create restores that clean snapshot.
sb = ps.sandboxes.create(template="base", ttl_seconds=3600)

# Generate a per-sandbox secret AFTER restore. It is unique to THIS sandbox
# and only ever lives in this one live guest's memory — never in the
# shared snapshot, so no other restore can ever see it.
token = secrets.token_urlsafe(32)
sb.filesystem.write("/run/secrets/token", token)

# Same idea for any per-instance identity: mint it now, inject it now.
sb.exec("ssh-keygen -A")                      # fresh, per-VM SSH host keys
sb.exec("systemd-machine-id-setup --commit")  # unique machine-id per restore

# If you fork this sandbox, do the SAME thing in each child: regenerate
# anything that must be unique, because the fork inherited the parent's RAM.
child = sb.fork()
child.filesystem.write("/run/secrets/token", secrets.token_urlsafe(32))
child.exec("ssh-keygen -A && systemd-machine-id-setup --commit")

This is the same discipline as the entropy reseed, generalized: treat the snapshot as a shared, non-secret substrate, and treat identity — tokens, keys, machine-id, random state — as something you mint per live instance, after the restore boundary. Bake the machine; inject the soul. Do that, and 'a snapshot is a photograph of your VM's RAM' stops being a warning and becomes just a fast way to boot.

The summary

A Firecracker snapshot is a byte-for-byte freeze of guest RAM, so anything secret in memory at bake time — tokens, TLS keys, decrypted data, and the kernel's CRNG state — is written to disk and restored into every clone. The entropy case is the most dangerous because it fails silently: restores that don't reseed can emit duplicate 'random' values, weak keys, and reused nonces, exactly as Firecracker's docs warn, which is why you expose virtio-rng and force a reseed on resume. Snapshot files are unencrypted at rest, so protect them like the secrets they contain. And the durable fix for all of it is ordering: bake a clean, credential-free template, and inject per-restore identity and reseeded entropy after the snapshot comes back — never before. PandaStack's core is open source under Apache-2.0, so you can read exactly how the restore path and per-sandbox injection work on your own hosts.