The Snapshot Clone Randomness Problem

Fork one snapshot into fifty microVMs and you get fifty machines that agree on what a random number is — which is the one thing they should never agree on. I'm Ajay; I build PandaStack, an open-source Firecracker microVM platform where snapshot-restore is the normal path for every single create, so this is not an abstract worry for me. It's the failure mode I have to design against on the hottest code path in the system. This post is about exactly what gets frozen into a snapshot, why the resulting collisions are a genuine security bug rather than a curiosity, and which layers the standard fix actually repairs — because it repairs fewer of them than most people assume.

The frustrating part is that nothing here looks broken. There's no crash, no error, no failed assertion. Two clones produce the same session token and both of them are behaving exactly as designed. Determinism is normally the thing you want from a snapshot: restore the image, get the same machine. Randomness is the one subsystem where 'get the same machine' is catastrophic, and it's the subsystem nobody puts in the restore checklist.

What a snapshot actually freezes

A memory snapshot is not a description of a machine. It's the machine's RAM, byte for byte, plus device and vCPU state. Anything a running process had computed and was holding in memory at snapshot time comes back on restore identical in every clone. When you enumerate what that covers for randomness specifically, the list gets uncomfortable fast — because it spans four different layers, and each one has a different fix.

• The kernel CSPRNG state. The internal state behind `getrandom()`, `/dev/urandom`, and `/dev/random` lives in kernel memory. Snapshot it and every restore starts from the same generator, at the same position, about to emit the same stream.
• Per-process userspace PRNG seeds. Python's `random` module, glibc's `rand()`, Node's V8 `Math.random()` internal state, Go's old top-level `math/rand` source, a Java `Random` seeded at startup — all of these live in the process's own heap. If the process was already running when you snapshotted, its seed is baked in.
• Cached identifiers. A UUID generated once at startup and stashed in a module-level variable. An instance ID. A worker ID. A `boot_id`. A per-process nonce prefix that some library computed at import time and now assumes is unique to this machine.
• In-flight TLS and session state. A connection mid-handshake, a client random already generated, a session ticket key loaded into a server's memory, a connection pool holding established sessions. Restore the snapshot and every clone believes it owns that same session.
• Anything that read randomness at boot and then cached it. SSH host keys generated on first boot and written to disk are the classic one — but the same applies to a service that generated a signing key at startup and kept it in memory, or a token cache warmed before the snapshot was taken.

You can see the frozen state from inside a guest with two files that have no business being identical across independent machines. The `boot_id` is supposed to be a fresh UUID per boot — but a restore isn't a boot, so unless something explicitly refreshes it, every clone reports the same one. It's the cheapest possible smoke test for 'did anything notice I was cloned?'

#!/usr/bin/env bash
# Run this in EVERY clone right after restore, then diff the output.
# If the boot_ids match across clones, nothing told the guest it was cloned.
set -euo pipefail

echo "host:      $(hostname)"

# Supposed to be a fresh UUID per boot. A restore is not a boot, so a
# snapshot-cloned guest happily reports the SAME boot_id in every clone
# until something regenerates it.
echo "boot_id:   $(cat /proc/sys/kernel/random/boot_id)"

# Kernel's entropy estimate. Frozen into the snapshot like everything else.
echo "entropy:   $(cat /proc/sys/kernel/random/entropy_avail)"

# The value that actually matters: a fresh draw from the kernel CSPRNG.
# These MUST differ across clones. If they don't, stop and fix the reseed
# path before anything in these guests generates a key or a token.
echo "getrandom: $(python3 -c \"import os; print(os.urandom(16).hex())\")"

# The VMGenID counter, if your kernel exposes it. A CHANGE in this value is
# the hypervisor telling the kernel 'you were cloned, distrust your state'.
cat /sys/devices/virtual/misc/vmgenid/generation_id 2>/dev/null \
  || echo "vmgenid:   (not exposed on this guest)"

Why this is a security bug, not a curiosity

It's tempting to file this under 'interesting virtualization trivia.' It isn't. Duplicate randomness is a direct path to three distinct classes of real vulnerability, and they get progressively worse.

The mildest is duplicate identifiers. Two clones generate the same 'random' UUID for a row and your database rejects the insert on a unique constraint — or worse, doesn't have that constraint and now two different objects share an ID. This one at least tends to announce itself, because databases are opinionated about primary keys. Consider it the friendly warning shot.

The middle tier is duplicate session material. Two clones serving two different tenants each mint a session token from the same PRNG stream, and now tenant A's token is a valid credential for tenant B's session — or the same password-reset token lands in two inboxes. Nothing errors. Nothing logs. You have handed one user another user's session and the only trace is a support ticket six months later that reads 'I logged in and saw someone else's account,' which you will spend a week failing to reproduce because you cannot reproduce it without cloning the same snapshot twice.

The severe tier is signature nonce reuse, and it deserves its own paragraph because it is not a degradation — it's a total break. ECDSA and DSA require a per-signature secret nonce. Sign two different messages with the same private key and the same nonce, and the private key falls straight out of the two signatures with a few lines of arithmetic. This is not theoretical: it's the exact bug that broke the PlayStation 3's code-signing key and has drained real cryptocurrency wallets. Now picture two clones of one snapshot, each holding an identical PRNG state, each signing a different message. Congratulations — you have published your private key in two pieces and posted both of them.

There's a fourth, quieter category worth naming: state that isn't secret but is assumed unique. Distributed ID generators (Snowflake-style) that derive a worker ID from randomness at startup. Retry jitter that's supposed to decorrelate a thundering herd but now fires in perfect lockstep across fifty clones. Sampling decisions that are meant to be independent. None of these leak a key, but all of them silently violate an assumption your system was built on, and the failure shows up as a load spike or a duplicate-key storm nobody can explain.

VMGenID: how the guest learns it was cloned

The clean fix for the kernel layer is a VM Generation ID. The mechanism is simple and rather elegant: the hypervisor exposes a 128-bit value in guest memory, and it changes that value whenever the VM is restored from a snapshot or forked. The guest kernel watches it. When the value changes, the kernel knows — with certainty, from outside itself — that it is now a clone of a machine that already existed, and that everything it believed about its own uniqueness is suspect. Modern Linux responds by immediately reseeding its CSPRNG, mixing in fresh entropy so that subsequent `getrandom()` calls diverge from the sibling clones. Firecracker exposes a VMGenID device for exactly this purpose.

The reason this needs hypervisor cooperation is that a guest fundamentally cannot detect its own cloning from the inside. Every source of information available to the guest was itself part of the snapshot. The clone has no memory of being cloned, because the memory is the thing that got copied. You need a signal from outside the copied region, which is precisely what a generation ID is: a value the guest didn't write and can't predict, changed by the layer that did the cloning.

Where that fresh entropy comes from is the other half of the story. A reseed is only as good as what it mixes in, and it needs a source the snapshot couldn't have pre-computed. In practice that's virtio-rng — a live pipe to the host's own CSPRNG, which keeps moving, so each clone's draw differs — and `RDRAND`/`RDSEED` on CPUs that expose it, which returns fresh bytes per call regardless of what the memory image froze. The generation ID is the trigger; those are the fuel.

Now the crucial limitation, because this is where most teams get a false sense of safety. VMGenID reseeds the kernel. It does not, and cannot, reach into a Python process's heap and re-seed the Mersenne Twister that `random.seed()` initialized before the snapshot was taken. The kernel has no idea that variable exists. The reseed fixes the layer that reads from the kernel after the reseed fires; it fixes nothing that already read and cached.

• Fresh `getrandom()` / `/dev/urandom` reads — Kernel RNG (with VMGenID): fixed automatically on restore, the reseed fires before the guest resumes doing work. Userspace PRNG seeded pre-snapshot: not applicable, this layer doesn't touch the kernel.
• A long-lived Python process holding `random` module state — Kernel RNG (with VMGenID): unaffected, the kernel can't see it. Userspace PRNG seeded pre-snapshot: still identical in every clone; must reseed itself explicitly.
• Node `Math.random()` in an already-running server — Kernel RNG (with VMGenID): unaffected. Userspace PRNG seeded pre-snapshot: identical stream per clone; use `crypto.randomBytes()` (kernel-backed) for anything that matters.
• A UUID cached in a module-level variable at import time — Kernel RNG (with VMGenID): unaffected, it was computed once and stored. Userspace PRNG seeded pre-snapshot: identical in every clone forever; must be regenerated on a restore hook.
• TLS session state / connection pool established pre-snapshot — Kernel RNG (with VMGenID): unaffected. Userspace PRNG seeded pre-snapshot: every clone thinks it owns the same session; tear down and re-establish after restore.
• A new process started after restore — Kernel RNG (with VMGenID): safe, it seeds from an already-reseeded kernel. Userspace PRNG seeded pre-snapshot: not applicable, there was no pre-snapshot seed to inherit.
• SSH host key baked into the rootfs image — Kernel RNG (with VMGenID): unaffected, it's on disk, not in the RNG. Userspace PRNG seeded pre-snapshot: identical across every clone; generate it after create, never at bake time.

Read that table twice. The single most common mistake I see is a team confirming VMGenID is wired up, declaring victory, and shipping a long-lived application server inside the snapshot — a server whose framework seeded its own PRNG at import time, three seconds before the snapshot was taken, and which will now hand out identical 'random' values in every clone for the rest of its life while the kernel underneath it is perfectly, uselessly well-seeded.

Treat post-restore reseeding as a lifecycle hook

The operational conclusion is that 'I was just restored' deserves to be a first-class lifecycle event in your application, the same way you treat SIGTERM or a readiness probe. Something in the guest should notice the generation ID changed and run a reseed routine, and that routine should cover the layers the kernel can't.

1. Reseed every userspace PRNG from the kernel. `random.seed(os.urandom(32))` in Python, `rand.New(rand.NewSource(...))` from a `crypto/rand` draw in Go — anything whose seed predates the snapshot. Do this after the kernel reseed, not before, or you've just carefully seeded from stale state.
2. Regenerate cached identity. Instance IDs, worker IDs, any UUID computed once at startup and held in memory. If your code has a module-level `INSTANCE_ID = uuid4()`, that line is a cloning bug wearing a disguise.
3. Tear down and re-establish TLS connections and pools. A restored connection is a connection two machines now believe they own. Drop them and reconnect rather than reasoning about what the peer thinks.
4. Refresh anything derived from `boot_id` or machine identity. Log correlation IDs, metrics instance labels, lock-holder identifiers — if two clones report as the same instance, your observability quietly merges two machines into one and your distributed lock stops being a lock.
5. Do not bake secrets into templates at all. This is the rule that makes the other four cheaper: SSH host keys, API tokens, signing keys, and database credentials should be generated on the live guest after create, never at bake time. A secret that never entered the snapshot cannot be cloned out of it.

That last one is the load-bearing discipline, and it's worth being blunt about: bake the environment, never the secrets. Every dependency, every compiled artifact, every warmed cache — bake all of it, that's what makes restores fast and cheap. But the moment something is supposed to be unique to one instance, it belongs on the far side of the restore boundary. A snapshot is a copy machine, and you should never feed a copy machine something you need exactly one of.

Testing for it: fork N ways and assert all distinct

The good news is that this is one of the most testable security properties you will ever encounter. There's no fuzzing, no threat modeling, no static analysis. You clone the thing N ways, ask each clone for its first random value, and assert that all N are distinct. If two match, you have a bug, and it's exactly the bug. The test is embarrassingly simple relative to the severity of what it catches — which is precisely why there's no excuse for not having it in CI.

On PandaStack this is quick to run because forks are cheap: a same-host fork lands in 400–750ms (cross-host 1.2–3.5s), so forking twenty ways and collecting a value from each is a matter of seconds rather than a nightly job. Here's the shape of it, checking all four layers at once — kernel, userspace PRNG, boot identity, and a cached module-level UUID.

"""Snapshot-clone randomness test: fork N ways, assert every clone diverges.

This is the whole test. If any two clones agree on a 'random' value, the
reseed-on-restore path is broken for that layer — and you found it before
a signing service did.
"""
from concurrent.futures import ThreadPoolExecutor
from pandastack import Sandbox

N = 20  # number of clones to fork from one snapshot

# 1. Build ONE machine and get it into the state you actually ship: a
#    long-lived process whose userspace PRNG was seeded BEFORE the snapshot.
#    That pre-seeding is the whole point — it's what the kernel can't fix.
base = Sandbox.create(template="base", ttl_seconds=900)
base.exec("mkdir -p /app")
base.exec(
    "cat > /app/probe.py <<'PY'\n"
    "import os, random, uuid\n"
    "# Seeded at import time -> frozen into the snapshot in every clone.\n"
    "CACHED_UUID = str(uuid.uuid4())\n"
    "print('kernel  ', os.urandom(16).hex())      # reseeded by VMGenID\n"
    "print('prng    ', random.random())           # NOT reseeded by VMGenID\n"
    "print('cached  ', CACHED_UUID)               # frozen at import\n"
    "print('boot_id ', open('/proc/sys/kernel/random/boot_id').read().strip())\n"
    "PY"
)
base.exec("python3 -c \"import random; random.seed(1234)\"")

snap = base.snapshot()   # freeze memory + disk
base.kill()


# 2. Fork that one snapshot N ways and collect the first random value
#    from each clone. Every fork starts from byte-identical memory.
def probe(_i: int) -> dict[str, str]:
    fork = snap.fork(ttl_seconds=300)
    try:
        out = fork.exec("python3 /app/probe.py", timeout_seconds=60).stdout
        return dict(
            line.split(None, 1) for line in out.strip().splitlines() if line
        )
    finally:
        fork.kill()


with ThreadPoolExecutor(max_workers=N) as pool:
    samples = list(pool.map(probe, range(N)))

# 3. Assert per layer. The kernel line MUST be all-distinct; the others tell
#    you which layers still need an application-level reseed hook.
for layer in ("kernel", "prng", "cached", "boot_id"):
    values = [s[layer] for s in samples]
    distinct = len(set(values))
    status = "OK  " if distinct == N else "DUPE"
    print(f"{status} {layer:8s} {distinct}/{N} distinct")

kernel_values = [s["kernel"] for s in samples]
assert len(set(kernel_values)) == N, (
    "kernel CSPRNG did NOT diverge across clones — reseed-on-restore is "
    "broken. Do not generate keys, tokens, or signatures on this platform."
)

Run that and the output is genuinely educational, because the four layers usually don't agree. On a correctly-configured platform the `kernel` line comes back 20/20 distinct — VMGenID did its job. The `prng` line will likely come back 1/20, because that Mersenne Twister was seeded before the snapshot and no hypervisor signal in the world is going to reach into it. The `cached` UUID will be 1/20 for the same reason. That gap between the first line and the rest is the entire thesis of this post, printed to your terminal in about ten seconds.

Wire this into CI against whatever template your production workloads actually restore from, not a minimal one. The bug lives in the gap between 'the kernel is fine' and 'my application server has been running since before the snapshot,' and a stripped-down test template has no application server to expose it. Assert hard on the kernel line and treat the userspace lines as a checklist of reseed hooks you still owe.

What a platform owes you, and what you still own

There's a clean division of responsibility here, and it's worth stating plainly because ambiguity about it is how this bug survives. The platform owns the kernel layer. On a system like PandaStack, where every create is a snapshot restore — around 179ms p50, 203ms p99, with only the very first spawn of a template paying the roughly 3s cold boot — reseed-on-restore isn't a feature, it's an invariant that has to hold on the hottest path in the product. The generation-ID device must be present, the guest kernel must act on it, and a fresh entropy source must be available for the reseed to draw from. Same for forks, which clone memory copy-on-write and are therefore the highest-multiplicity version of the same problem.

You own everything above the kernel. Your long-lived processes and their seeded PRNGs, your cached identifiers, your connection pools, and above all your template hygiene — because no hypervisor mechanism can unbake a secret you baked in. If you run your own Firecracker fleet with no platform in front of it, you own both halves: confirm the VMGenID device is attached and that your guest kernel reseeds on generation change, attach an entropy source, then audit your images for anything generated once at build time that was supposed to be per-instance.

Snapshots make machines cheap by making them identical. Randomness is the one place where identical is the bug — so every restore needs one moment where the machine is told, from outside itself, that it is no longer unique.

The bottom line

A snapshot freezes randomness at four layers: the kernel CSPRNG, userspace PRNG seeds, cached identifiers, and in-flight session state. Restore it into N machines and all four come back identical, which turns into duplicate UUIDs at best, cross-tenant session tokens in the middle, and full ECDSA private-key disclosure at worst. VMGenID — which Firecracker exposes as a device for exactly this reason — fixes the kernel layer by telling the guest from outside that it was cloned, so the kernel reseeds from virtio-rng and RDRAND before the guest does real work. It fixes nothing above the kernel, so long-lived processes must reseed themselves on a post-restore hook, and no secret should ever be baked into a template that gets cloned. Test it the easy way: fork N ways, take the first random value from each clone, assert all N differ. Fifty microVMs agreeing on a random number isn't a coincidence — it's a private key you've published in installments.