Snapshot, Restore, and the Connections You Left Open

A snapshot is an extremely thorough record of one machine's opinions. It captures, byte for byte, what the guest believed about the world at the instant it was frozen: which TCP connections it holds, what sequence numbers it expects next, which retransmit timers are armed, what MAC address lives behind the gateway, which hostname resolves to which IP, which TLS sessions are live, and how long that long-poll has been hanging. All of it comes back on restore exactly as it was. None of it was ever agreed to by anybody else.

I'm Ajay; I build PandaStack, an open-source Firecracker microVM platform where snapshot-restore is the normal creation path for every sandbox, so I get to watch this failure mode in production rather than read about it. It's the less-discussed sibling of the snapshot clone randomness problem. That one is about two machines agreeing on a secret they should have disagreed about. This one is about one machine confidently continuing a conversation the other party ended hours ago.

The reason it's under-discussed is that it produces no error at snapshot time and no error at restore time. Firecracker will happily freeze a guest with four hundred established sockets and happily bring it back. The complaint arrives later, from the application, in the form of a timeout that makes no sense.

What a snapshot freezes about the network

The mechanism is worth being precise about, because "the network state is saved" undersells how much of it there is. A memory snapshot captures guest RAM plus vCPU and device state. The guest kernel's entire networking stack lives in that RAM. So what you're preserving is not a summary — it's the live data structures.

• The socket table. Every entry in `/proc/net/tcp`, with its four-tuple, its state (ESTABLISHED, TIME_WAIT, FIN_WAIT), its send and receive queues, and the file descriptors in userspace pointing at it.
• Sequence and acknowledgement numbers. The exact byte offsets the guest expects to send and receive next, along with its window size and congestion-control state.
• Armed timers. Retransmit timers, delayed-ACK timers, keepalive timers, TIME_WAIT expiry — all of them counting down against a monotonic clock that stops when the vCPUs stop.
• The ARP/neighbour cache. Which MAC address the guest thinks sits behind each IP on its link, and how recently it verified that.
• The resolver's cache. Whatever `systemd-resolved`, `nscd`, or the application's own memoized `getaddrinfo` results held at freeze time, complete with TTLs that are also frozen.
• TLS session state. Live session keys, sequence counters, session tickets, and any handshake caught mid-flight — plus everything the peer's server-side session table would have to still agree with.
• Application-level connection pools. The pool object in your process's heap that says it has twenty healthy Postgres connections, twenty file descriptors it can hand out, and no reason to check.

Restore after a long pause: the peer moved on

Freeze a guest at 14:00 with a healthy connection to a database. Restore it at 22:00. From inside the guest, no time passed at all: the socket is ESTABLISHED, the pool considers it healthy, the next `SELECT` goes out on it without a second thought. From the database's point of view, that client stopped responding eight hours ago, got dropped by an idle timeout or a server restart, and had its session reclaimed some time before dinner.

There are three separate ways this goes wrong, and they fail on different timescales. The peer may have sent a RST or a FIN during the freeze — packets that arrived at a host with a paused guest and were simply dropped, because there was nobody to deliver them to. The peer may have timed out silently and closed with no packet at all, which is common for anything behind a load balancer with an idle timeout. And the host's own NAT/conntrack table, which is what makes the guest's private address reachable in the first place, will have swept the mapping long before eight hours elapsed; even if the peer somehow still cared, the return path no longer exists.

The result is a guest that resumes believing it has a healthy, established connection to a socket that has been closed since Tuesday. The application does not find out on resume. It finds out on the next write, which is either an immediate RST (fast, honest, recoverable) or nothing at all until the retransmit budget runs out, which can take minutes. That second case is the one that ruins your afternoon, because your health check passed, your readiness probe passed, and your first real request hung.

This is not unique to Firecracker. Any checkpoint/restore system inherits it: process-level checkpointers like CRIU have to explicitly decide what to do about open sockets, snapshot-based cold-start systems in the serverless world publish guidance about re-establishing connections after restore, and live migration only avoids the problem by keeping the pause down to milliseconds and carrying the network identity across with the VM. Check each system's own documentation for the specifics — the shared principle is that nobody has found a way to snapshot the other end of a TCP connection.

Inspecting the damage from inside the guest

You can see all of this directly. Run this twice — once immediately before you snapshot, once the instant the guest resumes — and diff the two outputs. Everything that survived unchanged is a claim the guest is making about a peer it has not spoken to since.

#!/usr/bin/env bash
# Run INSIDE the guest twice: once just before snapshotting, once the
# instant it resumes. Diff the two. Every ESTABLISHED row that survives
# is an opinion the guest holds about a peer that was never consulted.
set -uo pipefail

echo "=== wall clock (a restored guest resumes at BAKE time) ==="
date -u +%FT%TZ

echo "=== monotonic uptime (does not advance while the vCPUs are frozen) ==="
cut -d' ' -f1 /proc/uptime

echo "=== established sockets and the processes holding them ==="
# After a long restore these rows are usually fiction: the peer timed out,
# or RST'd into a host that had nobody to deliver the packet to.
ss -tanp state established

echo "=== raw kernel socket table (st=01 is ESTABLISHED) ==="
# tx_queue / rx_queue show bytes queued for a conversation that ended.
head -1 /proc/net/tcp
awk 'NR > 1 && $4 == "01"' /proc/net/tcp

echo "=== retransmit and timer state ==="
# A 'timer:(on,...)' with a climbing retrans counter right after resume is
# the guest shouting into a socket nobody on the far side is holding.
ss -tani state established | grep -E 'timer|retrans' || true

echo "=== ARP / neighbour cache ==="
# Learned before the snapshot. That MAC may now belong to another machine,
# or to nothing at all.
ip neigh show

echo "=== what the resolver still believes ==="
resolvectl statistics 2>/dev/null || cat /etc/resolv.conf

The `ip neigh` output is the quiet one people skip. A neighbour entry marked REACHABLE was verified at bake time and carries a validity window measured against a clock that stopped. On resume the guest may spend its first packets addressing a MAC that no longer answers, then re-ARP, then succeed — a small delay that shows up in your p99 and in nobody's logs.

Fork: two guests, one connection, no referee

Restore-after-a-pause is the mild version. Fork is where it gets genuinely funny, in the way that only production incidents are funny.

When you fork a running microVM, you get two guests with byte-identical memory. That includes byte-identical socket tables. Both children believe they own the same TCP connection: same source port, same destination, same sequence numbers, same TLS session keys, same send-side counters. Neither has any way to know the other exists, because the knowledge would have had to be in the memory, and the memory is the thing that got copied.

Best case, the platform's networking gives each child a different source address, the four-tuples diverge, and one or both children get an immediate RST from a peer that has no idea who they are. That's the good outcome: fast, loud, and recoverable by a reconnect. Worst case, both children actually reach the peer on the same connection and start writing. Now two independent processes are interleaving bytes into one ordered stream with two different notions of what byte 4,096 should be — which is the networking equivalent of two people typing into one Google Doc with the collaboration features switched off. TCP will do exactly what it was designed to do: deliver those bytes, in order, without complaint, to an application that will attempt to parse them.

TLS makes the failure crisper, which is a mercy. TLS records carry a sequence number that feeds into the record's authentication, so two forks writing on the same session produce records the peer can't authenticate, and it tears the connection down. You lose the connection instead of corrupting a stream. Take the win.

The practical rule is that fork multiplies whatever network state you left in the snapshot by N. If a restored guest holds one stale connection, a twenty-way fork holds twenty copies of one stale connection, all of which will retransmit at the same moment, all of which will reconnect at the same moment, and all of which will hit your database with a fresh handshake in the same millisecond. Congratulations: you've built a distributed system that spontaneously performs a synchronized denial-of-service on its own backend.

Timers, keepalives, and the monotonic clock that skipped

The guest's monotonic clock doesn't advance while its vCPUs aren't running. It also doesn't reset. So a restore is a discontinuity: the guest's notion of elapsed time jumps forward by however long the pause was, all at once, in a single instant between two instructions.

Every timer that would have fired during the pause is now overdue simultaneously. TCP retransmit timers fire in a burst. Keepalive probes fire in a burst. TIME_WAIT sockets expire in a burst. Application-level heartbeats, lease renewals, token refreshes, and circuit-breaker half-open transitions all fire in a burst. Congestion control, which reasons in terms of round-trip times measured against that same clock, briefly has a completely deranged view of the network.

Worse, all of this happens before your application code gets a chance to do anything sensible, because the kernel resumes first. By the time your resume hook runs, the guest has already tried to talk to several peers, and some of those attempts have already failed. Design accordingly: assume the first second after resume contains a flurry of doomed network activity, and make sure nothing in your system treats that flurry as a signal. A lease-renewal heartbeat that fires late, fails, and trips a failover is a self-inflicted outage triggered by a successful restore.

The mirror image applies to anything that measures liveness by absence. If a coordinator expects a heartbeat every 5 seconds and your guest was frozen for 90, the coordinator declared it dead long ago and may have reassigned its work. The guest resumes and cheerfully continues doing that work. Two workers, one task, no referee — the same shape as the fork problem, one layer up.

The wall clock: a real, well-known way to break TLS

Separate from the monotonic jump, the guest's wall clock resumes reading whatever it read when the snapshot was taken. If you baked a template in March and restore it in August, the guest believes it is March until something corrects it.

This breaks TLS, and it breaks it in a way that's initially baffling because the error blames the server. Certificate chain validation checks `notBefore` and `notAfter` against the local clock. Certificates rotate — commonly every 60 to 90 days for automated issuance. A guest whose clock is stuck months in the past will reject a perfectly valid, freshly-rotated certificate as "not yet valid," and it will do so for every HTTPS endpoint it touches, including your package registry, your object store, and your own API. This is a known operational failure mode for restored and long-suspended VMs generally, not a quirk of any one platform. We hit it on PandaStack and fixed it by making clock resynchronization part of the restore path itself, on restore, resume, and wake.

The fix belongs below the application: the platform should re-sync the guest clock as part of resuming it, before the guest does anything network-shaped. If you're running your own fleet, that means a host-provided time source and an explicit step-and-slew on resume — not "NTP will sort it out eventually," because eventually is measured in minutes and your first TLS handshake happens in milliseconds.

DNS caches and pools pointing at IPs that belong to someone else now

Cached DNS is the failure that survives your reconnect logic, which is what makes it special. You can dutifully close every socket and rebuild every pool, and still connect straight back to a machine that stopped being your machine weeks ago.

A cached A record has a TTL, but that TTL is measured against the frozen clock, so it does not expire on schedule. Meanwhile, in cloud environments, addresses are recycled aggressively: the IP your guest memorized for `db.internal` may now belong to a different service, a different tenant of your provider, or nothing. Connecting to it produces either a connection refused (fine), a hang against a dropped-packet firewall (slow, annoying), or — the interesting one — a successful TCP handshake with something that is not what you think it is, which then fails at the protocol or TLS layer with an error that sends you debugging entirely the wrong system.

The same applies one layer up, where the cache is harder to find. HTTP client libraries keep connection pools keyed by host, and those pools hold sockets to resolved addresses; a Python `requests` session, a Node `http.Agent`, a Go `http.Transport` all pin you to already-resolved endpoints. Service-discovery clients cache endpoint lists. And somewhere in every mature codebase there's a hard-coded IP a colleague resolved once during an incident in 2024 with a comment reading `// TODO: use DNS`.

What survives a pause, a restore, and a fork

These three operations sound similar and behave very differently, mostly as a function of elapsed time and multiplicity. Here's the same set of state, run through all three.

• TCP connections — resume after seconds: usually fine; the peer didn't notice the gap and host conntrack still holds the mapping. Restore after hours: the socket table says ESTABLISHED about a connection the peer closed on Tuesday, and you find out on the next write. Fork: two guests own the same four-tuple and the same sequence numbers, and at most one of them can be right.
• Monotonic clock and timers — resume after seconds: a small jump; a few timers fire slightly late and nothing cares. Restore after hours: a huge jump, so every overdue retransmit, keepalive, and lease renewal fires in one burst before your code runs. Fork: that identical burst happens in every child, at the same instant, aimed at the same peers.
• Wall clock and TLS validity — resume after seconds: negligible skew. Restore after hours or months: the clock reads bake time, so freshly-rotated certificates are rejected as not-yet-valid across every HTTPS call the guest makes. Fork: every child is wrong in exactly the same way, so your "it works on one of them" debugging heuristic gives you nothing.
• Host NAT / conntrack mapping — resume after seconds: still present, return path intact. Restore after hours: swept long ago, so even a peer that still cared has nowhere to send packets. Fork: N guests derived from state that described one guest's mapping.
• ARP / neighbour cache — resume after seconds: entries still valid. Restore after hours: entries name MACs that may have moved or vanished; the guest wastes its first packets before re-ARPing. Fork: identical stale cache in every child, so they all re-ARP together.
• DNS cache and connection pools — resume after seconds: harmless. Restore after hours: TTLs expired against a stopped clock, and cached IPs may now belong to someone else's service entirely. Fork: every child confidently dials the same possibly-reassigned address in parallel.
• An in-flight database transaction — resume after seconds: the server may still be holding the session; you might get away with it. Restore after hours: the session was reclaimed, so your COMMIT lands on a closed connection. Fork: two children each try to commit work the other doesn't know exists, which is the worst version of this on the entire list.
• Listening sockets and the accept path — resume after seconds: fine. Restore after hours: fine, because a listener holds no peer state to be wrong about. Fork: also fine — which is exactly why "stateless at the boundary" is the whole recommendation.

Read the last line next to the first. Inbound, connectionless, and re-establishable state comes through all three operations intact. Long-lived outbound state does not. That's the design guidance hiding in the table.

A brief word on identity: the network you baked vs the network you land in

There's a related problem the platform has to solve before your application ever sees a packet. A snapshot freezes the guest's own network identity too — its interface MAC, its IP, its routes, its default gateway — as those existed on the host where it was baked. Restore it onto a different host, or into a different slot on the same host, and the guest's baked-in identity has to be reconciled with the network it actually landed in.

PandaStack handles this by making the identity constant rather than fixing it up in the guest. The agent pre-allocates 16,384 /30 subnets per host as ready-made network namespaces with veth pairs and tap devices, and on restore it patches the tap's MAC address and the host-side routes to match the values frozen into the snapshot's metadata. The guest wakes up seeing precisely the IP, MAC, and gateway it had at bake time, and the host makes that true. It's also part of why creates are fast: a snapshot-restore create lands at p50 179ms and p99 around 203ms, and none of that budget is spent bringing up networking from scratch.

Worth being clear about the boundary: this solves the guest's identity, not its relationships. The platform can make sure your NIC looks the way you remember. It cannot make a database three availability zones away remember a session it closed hours ago.

The fix: treat restore as a reconnect event, not a resume

The whole class of problem collapses if you stop thinking of restore as "the machine continues" and start thinking of it as "the machine rejoins." A rejoining process is one that assumes nothing about its peers and re-establishes what it needs. That's not an exotic pattern — it's the same discipline that makes a process survive a network partition, and most robust systems already have most of it.

1. Quiesce before you snapshot. This is the highest-leverage step and the one people skip. Stop accepting work, let in-flight requests drain, close connection pools, and only then freeze. A snapshot taken at a quiescent point has almost no network state to be wrong about; a snapshot taken mid-request has all of it.
2. Re-sync the clock first on resume, before anything else. Wall clock and monotonic discontinuity both need correcting before your code makes a TLS handshake or evaluates a timeout. On a managed platform this should be the platform's job; verify it is.
3. Flush DNS at every layer. The system resolver, the HTTP client's connection pool, service-discovery caches, and any memoized `getaddrinfo` someone added for latency. Re-resolve, don't reuse.
4. Close and rebuild every pool. Do not attempt to validate a restored connection — a health check on a stale socket can pass by racing the RST. Close the file descriptors, build new connections, and accept the handshake cost. It is cheaper than the incident.
5. Regenerate anything that says which machine you are. Instance IDs, lock-holder identifiers, worker IDs, metrics labels. After a fork, N guests reporting the same identity will merge in your observability and un-lock your distributed locks.
6. Make workers reconnect-tolerant by design. Retry with backoff and jitter on connection errors, keep operations idempotent, and never treat a single failed heartbeat as authoritative. A worker that already survives a peer restart already survives a restore.
7. Prefer stateless at the boundary. Long-lived outbound connections are the thing that breaks. If a service can hold its cross-machine state in a request rather than in a socket, snapshot/restore/fork stops being a special case and becomes a non-event.

In code, the resume hook is unglamorous, which is the point. It's the same idea applied five times: refuse to trust anything the snapshot preserved about the outside world.

"""on_resume(): the first thing the guest runs after a restore or fork.

Every function here is the same idea applied to a different layer: refuse
to trust network state the snapshot preserved. The socket table, the DNS
cache, the TLS sessions and the clock all describe a world that stopped
existing the instant the snapshot was taken.
"""
import os
import socket
import subprocess
import time

import httpx

import myapp.http                       # holds a module-level httpx.Client
from myapp.cache import redis_client    # alive when the snapshot was taken
from myapp.db import pool as db_pool    # e.g. psycopg_pool / SQLAlchemy


def resync_clock() -> None:
    """Wall clock resumes at BAKE time. TLS chain validation reads it."""
    # Do this before any HTTPS call, or every freshly-rotated certificate
    # comes back as "not yet valid" and you go debugging the wrong server.
    subprocess.run(["hwclock", "--hctosys"], check=False)
    subprocess.run(["chronyc", "makestep"], check=False)


def flush_dns() -> None:
    """Cached A records expired against a clock that wasn't running."""
    subprocess.run(["resolvectl", "flush-caches"], check=False)

    # The application-level caches are the ones nobody remembers: a
    # memoized getaddrinfo someone added for latency, a service-discovery
    # endpoint list, an IP a colleague hard-coded during an incident.
    cache = getattr(socket, "_app_addr_cache", None)
    if cache is not None:
        cache.clear()


def reset_connections() -> None:
    """Close, never validate. A health check can pass by racing the RST."""
    # close() sends a FIN into a conversation that ended hours ago. Fine --
    # the goal is dropping our own fds, not being polite to a dead peer.
    for closer in (db_pool.close, myapp.http.client.close, redis_client.close):
        try:
            closer()
        except Exception:
            pass  # it was already broken; that is the entire premise

    db_pool.open()                          # fresh TCP + fresh TLS session
    myapp.http.client = httpx.Client()      # new pool, forces re-resolution
    redis_client.connect()


def refresh_identity() -> None:
    """After a fork, N guests claiming one identity is N-1 too many."""
    os.environ["INSTANCE_ID"] = os.urandom(16).hex()


def on_resume() -> None:
    resync_clock()        # 1. time, before anything evaluates a timeout
    flush_dns()           # 2. names, before anything opens a socket
    refresh_identity()    # 3. who am I, before anything takes a lock
    reset_connections()   # 4. sockets, now that 1-3 are true
    open("/run/app-ready", "w").close()   # 5. readiness LAST


if __name__ == "__main__":
    on_resume()
    print("rejoined at", time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()))

Note the ordering, because it isn't arbitrary. Clock before DNS, because DNS TTLs and TLS both read the clock. Identity before connections, because a pool may authenticate using it. Readiness last, because everything above is a lie until it has run.

Quiescing before you freeze, in practice

The other half of the job happens at snapshot time. The best resume hook in the world is still cleaning up a mess you chose to create. Freezing a machine that has drained its pools is dramatically less exciting than freezing one mid-transaction, and it costs you one extra step in the bake pipeline.

from pandastack import Sandbox

# 1. Build the machine and get it genuinely warm -- deps installed,
#    caches primed, imports done. This is what makes restores worth it.
sbx = Sandbox.create(template="base", ttl_seconds=3600)
sbx.exec("pip install -r /app/requirements.txt", timeout_seconds=600)
sbx.exec("python3 -c 'import myapp'")   # pay import cost once, in the bake

# 2. QUIESCE, then freeze. This is the step people skip, and it is the
#    difference between snapshotting a machine and snapshotting a machine
#    mid-sentence. Stop taking work, drain, close pools, verify.
sbx.exec("systemctl stop app-worker")
sbx.exec("python3 /app/quiesce.py")     # drain in-flight + close pools
print(sbx.exec("ss -tan state established | tail -n +2 | wc -l").stdout)  # 0

snap = sbx.snapshot()

# 3. Fork it. A same-host fork lands in 400-750ms (cross-host 1.2-3.5s),
#    so branching a warm machine is a loop rather than an architecture --
#    but every child inherits the parent's socket table, so every child
#    rejoins the network on its own terms before it does any work.
for task in tasks:
    child = sbx.fork()
    child.exec("python3 /app/on_resume.py")        # clock, DNS, identity, pools
    child.exec(f"python3 /app/run.py {task.id}", timeout_seconds=300)
    child.kill()

That `wc -l` line is worth keeping as an assertion rather than a print. "Zero established outbound connections at snapshot time" is a property you can enforce in CI, and it's a far better use of a test than trying to reproduce a stale-connection incident after the fact.

A snapshot records what one machine believed. A connection is what two machines agreed. You cannot freeze an agreement by writing down one side of it — you can only freeze the belief, and beliefs go stale.

The bottom line

Snapshot and restore preserve the guest's entire network worldview — socket table, sequence numbers, armed timers, ARP cache, DNS cache, TLS sessions, connection pools — and preserve exactly none of the agreement those things depended on. Restore after a long pause and the peer has closed, the RSTs were dropped into a paused guest, and the host's conntrack mapping expired; the guest resumes confident about a connection that ended hours ago. Fork and you multiply that by N, with two children owning one four-tuple and, in the worst case, interleaving bytes into a single ordered stream. Monotonic time jumps, so overdue timers fire in a burst before your code runs. The wall clock resumes at bake time, which breaks TLS validation against any certificate that rotated since — a real and well-documented failure mode. And cached DNS survives your reconnect logic, pointing at addresses that may have been reassigned.

The remedy is one sentence: treat restore as a reconnect event, not a resume. Quiesce before freezing, re-sync the clock and flush DNS before touching the network, close and rebuild pools instead of validating them, regenerate machine identity, make workers reconnect-tolerant, and keep long-lived cross-machine state out of the snapshot in the first place. Do that and snapshot-restore goes back to being what it should be — a very fast way to get a warm machine — instead of a very fast way to distribute a stale opinion.