Guest Clocks and the TSC After a Firecracker Restore

Here is a bug that looks like sorcery the first time it bites you: you snapshot a perfectly healthy microVM, restore it a day later, and now every HTTPS request out of the guest fails certificate validation. Nothing in the guest changed. The certs are fine. The network is fine. What broke is that the guest is convinced it's still yesterday — it woke up believing no time has passed since the snapshot, because from its own clock's point of view, none has. Restore it a week late and it insists it's still last Tuesday, and every TLS peer it talks to disagrees. This post is about why that happens: how a guest reads time at all (the TSC, kvmclock, wall-clock), what a snapshot does to each of those, the concrete ways a frozen clock breaks real software (TLS, JWTs, cron, monotonic clocks), the TSC-offset mechanism underneath, why VMGenID exists to shout "you were restored," and the handful of mitigations that actually work. It's a Firecracker internals tour with a very real operational lesson at the end.

How a guest reads time: TSC, kvmclock, wall-clock

Before you can understand what a snapshot breaks, you need the three layers a guest stacks to answer "what time is it?" — because a snapshot hits each one differently, and the interesting failures live in the seams between them.

• The TSC (Time Stamp Counter) — a per-CPU hardware register that counts up, tick by tick, since the core powered on. Reading it is a single cheap instruction (RDTSC), which is why it's the substrate for almost all fast timekeeping. It is a monotonic counter, not a wall clock: it tells you how many ticks have elapsed, not what date it is. The guest divides ticks by the TSC frequency to get elapsed nanoseconds.
• kvmclock (the KVM paravirtual clock) — a shared memory page KVM exposes to the guest. Instead of the guest trusting a raw hardware TSC that a hypervisor might rescale or migrate out from under it, KVM publishes a small structure — a reference TSC value, a system-time base in nanoseconds, and a scale/shift to convert ticks to ns. The guest reads its TSC, applies that published transform, and gets a nanosecond timestamp that KVM controls. This is what lets the host keep guest time coherent across vCPU migration and (in principle) across a restore.
• Wall-clock time — the actual date-and-time ("2026-07-20 14:03:11 UTC"). The guest computes it as a base wall-clock value (seeded once, at boot, from an emulated RTC or a paravirtual wallclock MSR) plus the elapsed nanoseconds since that base, measured via kvmclock/TSC. date, gettimeofday(), certificate validity checks — everything human-facing reads this derived wall-clock, and it is entirely a function of "the base we were seeded with" plus "how much monotonic time we think has passed."

The critical thing to internalize: wall-clock time in a guest is derived, not stored live. There is no ticking real-world clock inside the VM. There's a base captured once, and a monotonic counter measuring elapsed time since that base. Freeze the counter and reset the base wrong, and the guest's idea of "now" is frozen with it — which is exactly what a snapshot does.

The frozen-clock problem: your VM wakes up in the past

A Firecracker snapshot is a byte-for-byte freeze of guest state — RAM, vCPU registers, and device state — at one instant. Time is part of that state. The guest's wall-clock base, its kvmclock structure, its notion of elapsed nanoseconds: all frozen. When you restore, the guest resumes mid-instruction as if a pause button had been pressed and released. From inside the guest, zero time elapsed between snapshot and restore — because nothing in its state records that any did.

So if you bake a template at 09:00 on Monday and restore a sandbox from it at 09:00 on Wednesday, the restored guest believes it is Monday at 09:00. Its clock will now tick forward correctly from that moment — one second per second — but it starts two days behind reality and stays two days behind until something drags it back. The guest is running a perfectly accurate clock that happens to be set to the wrong day. It's not drifting; it's displaced. And it will confidently timestamp logs, validate certificates, and check token expiry against a date that is flatly, dangerously in the past.

# Inside a guest restored from a snapshot baked two days ago.
# The clock is frozen at bake-time until something resyncs it.

$ date -u
Mon Jul 18 09:00:04 UTC 2026        # <- reality is Wednesday. The guest disagrees.

# Every TLS handshake now fails: the guest thinks the server's
# certificate 'notBefore' is in the FUTURE (or its own CA bundle
# looks not-yet-valid), so validation rejects it.
$ curl -sS https://api.example.com/health
curl: (60) SSL certificate problem: certificate is not yet valid
More details here: https://curl.se/docs/sslcerts.html

# The monotonic clock is equally displaced — anything that scheduled
# 'run in 60s' at bake time may fire the instant the VM resumes.
$ cat /proc/uptime
2.14 0.98                            # <- 'uptime' resets to seconds-since-restore

What actually breaks: TLS, JWTs, cron, monotonic clocks

A frozen clock is not a cosmetic nuisance — it silently breaks a specific and painful set of things, all of which depend on the guest's idea of "now" being roughly correct.

TLS certificate validation

This is the one that hurts most, because it takes down all outbound HTTPS at once. Certificate validation checks that the current time falls within the cert's notBefore..notAfter window. A guest stuck in the past can fail this two ways. If it's stuck before a cert's notBefore, it rejects the cert as "not yet valid" — common when the peer rotated to a freshly issued cert after your bake date. And if the guest is stuck far enough in the past, short-lived certs (think 90-day ACME certs, or 24-hour internal ones) that were valid at bake time may look valid but the peer has since rotated, so the guest is validating against a chain that no longer matches. Either way: HTTPS breaks, and the error ("certificate is not yet valid") is baffling until you check date.

JWT and token expiry

JWTs carry iat (issued-at), nbf (not-before), and exp (expiry) claims, all checked against the local clock. A guest in the past will happily accept tokens that have actually expired (it thinks exp is still in the future) and — worse for correctness — may reject freshly-minted tokens whose nbf/iat it reads as being in the future. Session logic, OAuth flows, and signed webhooks all wobble. The failures are intermittent and identity-shaped, which makes them miserable to diagnose if you don't already suspect the clock.

Cron and scheduled jobs

Anything scheduled against wall-clock time misfires. A cron job set for "02:00 daily" evaluated against a guest that thinks it's a different day may fire immediately, fire twice, or skip. Then, when the clock finally steps forward to reality, the scheduler sees a huge jump and may re-evaluate every missed slot at once. Rate limiters and token-bucket refills keyed on wall-clock time get similarly confused.

Monotonic clock jumps

Even code that correctly uses CLOCK_MONOTONIC (which is supposed to never go backward) isn't immune. A monotonic clock's zero point is boot; on restore, the guest's monotonic clock effectively resets to "seconds since restore," so timers that were mid-flight when you snapshotted can complete instantly on resume, and any code holding a monotonic timestamp from before the snapshot now measures a nonsensical interval. Connection keepalives, retry backoffs, and I/O timeouts all read the resume as a discontinuity.

Under the hood: the TSC offset and why restore doesn't fix time

Here's the mechanism that makes the freeze precise. The TSC is per-core and counts from power-on, so a guest's TSC and the host's TSC don't start from the same value. KVM bridges that gap with a per-vCPU TSC offset: guest_tsc = host_tsc + offset. The guest reads what looks like its own private, boot-relative counter, but under the hood it's the host TSC plus a fixed offset the hypervisor controls. On CPUs with TSC scaling, there's also a multiplier so a guest baked on a 2.5 GHz host can present a stable frequency on a 3.0 GHz host.

When Firecracker snapshots, it captures the guest's TSC value (and the frequency it was running at). On restore, it reprograms the TSC offset so the guest's TSC resumes at exactly the value it had at snapshot time — that's what makes resume seamless and monotonic-safe from the guest's perspective. But notice what that guarantees and what it doesn't. It guarantees the guest's monotonic counter doesn't jump backward. It does absolutely nothing to advance wall-clock time to reflect the hours or days that passed on the wall. The TSC-offset restore is faithfully reconstructing the guest's frozen sense of time — which is precisely the problem. The mechanism designed to make restore seamless is the same mechanism that leaves the wall clock in the past.

VMGenID: the hardware signal that says "you were restored"

The guest has a real epistemic problem: from the inside, a restore is invisible. Nothing in the normal execution flow tells the guest kernel "you were just resumed from a snapshot that's N days old." It just keeps running the next instruction. So the ecosystem invented an explicit out-of-band signal: the Virtual Machine Generation ID (VMGenID), a small paravirtual device that exposes a 128-bit identifier. The host changes that identifier whenever the VM's execution is duplicated or resumed from a point-in-time image — which is exactly what a snapshot restore (and a fork) is.

The guest kernel watches for VMGenID changes and treats a change as a hard signal: "my timeline was branched or rewound — anything I assumed about continuity is now suspect." The canonical use is entropy (the kernel reseeds its CRNG on a VMGenID change so two restores of one snapshot don't share a random stream), but the same signal is exactly what a time-sync daemon or an on-restore hook wants: it's the trigger to say "don't trust my wall clock, go re-establish it." Firecracker implements a VMGenID device and bumps it on restore precisely so guests can detect the discontinuity they otherwise couldn't see. We cover the device itself in depth at /blog/firecracker-vmgenid-device-explained; here it's the linchpin of the guest-side mitigation.

Mitigations: host-pushed clock step vs guest NTP on VMGenID

There are three postures you can take toward the frozen clock, and only two of them are acceptable in production. Here they are across correctness, latency, and complexity:

• No resync (do nothing) — A correctness: broken; the guest runs in the past until NTP happens to notice (which can be minutes, if it's even installed and permitted to step) — every TLS/JWT/cron path is wrong in the meantime. B latency: zero added cost, because you added nothing. C complexity: none — and that's the trap; it looks fine in a demo that restores seconds after baking, then detonates the first time you restore a day-old snapshot. Not an option for anything that talks HTTPS.
• Host/agent pushes a clock step on restore — A correctness: strong; the host knows the true time and injects it into the guest immediately on resume (settimeofday via the guest agent, hwclock --hctosys from a synced emulated RTC, or chronyc makestep), so the clock is right before the first instruction of real work runs. B latency: negligible — a single set-time call in the restore path, well under the restore step itself. C complexity: moderate — the platform owns a guest-agent hook on every restore/resume/wake, but it's centralized and doesn't depend on the guest having a working network or NTP. This is the belt.
• Guest detects VMGenID change and forces an NTP step — A correctness: strong and self-contained; the guest watches VMGenID, and on a change forces chronyc makestep (or an ntpd -gq / step) rather than waiting for a slow slew. B latency: small but non-zero — it depends on reaching an NTP server, so it's gated on the guest's network being up post-restore. C complexity: moderate and lives in the guest image, so it works even if the host doesn't push anything — but you must configure the time daemon to STEP large offsets (not gently slew them), or a two-day gap takes forever to correct. This is the suspenders.

In practice the robust answer is both: the host pushes an authoritative step on restore so the clock is correct instantly and offline-safe, and the guest also reseeds/steps on the VMGenID change as a backstop for forks and any restore path the host hook missed. The one thing you must never do is let application code run against an un-stepped, just-restored clock — the same discipline as reseeding entropy after restore, and for the same reason: the snapshot froze a value that's now wrong, and you have to correct it before anything reads it.

# Agent-side fix, run in the guest on restore/resume/wake, BEFORE any
# app work. Step the clock to the true time the host knows; do not slew.

# Option A: host injects the authoritative time via the guest agent.
# ('date -s' sets the system clock in one step — the correct primitive
#  for a large, known offset, unlike a slow NTP slew.)
date -s "$(date -u -d @${HOST_EPOCH} '+%Y-%m-%d %H:%M:%S') UTC"

# Option B: the emulated RTC was kept at real time by the host; pull it
# into the system clock in one hardware-to-system step.
hwclock --hctosys --utc

# Option C: let chrony correct it, but force a STEP for the huge offset
# instead of waiting for it to slew seconds-per-day back to reality.
chronyc makestep          # apply a one-shot step now
chronyc waitsync 5        # block until the clock is actually synced

# Sanity check: the guest should now agree with the wall.
date -u

How a sub-second restore platform handles it

This stops being an abstract caveat the moment your whole model is snapshot-restore on every create. On PandaStack there is no warm pool — every sandbox is created by restoring a baked template snapshot (a create lands around a ~49ms restore step, p50 179ms and roughly 203ms p99, versus ~3s for a genuine first cold boot). A template snapshot might be baked once and then restored hours or days later, thousands of times. Every one of those restores inherits the bake-time clock, so if the platform did nothing, every fresh sandbox would boot into the past and its first HTTPS call would fail — which is exactly the failure we hit and fixed.

The fix is host-authoritative and unconditional: the agent steps the guest clock to the real current time on every restore, resume, and wake — before the sandbox is handed back as ready — so a sandbox restored from a week-old seed nonetheless comes up believing it's now, and TLS just works. It pairs with the entropy reseed (both are "correct a value the snapshot froze, before any code reads it"), and it's why a restored PandaStack sandbox doesn't greet you with 'certificate is not yet valid.' The core is open source under Apache-2.0, so you can read the restore path and the clock-step hook on your own hosts. For the RAM-side version of the same lesson — secrets and entropy frozen in vm.mem — see /blog/firecracker-snapshot-secrets-security, and for the snapshot/fork primitives underneath all of this, /blog/snapshot-and-fork-explained.