Guest Clocks, Snapshots, and the Time Travel Problem

Here is a bug I got to experience personally, at a moment of my life I would have preferred to spend otherwise. A fleet of restored microVMs started failing every outbound HTTPS call at roughly the same moment. Not some of them. Every guest, every request, all at once, with TLS handshake errors that read exactly like a network problem — the kind of error that sends you straight to the firewall rules, the egress NAT, the DNS resolver, the CA bundle, and then eventually to a quiet corner to reconsider your career. The network was fine. The CA bundle was fine. The guests believed it was a date in the past, an upstream service had rotated its certificate to one issued after that date, and every guest in the fleet was independently and correctly concluding that this certificate was not yet valid.

I'm Ajay; I built PandaStack, which restores a Firecracker snapshot on every sandbox create rather than keeping a warm pool of idle VMs. That design is why we get a restore step around 49ms and a p50 create of 179ms — and it is also why the guest clock problem is not an edge case for us but a structural property of the platform we had to go fix. The fix was to sync the guest's clock on every restore, resume, and wake. This post is what I learned along the way: how a Linux guest actually keeps time, what a memory snapshot freezes about that, the failure modes in the order they will ruin your afternoon, and a 60-second runbook to tell a frozen clock apart from a genuine network problem before you start blaming your CDN.

How a Linux guest actually keeps time

There are two questions a computer answers about time, and they are much less related than people assume. "How much time has elapsed since some arbitrary point?" is a counting problem. "What is today's date and time?" is a knowledge problem — the machine has to be told, because no amount of counting tells you what year it is. Linux keeps these separate on purpose: `CLOCK_MONOTONIC` answers the first and only ever moves forward at a steady rate; `CLOCK_REALTIME` answers the second and can be stepped backward or forward by an administrator, by NTP, or by anything else with the right capability.

The counting comes from a clocksource. On x86 that is usually the TSC — a per-core counter that ticks at a known rate — read through a fast userspace path so that a program calling `clock_gettime()` in a tight loop doesn't have to trap into the kernel every time. Under KVM there's typically also `kvm-clock`, a paravirtualized clocksource: instead of the guest trying to interpret raw hardware, the host publishes a structure the guest reads to derive time, which lets the hypervisor be honest about things the guest can't see, like a vCPU being descheduled for 40ms while the host does something else. Check `/sys/devices/system/clocksource/clocksource0/current_clocksource` to see which one your guest chose; the specifics vary by kernel version and CPU features, so verify against your own kernel rather than my summary.

The knowledge part is the interesting half. At boot, the guest reads wall-clock time exactly once, from a hardware-ish source — an RTC, a paravirtualized equivalent, or whatever the platform offers — and from then on it maintains that value by counting. That is the whole mechanism. The guest is not continuously asking anyone what time it is. It learned the date once, at boot, and has been adding ticks to it ever since, with an NTP daemon occasionally nudging the rate or stepping the value if you installed one. A machine's sense of the calendar is a single number it was told at birth, plus arithmetic.

What a snapshot actually freezes

A Firecracker snapshot is a memory image plus device state. It is a complete, byte-accurate photograph of a machine mid-thought. Everything the guest knew, it still knows on restore — including its notion of what time it is, because that notion lives in guest memory and kernel data structures like any other variable. There is no separate "time" field the hypervisor thoughtfully refreshes for you on the way back in. The clock is state, and snapshots preserve state. That is the entire feature.

So restore that image an hour later, or a day, or a month, and the guest resumes mid-instruction with a wall clock reading bake time. Nothing in the guest notices, and I want to be precise about why: from inside the VM, no time passed. There was no gap to detect. The counter it uses to advance its clock resumed from the value it was suspended at. The guest did not go to sleep and wake up — from its own perspective it never stopped. It is not confused; it is confidently, internally consistently wrong, which is a much harder failure mode than confusion. A confused machine logs something. This one carries on with total composure, timestamping everything in a date that has already happened, and only reveals the problem when it tries to talk to something outside itself that knows better.

It gets more interesting on fork. When you fork a snapshot — 400–750ms same-host on PandaStack, 1.2–3.5s cross-host — you don't get one machine living in the past, you get N of them, all convinced they are the same machine at the same instant. This is the clock cousin of the snapshot randomness problem: a restored guest also resumes with the same seeded PRNG state, the same `/dev/urandom` pool, the same in-flight nonces. Time and entropy fail together, for the same reason, and the combination is genuinely dangerous — identical clocks plus identical entropy is how you get colliding UUIDs with identical timestamps across children that are supposed to be independent workers.

The failure modes, in order of how much they'll ruin your afternoon

1. TLS certificate validation, which goes first and loudest

Certificate validation is a date comparison. Every X.509 certificate carries `notBefore` and `notAfter`, and every TLS client checks the current time against that window. A guest whose clock is behind will reject a certificate issued after bake time as not yet valid — the certificate is fine, the guest is wrong, and the error message is about the certificate. It will also happily accept a certificate that expired between bake time and now, which is the same bug pointed at your security posture instead of your availability.

The reason this is the worst one is the correlation. Certificates rotate on their own schedule, not yours. The day an upstream provider rotates to a cert issued after your bake date, every restored guest in your fleet starts failing outbound HTTPS simultaneously, without you deploying anything. A fleet-wide, instantaneous, deploy-independent failure of all outbound calls has a very strong prior attached to it, and that prior is "network." So you go debug the network. TLS is the messenger here, and the messenger gets shot, repeatedly, for about forty minutes.

2. Signed requests and tokens rejected as skewed

Anything that signs a timestamp to prove freshness breaks next. AWS SigV4 puts a timestamp in the signature and servers reject requests outside a tolerance window. Webhook signature schemes generally do the same, precisely so that a captured request can't be replayed a week later — which is, from the server's point of view, exactly what your restored guest looks like. JWTs carry `iat` and `exp`; a guest in the past will mint tokens that look like they were issued before they should have been, and will reject valid inbound tokens as not yet issued. OAuth refresh logic that says "renew when expiry is within five minutes" never fires, because by the guest's arithmetic expiry is comfortably distant. Each of these produces a different, confidently-worded error from a different vendor, none of which says "your clock is wrong," and collectively they will send you on a tour of five unrelated integration docs.

3. Cron and timer wheels: a storm, or silence

Scheduled work goes one of two ways depending on how the fix arrives. If the clock stays frozen in the past, everything scheduled for real-world now simply never fires — your nightly job doesn't run, and it doesn't error either, because from the guest's view it isn't time yet. If instead something steps the clock forward by a large jump, schedulers that compute "how many intervals did I miss" can decide they missed several hundred and attempt to catch up all at once. A daily job discovering it has three weeks of backlog and deciding to run all of it concurrently is a self-inflicted denial of service performed with complete correctness by software doing exactly what you told it.

4. Database timestamps, TTLs, and replication

Now the wrong time gets written down and outlives the incident. Rows land with `created_at` values in the past, so anything ordering by timestamp interleaves them incorrectly with rows written by machines that know what day it is. Cache TTLs computed as now-plus-an-hour expire immediately relative to a correct reader, or never expire at all. Last-write-wins conflict resolution is the genuinely alarming one: if two nodes resolve a conflict by comparing timestamps and one of them is a month behind, the older-looking write loses regardless of what actually happened first. And unlike a failed HTTPS call, which you retry, bad timestamps are durable. You fix the clock and the wrong data stays.

5. Logs timestamped in the past, so you conclude nothing happened

This is fifth in blast radius and first in wasted human hours. A guest emitting logs stamped at bake time ships them into your aggregator, where they sort into a window from three weeks ago. You open your dashboard, filter to the last fifteen minutes, see nothing, and reasonably conclude the service isn't even running — or worse, that logging is broken. The evidence you need is present, indexed, and searchable; it's just filed under a date you have no reason to look at. Trace correlation degrades the same way, with spans landing outside the trace window and getting dropped or orphaned. Every diagnostic tool you'd reach for is quietly showing you the wrong slice of reality while the thing you're hunting sits in plain sight, three weeks up the scroll.

CLOCK_MONOTONIC lies across a restore too

The standard advice is to use `CLOCK_MONOTONIC` for anything measuring duration, because it can't be stepped by an admin or dragged around by NTP. That advice is correct and you should follow it — but understand what it does and doesn't buy you here. `CLOCK_MONOTONIC` guarantees it never goes backward. It does not guarantee it tracks reality. Across a snapshot restore, the guest's monotonic clock resumes near where it left off, so a measurement started before the snapshot and finished after it will report a duration of a few milliseconds for an interval that contained your entire lunch break, or your entire weekend.

The practical consequences are all in code that reasons about elapsed time across a suspension boundary. A connection pool's idle-timeout logic decides a socket has been idle for two seconds and hands it out; the peer closed it three weeks ago. A rate limiter's token bucket refills based on elapsed monotonic time, sees almost none, and throttles a workload that has in fact been dormant for a month. A circuit breaker in its open state waits for a cooldown that, by its own accounting, has barely started. A lease or lock with a monotonic deadline believes it still holds something that expired long ago on every other machine in the system — which is a correctness bug, not a performance one.

There is no clever monotonic trick that fixes this, because the guest genuinely has no in-band way to know a gap occurred. Something outside it has to say so. That is exactly what Firecracker's VMGenID device is for, and it's the right primitive to reach for.

The fixes, honestly ranked

Best: the platform resyncs the clock at restore, resume, and wake

The correct place to fix this is the layer that knows a restore happened, and that layer is the platform, not your application. The hypervisor's supervisor knows the real time, knows it just resumed a guest from a memory image baked at some earlier point, and can step the guest's clock before your code gets a chance to notice. Every guest gets fixed, including guests running code written by people who have never heard of any of this — which, if you're running other people's workloads, is most of them.

This is what PandaStack's agent does now: a clock sync on restore, on resume, and on wake from hibernate. All three, because all three are the same hazard wearing different names. The one that's easy to miss is wake — an app that scaled to zero and came back an hour later has exactly the same frozen-clock problem as a cold restore, and it's the path least likely to be covered by whatever you built for the obvious case. If you operate your own snapshot infrastructure, this is the single highest-leverage change available to you, and it's not a large one.

Good: chrony inside the guest, configured to step

Defense in depth belongs inside the guest, and the critical detail is that an NTP daemon's default behavior is wrong for this case. NTP normally corrects small errors by slewing — subtly adjusting the tick rate so the clock walks toward correct without any discontinuity, because sudden jumps break things. That's the right default for a machine that's a few hundred milliseconds off. It is entirely wrong for a machine that's three weeks off, where slewing at the usual rate would take approximately forever, and many daemons will simply refuse to correct an offset that large and log a complaint instead.

What you want is `makestep`: step the clock immediately if the offset exceeds a threshold. Configure it to apply on every update, not just the first few, because a restored guest can be hours off at any point in its life, not only at boot.

# /etc/chrony/chrony.conf — for a guest that may be restored from a
# snapshot baked at an arbitrary point in the past.

# Use whatever NTP source your environment provides. On a cloud host the
# link-local metadata NTP endpoint is usually the lowest-latency option;
# verify the correct address against your provider's docs.
server 169.254.169.123 iburst prefer
pool   pool.ntp.org iburst maxsources 3

# THE IMPORTANT LINE.
# Step the clock (jump, don't slew) whenever the offset exceeds 1 second.
# The '-1' means "on every update, forever" -- NOT just the first N
# updates, which is the common default and is exactly wrong for a guest
# that can be restored from an old snapshot at any moment.
makestep 1.0 -1

# Let chrony write corrections back so a later boot starts closer.
rtcsync
driftfile /var/lib/chrony/chrony.drift

# Verify after a restore. 'System time' is the offset from true time; if
# it reads in hours or days, you have found your bug.
#   chronyc tracking
#   chronyc makestep      # force an immediate step, right now
#   timedatectl status    # 'System clock synchronized: yes' or you have work to do

Two caveats. First, this is a race: chrony fixes the clock some seconds after resume, and if your application makes its first outbound HTTPS call before that, it still fails — so treat guest NTP as a safety net under a platform-level sync, not as a replacement for one. Second, a stepping clock is itself a hazard for anything measuring wall-clock durations, which is a good reason to make sure such code is using `CLOCK_MONOTONIC` in the first place.

Also: VMGenID as the "you have been restored" signal

Firecracker exposes a VM Generation ID device — a value that changes when a VM resumes from a snapshot, giving guest software an in-band way to learn that it has been restored. It was designed primarily for the entropy half of this problem (reseed your PRNG, because you are no longer the only machine with this state), but it's the same signal for the clock half, and the same signal for anything else your software should reconsider after discovering it has been asleep. The generation ID changed, therefore: reseed randomness, resync time, drop connections you were holding, invalidate leases you thought you held, and re-derive anything you cached with a monotonic deadline.

The catch is that guest software has to actually watch for it, which most software doesn't, because most software was written for machines that don't get photographed and restored. Check the current device support and interface details against Firecracker's own documentation rather than my summary — this area has moved. But architecturally it's the right shape: a restore is an event, the guest deserves to be told, and everything that should react to it can hang off that one notification.

And: keep bakes fresh so the drift window stays small

This is the least clever item and it does real work. The size of your problem is exactly the age of your snapshot. A template baked this morning produces guests that are hours wrong; a template baked last quarter produces guests that are a season wrong, and a season is more than long enough for every upstream certificate you depend on to have rotated. Re-baking templates on a schedule doesn't fix the mechanism — the guest still resumes in the past — it just keeps the past close enough that most of the failure modes stay latent. Treat it as damage reduction rather than a fix, and do it anyway, because it also keeps your base images patched.

The 60-second runbook: frozen clock, or genuine network problem?

Both hypotheses present as "TLS is failing everywhere." Here is how to separate them fast enough that it doesn't matter which one you guessed first. The whole check is one round trip into a guest.

# ---- Run these INSIDE the guest, and compare against the host. ----

# 1. The one that usually ends the investigation.
#    Compare guest wall-clock to a machine you trust.
date -u                      # in the guest
# ...vs the host / your laptop:
date -u
# If these differ by more than a few seconds, stop here. It's the clock.

# 2. Is anything even trying to keep time in this guest?
timedatectl status
#   System clock synchronized: no    <- nobody is correcting the drift
#   NTP service: inactive            <- ...because nothing is running

# 3. If chrony IS running, ask it how wrong it thinks it is.
chronyc tracking
#   Look at 'System time'. Milliseconds = healthy.
#   Hours or days = a restored snapshot that nobody resynced.
#   'Leap status : Not synchronised' = it gave up, likely because the
#   offset was too large to slew and makestep wasn't configured.

# 4. The decisive TLS test. Ask the certificate directly.
echo | openssl s_client -connect api.example.com:443 -servername api.example.com 2>&1 \
  | grep -Ei 'verify|notBefore|notAfter'
#
#   "certificate is not yet valid"  -> FROZEN CLOCK. The cert was issued
#                                      AFTER the guest thinks it is now.
#                                      A network problem cannot produce this.
#   "certificate has expired"       -> could be either; compare notAfter
#                                      against BOTH clocks before deciding.
#   "unable to get local issuer"    -> real CA-bundle problem, not time.
#   connection refused / timeout    -> real network problem, not time.

# 5. Confirm the fix in place before you go re-bake anything.
sudo chronyc makestep && date -u

Step 4 is the discriminator, and it's worth understanding why it's decisive rather than merely suggestive. "Certificate is not yet valid" is a claim the guest can only make by comparing a `notBefore` date against its own clock and finding the future. No firewall, no proxy, no MTU issue, no DNS misconfiguration can produce that string. It is a pure statement about local time, and it means the guest's clock is behind the certificate's issuance date. Contrast it with a timeout or a connection refusal, which are statements about packets and have nothing to do with time at all. One error is about arithmetic; the other is about the network. They look similar in a dashboard and they are not remotely the same problem.

• Onset — Frozen guest clock: fleet-wide and instantaneous, on a day you deployed nothing; correlates with someone else's certificate rotation. Genuine network/TLS misconfig: correlates with your change — a deploy, a firewall edit, a CA-bundle update, a DNS cutover.
• Blast pattern — Frozen guest clock: every restored guest fails identically, while a freshly cold-booted guest on the same host is perfectly fine. Genuine network/TLS misconfig: fails by host, subnet, or route, and a fresh guest fails exactly the same way.
• The openssl verdict — Frozen guest clock: "certificate is not yet valid" (only local time can produce this). Genuine network/TLS misconfig: "unable to get local issuer certificate", connection refused, or a timeout — statements about packets and trust stores, not dates.
• `date -u` inside the guest — Frozen guest clock: differs from the host by hours, days, or a whole month, matching your template bake date. Genuine network/TLS misconfig: matches the host to within a second or two.
• `chronyc tracking` — Frozen guest clock: 'System time' offset in hours or days, often 'Leap status: Not synchronised' because the offset was too big to slew. Genuine network/TLS misconfig: offset in milliseconds, synchronised, entirely healthy.
• Where the logs are — Frozen guest clock: present and indexed, but filed under the bake date, so your last-15-minutes dashboard is empty and you conclude nothing ran. Genuine network/TLS misconfig: right where you'd expect, timestamped now, with the errors you're looking for.
• What fixes it — Frozen guest clock: step the clock (`chronyc makestep`) and the failures stop instantly, with no other change. Genuine network/TLS misconfig: stepping the clock changes absolutely nothing, which is itself a useful result.

Verifying it from your own test suite

Once you know the failure mode exists, assert against it. This is a cheap test that catches an entire category of expensive incident, and it belongs in whatever suite runs against your sandbox provider — mine included. Create a sandbox, ask the guest what time it thinks it is, and compare against the machine running the test.

import time
from datetime import datetime, timezone
from pandastack import Sandbox

MAX_SKEW_SECONDS = 5.0


def guest_utc(sbx: Sandbox) -> datetime:
    """Ask the guest what year it thinks it is."""
    r = sbx.exec("date -u +%Y-%m-%dT%H:%M:%S", timeout_seconds=10)
    if r.exit_code != 0:
        raise RuntimeError(f"could not read guest clock: {r.stderr}")
    return datetime.strptime(
        r.stdout.strip(), "%Y-%m-%dT%H:%M:%S"
    ).replace(tzinfo=timezone.utc)


def test_restored_guest_is_not_living_in_the_past():
    # Every create here is a snapshot restore, so this exercises exactly
    # the path where a frozen clock would show up.
    sbx = Sandbox.create(template="base", ttl_seconds=300)
    try:
        host_now = datetime.now(timezone.utc)
        skew = abs((guest_utc(sbx) - host_now).total_seconds())
        assert skew < MAX_SKEW_SECONDS, (
            f"guest clock is {skew:.0f}s off -- it is living at bake time. "
            f"Outbound TLS will fail the moment an upstream cert rotates."
        )

        # The real-world assertion: can the guest complete a TLS handshake
        # against something with a recently-rotated certificate? A frozen
        # clock fails here with 'certificate is not yet valid'.
        r = sbx.exec(
            "curl -fsS -o /dev/null -w '%{http_code}' https://api.github.com",
            timeout_seconds=20,
        )
        assert r.exit_code == 0, f"outbound TLS failed: {r.stderr}"

        # And confirm the guest ISN'T merely coincidentally correct:
        # let a little wall-clock time pass and check it advanced.
        before = guest_utc(sbx)
        time.sleep(3)
        assert (guest_utc(sbx) - before).total_seconds() >= 2
    finally:
        sbx.kill()

The last assertion in that test looks paranoid and isn't. A guest can read the correct date at the instant you check it and still have a clock that isn't advancing properly — the two are separate failures, and "correct once" is not the same as "correct." Checking that time actually moves costs you three seconds and distinguishes a working clock from a lucky one.

A restored VM is not a machine that lost track of time. It's a machine that kept perfect track of the wrong time, which is why nothing inside it ever reports a problem. The first thing to notice is always something outside, and it's usually TLS — which then takes the blame for a crime committed by a memory image.

What to actually take away

Snapshot-restore is a genuinely excellent trade. It is how you get a fresh, hardware-isolated machine in ~49ms of restore instead of ~3 seconds of cold boot, and it is why per-request or per-agent-run VMs are economically sane at all. But every fast path has a bill attached, and this is the one nobody prints on the box: a snapshot preserves state faithfully, wall-clock time is state, and "faithfully preserved" and "correct" are different words. The same property that makes restore fast is the property that makes the guest wrong about the date.

So: sync the clock at restore, resume, and wake — all three, in the platform, where a single fix covers every workload including the ones whose authors have never thought about this. Run chrony inside the guest with `makestep` as a backstop, knowing it's a backstop and it races. Use `CLOCK_MONOTONIC` for durations and know it lies across a suspension boundary too, so anything holding a lease or a lock across one needs an external signal — VMGenID is the right one. Re-bake templates on a schedule to keep the drift window small. And put `date -u` at the top of your TLS runbook, above the firewall rules, because the five seconds it costs will one day save you an afternoon you had other plans for.