Your Backup Is a Hypothesis: DR Drills in Disposable microVMs

Open your incident runbook and find the step that says "restore from backup." Read what comes after it. In most runbooks I have been shown, the answer is: nothing. That is the whole step. Five words standing in for a procedure nobody has executed end to end, against a backup nobody has read back, into an environment nobody has stood up, inside a time budget somebody typed into a compliance document because the form had a field for it.

I'm Ajay; I build PandaStack, a Firecracker microVM platform, and disaster-recovery drills keep showing up as an isolation problem wearing a reliability costume. The reason teams do not drill is not laziness. It is that a real drill needs production data in a real environment, and the two environments everyone actually has are production (insane) and a staging stack that stopped resembling production sometime in 2024 (useless). This post is about building the third one on demand, and throwing it away afterwards.

A backup you have never restored is a hypothesis

Backups fail quietly and in ways that only surface on the way back out. The job exits zero because the upload succeeded, not because the dump is coherent. The retention policy silently rolled off the only copy taken before the corruption. The archive is encrypted with a key that lives in the system you are trying to recover. Half the schema is in a second cluster nobody put in the job. The dump has been truncated for six weeks because a disk filled up on a host whose alerting was migrated and never re-pointed. Every one of those is invisible from the backup side and obvious ten seconds into a restore.

RPO and RTO in a compliance document are fiction until a stopwatch has been involved. Until then they are aspirations with units attached.

What a real drill actually has to prove

"We ran a DR test" covers an enormous range of rigor, from restoring a 40 MB dump on a laptop to genuinely reconstituting a service. A drill that is worth the calendar invite has to close four claims, in order, and each one fails independently of the others.

1. The backup is readable. Not present, not the right size, not listed in the console — readable, by the tool you would use, at the version you would use it at, with the key you can actually obtain during an outage.
2. The procedure in the runbook is the procedure that works. Run the document verbatim. If you improve it while running it, you tested yourself, not the document, and the person who runs it at 03:00 will not have you.
3. The application comes up against the restored data. A restored database is not a restored service. Migrations may be ahead of the dump, a required extension may be missing, and the app may be perfectly happy to boot and be quietly wrong.
4. The whole thing fits inside the RTO you promised. Measured from the moment the incident starts, including the part where somebody has to find the runbook, not from the moment `pg_restore` begins.

The reason nobody drills: there is nowhere to do it

Restoring into production is obviously out. That leaves the staging environment, which fails for a duller reason than danger: it does not resemble the thing you are trying to prove you can rebuild. Different Postgres minor version, different extensions, a schema three migrations ahead or behind, a dataset that is 200 MB of fixtures instead of the shape and volume that makes your restore slow. A successful restore into staging proves that a small synthetic dataset can be loaded into a machine that is not production. Nobody was worried about that.

Size is the part people underestimate. Restore time is not linear in anything convenient — index rebuilds, WAL replay, and constraint validation dominate, and they dominate differently at 20 GB than at 2 TB. An RTO validated against fixtures is a number about fixtures. So the environment has to be production-shaped and production-sized, which means it has to hold production data, which means the moment you build it you have built a second production — with the same customer records, the same PII, and, if you are careless, the same reach.

Restored data is production data, and a half-restored app is a live one

This is the part that turns a DR drill into an incident report, and it is always the same shape. The restore works. The application boots against real rows. And then the application does what it was written to do: it reconciles, it retries, it sends. The rows say a thousand password resets are pending, so it sends a thousand password resets. The queue says these charges never settled, so it settles them. Your drill has just become an outbound event affecting real customers, and the postmortem sentence is "we were testing our disaster recovery."

• Email and SMS. Every transactional path fires against real addresses, and the classic version of this is a startup queue that drains on boot into tens of thousands of real inboxes.
• Payment processors. Live API keys in the restored config plus a reconciliation job is how a drill issues refunds, retries charges, or cancels subscriptions in a real account.
• Webhooks and third-party callbacks. Your partners cannot tell a drill from production, and they will happily act on what you send them.
• The message bus. A restored consumer with the real broker credentials starts competing for production partitions and eating messages the real service needed.
• Object storage and DNS. A restored deploy job that writes to the production bucket or updates a real record does not need to be malicious to be catastrophic.
• The cloud metadata endpoint. A link-local address that hands out credentials scoped to whatever role the drill host happens to carry is a fine way to give a test environment production powers.

The microVM shape: a sealed environment per exercise

The shape that works is one disposable microVM environment per drill, created for the exercise and destroyed at the end. It gets production's data and none of production's reach. That inversion is the whole trick: the drill needs the data to be real, and needs everything else about it to be fake.

• Its own network namespace and its own guest kernel, so the drill's Postgres and the drill's app are not sharing a machine with anything you care about.
• Default-deny egress with exactly one allowed destination: the backup store, read-only. No SMTP, no payment API, no internal service discovery, no metadata endpoint.
• Production-sized data restored from the real backup chain — the same artifact your on-call would reach for, not a curated export.
• Inert-but-valid credentials pointed at local sinks, so attempted side effects are recorded rather than either executed or silently swallowed.
• A TTL on the VM, because an environment holding a full copy of customer data must not outlive the exercise that justified it. The drill box that quietly ran for eight months is its own compliance finding.
• Metadata on the sandbox — drill id, backup artifact, owner — so "what is currently holding a production data copy" is a query rather than an archaeology project.

The practical argument for microVMs here is that the environment stops being a standing asset you maintain and becomes a thing you conjure. On PandaStack a sandbox is created by restoring a baked snapshot rather than cold-booting: p50 179ms, p99 203ms, with the restore step itself around 49ms, and only the first-ever boot of a template costing about 3 seconds. The restore of your data is the slow part, as it should be. The environment around it is not.

# drill.py -- one sealed microVM per disaster-recovery drill.
# It gets production DATA and none of production's REACH: no route to
# internal services, egress limited to the backup store, and every
# secret replaced with something valid-looking and inert.
import json
import time
from pandastack import Sandbox

DRILL_ID = "2026-Q3-payments-restore"
BACKUP_URI = "s3://acme-backups/payments/2026-08-25T02:00Z/base.dump"

# The runbook, verbatim, exactly as the on-call engineer would find it.
# Do not fix it while running it -- a bug found here is the deliverable.
RUNBOOK = open("runbooks/payments-restore.sh").read()

# Credentials that parse and refuse. The local sinks matter: a missing
# key turns the send path into a silent no-op, and a silent no-op reads
# as success in the drill report.
FAKE_ENV = "\n".join([
    "SMTP_HOST=127.0.0.1",
    "SMTP_PORT=2525",                       # sink that logs and drops
    "STRIPE_API_KEY=sk_test_drill_inert_0000",
    "WEBHOOK_BASE_URL=http://127.0.0.1:9",  # discard port
    "S3_ENDPOINT=http://127.0.0.1:9",
    "DRILL=1",
])


def run_drill():
    started = time.time()
    sbx = Sandbox.create(
        template="base",
        ttl_seconds=14400,          # a drill VM must not outlive the drill
        metadata={
            "drill": DRILL_ID,
            "purpose": "dr-drill",
            "data": "production-copy",   # so audit can find it later
            "egress": "backup-store-only",
            "owner": "platform-oncall",
        },
    )
    sbx.filesystem.write("/drill/runbook.sh", RUNBOOK)
    sbx.filesystem.write("/drill/app.env", FAKE_ENV)
    sbx.filesystem.write("/drill/verify.sql", open("drills/verify.sql").read())

    # One call, one stopwatch. timeout_seconds is set to the RTO you
    # published, so blowing the budget is a failure the harness reports
    # rather than something you notice in the logs on Monday.
    r = sbx.exec(f"bash /drill/runbook.sh {BACKUP_URI}", timeout_seconds=3600)

    result = {
        "drill": DRILL_ID,
        "exit_code": r.exit_code,
        "runbook_ms": r.duration_ms,
        "wall_s": round(time.time() - started, 1),
        "timeline": sbx.filesystem.read("/drill/timeline").decode(),
        "verify": sbx.filesystem.read("/drill/verify.out").decode(),
        "attempted_sends": sbx.filesystem.read("/drill/sink.log").decode()[-4000:],
        "stderr_tail": r.stderr[-4000:],
    }

    if r.exit_code == 0:
        # Restored and healthy. This exact state is the most valuable
        # object of the day -- freeze it before anyone touches it.
        sbx.snapshot()

    print(json.dumps({k: v for k, v in result.items() if k != "timeline"}, indent=2))
    return sbx, result   # caller kills it when the game day is over

Note the `attempted_sends` field. The most useful artifact of my last few drills was not the restore timing, it was the local sink's log — proof that the restored application, left alone for four minutes, tried to make several thousand outbound calls it would have made against real customers if this had been staging with real keys in it.

The restore itself: numbers, not adjectives

The runbook script is the part that has to be boring and instrumented. Timestamp every phase boundary, because the aggregate number is useless for planning — knowing that fetching the artifact dominates points at a different fix than knowing that index rebuilds do. And put the verification query in the same script, so "we restored it" and "we checked it" cannot drift apart.

#!/usr/bin/env bash
# runbooks/payments-restore.sh -- runs INSIDE the sealed drill VM.
# Every phase boundary is stamped, because "about twenty minutes" is a
# feeling, not an RTO.
set -uo pipefail

BACKUP="${1:?usage: payments-restore.sh s3://bucket/path/base.dump}"
DB=payments
now() { date -u +%s.%3N; }
mark() { echo "$1 $(date -u +%Y-%m-%dT%H:%M:%SZ) $(now)" >> /drill/timeline; }

t0=$(now); mark drill.start

# 1. Is the artifact readable AT ALL? Fetch and checksum before you
#    spend forty minutes discovering it was truncated on 12 July.
aws s3 cp "$BACKUP" /drill/base.dump
sha256sum /drill/base.dump | tee /drill/artifact.sha256
pg_restore --list /drill/base.dump > /drill/toc.txt || {
  echo "FATAL: dump is not readable by pg_restore"; exit 2; }
t_fetch=$(now); mark restore.fetch.done

# 2. The restore. -j is the knob people forget; measure with the value
#    the runbook actually tells the operator to use.
createdb "$DB"
pg_restore --no-owner --no-privileges --exit-on-error -j 4 -d "$DB" /drill/base.dump
rc=$?
t_restore=$(now); mark restore.done
[ "$rc" -eq 0 ] || { echo "RESTORE FAILED rc=$rc"; exit 1; }

# 3. Verification: prove the bytes mean something. Row counts, the
#    freshness your RPO claims, and one invariant that only holds if
#    the data is internally consistent.
psql -tAX -v ON_ERROR_STOP=1 -d "$DB" > /drill/verify.out 2>&1 <<'SQL'
SELECT 'orders='           || count(*)        FROM orders;
SELECT 'customers='        || count(*)        FROM customers;
SELECT 'newest_order='     || max(created_at) FROM orders;
SELECT 'rpo_gap_seconds='  || extract(epoch FROM now() - max(created_at))::int
  FROM orders;
SELECT 'orphan_payments='  || count(*) FROM payments p
  LEFT JOIN orders o ON o.id = p.order_id WHERE o.id IS NULL;
SELECT 'null_totals='      || count(*) FROM orders WHERE total_cents IS NULL;
SQL
verify_rc=$?
t_verify=$(now); mark verify.done

# 4. The app, against the restored data, with the inert env. A restored
#    database is not a restored service.
set -a; . /drill/app.env; set +a
/opt/payments/bin/migrate --check >> /drill/app.log 2>&1
setsid /opt/payments/bin/serve >> /drill/app.log 2>&1 &
for _ in $(seq 120); do
  curl -sf localhost:8080/healthz >/dev/null && break; sleep 1
done
curl -sf localhost:8080/healthz >/dev/null || { echo "APP NEVER CAME UP"; exit 3; }
t_app=$(now); mark app.healthy

printf 'fetch+read %.1fs\nrestore    %.1fs\nverify     %.1fs\napp up     %.1fs\nTOTAL      %.1fs\n' \
  "$(echo "$t_fetch - $t0"        | bc)" \
  "$(echo "$t_restore - $t_fetch" | bc)" \
  "$(echo "$t_verify - $t_restore"| bc)" \
  "$(echo "$t_app - $t_verify"    | bc)" \
  "$(echo "$t_app - $t0"          | bc)"
cat /drill/verify.out
exit "$verify_rc"

Measuring RTO honestly means starting the clock earlier than you want to

The number in the script above is a data-layer restore time. It is not your RTO, and reporting it as one is how organisations end up confidently wrong by an hour. RTO is measured from the start of the incident to the return of service, and it includes every human step you keep forgetting to count.

• Detection to decision. How long between the alert and someone declaring that a restore is happening? On a real night this is frequently the largest single block, and it is entirely absent from your script.
• Finding the artifact. Which backup, from when, at which of your two RPO tiers? The drill should make the operator choose, not hand them a URI.
• Getting the key and the credentials. If the secret manager is in the failure domain, the drill discovers that. This is the failure I see most often and the one nobody writes down.
• Provisioning the target. Cold infrastructure has its own lead time, and in a real regional event it has a queue in front of it.
• The restore and verification themselves — the part you actually measured.
• Cutover: migrations, DNS or connection-string changes, cache invalidation, and the reconciliation of everything that happened during the gap.
• Confidence. The interval between "the service is up" and "we are willing to send traffic to it," which is real time on a real clock even though no machine is doing anything.

Fork the restored state so five engineers can break it five ways

The second half of a good game day is failure injection: now that the system is back, break it and see whether the recovery steps in the runbook work. The traditional constraint is that you have one restored environment, so scenarios run sequentially, each one contaminating the next, and by scenario four you are debugging the residue of scenario two. Five engineers take turns while four of them read Slack.

Snapshotting the restored-and-healthy state removes that constraint. Freeze it once, then fork it per scenario: each child starts from identical memory and an identical copy-on-write disk, which means every engineer gets their own universe to ruin and every result is comparable to every other. On PandaStack a same-host fork lands in the 400–750ms range and a cross-host fork in the 1.2–3.5s range, so "give me another restored payments stack" stops being a scheduling negotiation. It also means a scenario that goes badly wrong costs you a `kill()` rather than the rest of the afternoon and a second restore.

# gameday.py -- failure injection against the RESTORED state, in
# parallel, with every scenario starting from identical bytes.
from concurrent.futures import ThreadPoolExecutor
from pandastack import Sandbox

# Each scenario is a thing that has actually happened to somebody,
# paired with the runbook step that claims to fix it.
SCENARIOS = {
    "primary-killed":   "pkill -9 -f 'postgres.*checkpointer'",
    "disk-full":        "fallocate -l $(df --output=avail -m /var | tail -1)M /var/hog",
    "dns-blackhole":    "iptables -A OUTPUT -p udp --dport 53 -j DROP",
    "connection-storm": "pgbench -c 400 -j 8 -T 90 payments",
    "partial-restore":  "psql -d payments -c 'TRUNCATE ledger'",
}


def inject(parent: Sandbox, name: str, break_cmd: str) -> dict:
    child = parent.fork()          # same restored data, copy-on-write
    try:
        child.filesystem.write("/drill/scenario", name)
        broke = child.exec(break_cmd, timeout_seconds=180)

        # The actual question: does the documented recovery step work?
        fix = child.exec("bash /drill/recover.sh", timeout_seconds=900)
        health = child.exec(
            "curl -s -o /dev/null -w '%{http_code}' localhost:8080/healthz"
        ).stdout.strip()

        return {
            "scenario": name,
            "injected_exit": broke.exit_code,
            "recover_exit": fix.exit_code,
            "recover_ms": fix.duration_ms,
            "health": health,
            "tail": fix.stderr[-1500:],
        }
    finally:
        child.kill()               # one scenario, one VM, no residue


parent, result = run_drill()
assert result["exit_code"] == 0, "no point injecting faults into a failed restore"

try:
    with ThreadPoolExecutor(max_workers=5) as pool:
        outcomes = list(pool.map(lambda kv: inject(parent, *kv), SCENARIOS.items()))
finally:
    parent.kill()                  # the drill environment dies with the drill

for o in sorted(outcomes, key=lambda o: o["recover_ms"], reverse=True):
    ok = "OK " if o["recover_exit"] == 0 and o["health"] == "200" else "FAIL"
    print(f"{ok} {o['scenario']:<18} recover={o['recover_ms']}ms health={o['health']}")
# A FAIL here is the point of the exercise. It means a step in the
# runbook does not do what the runbook says it does.

Write down the failures without softening them. "The documented recovery step for a full disk assumes a log-rotation cron that was removed in March" is the entire value of the day. A game day where everything worked either means you are in excellent shape or means you picked scenarios you already knew the answer to, and the second is much more common than the first.

If the database is managed, the drill changes shape but not purpose

When the database is a managed service, you are no longer drilling `pg_restore` — you are drilling the restore control plane, the clock, and everything downstream of it. PandaStack's managed Postgres, for example, supports point-in-time restore and cloning into a new database from the archive, so a drill can ask for a copy as of a specific timestamp and get a separate database with its own id while the source is untouched. That makes the destructive half of the exercise safe by construction, since the thing you are about to abuse is a clone.

What that does not do is exempt you from measuring. Provisioning a managed Postgres has its own cost — creation lands in the 30–90 second range on our side, before any of your data arrives — and the restore duration for your dataset is a number you have to obtain by running it, not by reading a docs page. Time it yourself, in your region, at your data size, with your extensions, and check the same invariants you would check after a self-managed restore. Then point the application at the restored copy and confirm it comes up, because "the database restored" and "the service recovered" remain different claims no matter who operates the database.

The best drill is the one nobody scheduled

An annual game day proves your backup chain worked on one Tuesday in October. Everything that breaks it — a schema change, a new service with its own database, a rotated key, a bucket lifecycle rule, a job that started exiting zero on failure — happens in the eleven months in between. The drill you actually need is the one that runs every week without a calendar invite and tells you the day it stops working.

Fully automated restore-verify is a much smaller ask than a game day: create a sealed sandbox, pull the most recent backup, restore it, run the assertions, record the phase timings, destroy the environment, fail the pipeline if anything regressed. It needs no humans and no meeting, and because the environment is created and destroyed per run there is no standing box accumulating a production data copy. Track the phase timings as a series rather than a pass/fail — restore time creeping toward your RTO over six months is the signal you want, and it is invisible if you only check the exit code.

Four places to run a restore drill

Same exercise, four venues. Characterizations of any specific product's isolation, provisioning, and billing behavior should be verified against that vendor's own documentation, since those details differ by configuration and they change.

• Fidelity to production — Production itself: perfect, and that is the problem. Long-lived staging: whatever it was when someone last cared, typically a different minor version and a fixture dataset. Cloud VMs provisioned per drill: as faithful as your infrastructure code, which is a real and useful test in itself. Sealed microVM environment: template baked from the production image, restored with the real backup artifact at real size.
• Blast radius if the app starts sending — Production itself: there is no blast radius, only production. Long-lived staging: usually holds at least some real credentials, which is how drills email real customers. Cloud VMs provisioned per drill: a real boundary, but it inherits the account's IAM role and VPC routes unless you were deliberate. Sealed microVM environment: own guest kernel, own network namespace, default-deny egress with one allowlisted destination.
• Time to get an environment — Production itself: instant and unavailable. Long-lived staging: instant, until you find someone else is mid-release on it. Cloud VMs provisioned per drill: minutes of boot and provisioning per attempt, which quietly caps how many scenarios fit in a day. Sealed microVM environment: snapshot restore at p50 179ms / p99 203ms; the data restore is the slow part, as it should be.
• Parallel failure-injection scenarios — Production itself: not applicable, please stop. Long-lived staging: one at a time, with each scenario contaminating the next. Cloud VMs provisioned per drill: parallel if you pay for N environments and restore the data N times. Sealed microVM environment: snapshot the restored state once and fork per scenario — same-host forks in the 400–750ms range, identical starting bytes.
• Cleanup and data hygiene — Production itself: n/a. Long-lived staging: the production copy from the last drill is still sitting there, and probably in the backup of staging by now. Cloud VMs provisioned per drill: terminate the instance, then remember the volume, the snapshot, the security group. Sealed microVM environment: kill the VM and memory, disk, and every running process go with it, with a TTL as the backstop for the ones you forget.

None of this is free. You are maintaining a drill template, a runbook that is executable rather than prose, and an assertion suite that has to be updated when the schema moves. The honest version of the trade is that you have converted an annual ritual into a piece of infrastructure, and infrastructure needs owners.

What you get back is that the sentence changes. Instead of "we have backups," you get "we restored last Thursday's artifact into a sealed environment, the data-layer restore took this long, the app came up against it, two runbook steps were wrong and are now fixed, and here is the timeline file." The first sentence is a hypothesis. The second one has a stopwatch in it. And the difference between them is usually discovered at the worst possible moment by the person least equipped to enjoy it.