Database Branching with Copy-on-Write MicroVMs

There is a specific flavour of afternoon that every backend engineer has lived through. You need to know what a migration does to real data — not to the eleven rows in your fixtures, but to the table with two hundred million rows and the index nobody remembers adding. So you pull the nightly dump. It is 400 GB. You start the restore, you go get coffee, you come back, it is 40% done, and somewhere in there you accept that today is now about the restore and not about the migration. This is how you discover that your afternoon has an owner and it isn't you.

I'm Ajay; I built PandaStack. This post is about database branching — the primitive that makes "give me a private copy of real data, now" a sub-second operation instead of an afternoon — and specifically about the two very different ways it gets built. One branches the storage layer at the page level. The other branches the whole machine: disk and memory, so the branch wakes up with a warm buffer pool and a Postgres that has no idea it was cloned. Both are copy-on-write. They fail in different places, and the differences matter more than the marketing does.

What "database branching" actually means

The pitch is a metaphor borrowed from git: branch your data the way you branch your code. Underneath the metaphor, the concrete promise is narrower and more useful. A branch is a database that (a) contains real, production-shaped data as of some moment, (b) is independently writable, (c) is invisible to everyone else's branches, (d) appears essentially instantly regardless of how large the data is, and (e) can be deleted without ceremony.

Property (d) is the one that requires engineering. The other four you can get today with `pg_dump` and patience. The reason branching feels like a new capability rather than a rebranded restore is that it decouples the cost of having a copy from the size of the thing being copied — which is exactly what copy-on-write does, and why every implementation of database branching is a copy-on-write system wearing a product name.

Neon and PlanetScale are the two vendors that made the term mainstream, and they took visibly different routes — worth verifying against their own docs rather than trusting my summary, since both products move fast. Neon's model is storage-layer: compute is separated from a multi-version page store, and a branch is described as a copy-on-write clone that shares the parent's data and records subsequent writes as deltas, with branches creatable as of an earlier point in time within a retention window. PlanetScale's model grew out of Vitess and schema safety: branches are isolated MySQL instances that start from the source branch's schema, development branches are where you make schema changes, and those changes reach production through a deploy request with a schema diff, review, and conflict checking against the deploy queue rather than by direct DDL. One optimizes for cheap data copies at arbitrary points in time; the other optimizes for never letting a dangerous DDL statement touch production unreviewed. Both call it branching.

Two ways to build it: branch the storage, or branch the machine

Storage-layer branching

In the storage-layer approach, you rewrite the bottom of the database. The engine's pages don't live on a local filesystem; they live in a log-structured, multi-version page store that keeps the history of every page rather than only its current contents. WAL is ingested into that store, and any page can be reconstructed as of any position in the log within the retention window.

Branching in that world is almost free by construction, because the store already holds the history. Creating a branch means writing a small amount of metadata: a new namespace that says "reads fall through to the parent's pages as of position X; writes go into my own delta." No data moves. The parent isn't touched and isn't slowed down. And because the branch point is a position in a log, branching "as of two hours ago" is the same operation as branching "as of now" — time travel comes free with the data structure, which is the real elegance of the design.

The costs are structural. You have to build and operate a distributed page store, which is a serious systems project. You are coupled to one engine's page format and WAL semantics, so it doesn't generalize to whatever else is in your stack. And a branch's compute is a fresh process: it starts with an empty buffer pool, an empty plan cache, and cold statistics, so the first queries against a brand-new branch pull pages over the network and behave nothing like the parent did. Your branch has your data but not your parent's warmth.

Whole-machine branching

The other approach doesn't modify the database at all. You run the database inside a microVM, and you branch the microVM. A fork captures both halves of the machine's state — the disk and the RAM — and hands you a second machine that resumes from that exact instant.

This is a strictly stronger notion of "copy" than the storage-layer one, and the difference is entirely in the memory. The branch inherits the parent's shared buffers already populated with hot pages. It inherits the running postmaster, its backends, its prepared statement caches, its planner statistics as loaded, its extension state, the OS page cache underneath. It does not "start Postgres and connect to a copy of the data." It is the same Postgres, continued, on a second machine. Which means it isn't Postgres-specific either — the mechanism has no idea what's running inside the guest, so branching MySQL, ClickHouse, Redis, or an entire application stack with its database is the same operation.

The honest cost: the branch's fidelity is now tied to the fidelity of the snapshot, and a database is the single most demanding thing you can ask a snapshot to capture correctly. More on that below, because it's the part most write-ups skip.

Storage-layer branching vs. whole-machine fork

• What is branched — Storage-layer branching: the page store beneath the engine; a new namespace pointing at the parent's pages as of a log position. Whole-machine fork: the entire microVM — rootfs blocks and guest RAM together — so the running process is branched, not just its bytes on disk.
• Where copy-on-write happens — Storage-layer branching: inside the store, per page, at the granularity the engine writes. Whole-machine fork: at two layers at once — filesystem extents (XFS reflink) or block ranges (dm-snapshot) for the disk, and 4 KiB pages via MAP_PRIVATE for the memory.
• What the branch inherits — Storage-layer branching: the data, and only the data; the compute is a fresh, cold process. Whole-machine fork: the data plus the warm buffer pool, the live postmaster, the planner state, and every cache the parent had populated.
• First-query behaviour — Storage-layer branching: cold — pages fault in from the store, so early queries are unrepresentative of the parent's performance. Whole-machine fork: warm — the branch performs like the parent did at the instant it was forked, which is what you want when you are timing a migration.
• Point-in-time branching — Storage-layer branching: continuous within the retention window; any log position is a valid branch point. Whole-machine fork: discrete — you can only branch from snapshots you took, so your granularity is your snapshot cadence. This is a real limitation, not a footnote.
• Engine coupling — Storage-layer branching: deep; the store implements one engine's page and WAL semantics, and porting to another engine is a rewrite. Whole-machine fork: none; the hypervisor doesn't know or care what is running inside the guest.
• Operational surface — Storage-layer branching: you (or your vendor) run a distributed storage system with its own failure modes, compaction, and retention. Whole-machine fork: you run VMs, which you were already doing, with snapshot artifacts as ordinary files in object storage.
• Cost of a branch — Storage-layer branching: metadata plus the deltas the branch writes. Whole-machine fork: metadata plus the disk blocks and memory pages the branch dirties. Both converge on the same economics — a branch costs what it changes — but the whole-machine fork also carries the branch's live RAM footprint for as long as it exists.

Storage-layer branching copies the data. A whole-machine fork copies the machine that was already thinking about the data.

The copy-on-write mechanics, honestly

Nothing about a whole-machine fork is magic, and it's worth seeing the primitives at their real size. The disk half is a reflink: on a reflink-capable host filesystem, cloning a file creates a second file whose extents are shared with the first, with the filesystem recording that those extents now have two owners. The work is proportional to the number of extents, not the number of bytes, so a 4 GiB database image and a 40 GiB one clone in roughly the same handful of milliseconds.

# The trunk: a Postgres microVM's rootfs, fully migrated and seeded.
$ ls -lh /var/lib/pandastack/templates/postgres-16/rootfs.ext4
-rw-r--r-- 1 root root 4.0G Jul 29 09:14 rootfs.ext4

# A full copy is O(data): it reads and writes every byte.
$ time cp rootfs.ext4 branch-a.ext4
real    0m9.104s

# A reflink clone is O(metadata): the extents get a second owner.
# Constant time in the size of the image.
$ time cp --reflink=always rootfs.ext4 branch-b.ext4
real    0m0.004s

# Free space is unchanged by the reflink -- there is ONE physical copy
# of those blocks with two names pointing at it. (Note that `du` will
# still report 4.0G for each file; it counts referenced extents, not
# unique ones. `df` is the honest witness here.)
$ df -h --output=avail /var/lib/pandastack | tail -1
 812G

# Same contract one layer lower, for hosts whose filesystem cannot
# reflink: a read-only origin device plus a private COW store that
# absorbs every block this branch writes.
$ dmsetup create branch-c --table \
    "0 8388608 snapshot /dev/vg0/pg-trunk /dev/vg0/branch-c-cow P 8"

The memory half is the same idea applied to RAM. When the snapshot's memory file is mapped `MAP_PRIVATE`, the guest's pages are backed by that shared file and nothing is eagerly copied. A read fault maps the page straight out of the shared file; a write fault copies exactly that one 4 KiB page private to this branch and leaves the original intact for every other branch. Postgres's shared buffers, which are the expensive thing to warm up, are shared across every branch until a branch dirties them.

Put together, forking a live database VM is a metadata clone of the disk, a private mapping of the memory, and a resume. On PandaStack that lands in roughly 400–750ms on the same host. Cross-host is 1.2–3.5s, and the reason is worth internalizing rather than memorizing: reflink only works within one filesystem, and the parent's resident RAM is not on the destination machine, so the artifacts have to travel before any of the cheap tricks apply. Branch locality is a real performance property. The deeper mechanics are in /blog/copy-on-write-rootfs and /blog/copy-on-write-memory-fork-explained.

The cost model: a branch costs what it writes

This is the part to hold onto, because it predicts everything about how branching behaves at scale. A branch begins sharing essentially all of its disk blocks and memory pages with its parent, and diverges only as it writes. Its marginal storage cost is the set of blocks it dirties, and its marginal memory cost is the set of pages it dirties.

• A branch that runs read-only analytics against production-shaped data is nearly free on disk, forever. It touches nothing.
• A branch that inserts a few thousand test rows and runs a suite pays for a few thousand rows' worth of blocks, plus whatever WAL it generates. Rounding error.
• A branch that runs `ALTER TABLE` on a large table and triggers a full rewrite pays for the entire table, twice — old extents stay pinned by the parent while the branch allocates new ones. This is a genuinely useful signal: your storage bill for the branch tells you exactly how much of the table your migration rewrote.
• A branch that lives for weeks under continuous writes eventually approximates a full copy. Copy-on-write buys you cheap creation, not cheap immortality. Branches are meant to be deleted.
• Fan-out is cheaper than depth intuitively suggests: fifty branches off one trunk, each writing a little, cost the trunk plus fifty small deltas — not fifty databases. This is why per-developer and per-agent branches are economically sane in a way that fifty restored dumps never were.

What people actually do with a branch

Four use cases account for nearly all of the value, and they're all variations on "I need to do something to real data that I am not allowed to do to real data."

The newest one, and the one that changed my mind about how important this primitive is: an AI agent that has been asked to write a migration. A coding agent can generate plausible-looking DDL from a schema dump all day, but plausible DDL is not the deliverable — the deliverable is DDL that has been run, against realistic data, with the lock behaviour and duration observed. An agent with a branch can propose a migration, run it, read the error, fix it, and run it again, iterating against ground truth instead of against its own confidence. Give it a shared staging database instead and you have handed an autonomous system write access to the environment your entire team depends on, which is a category of decision that tends to generate a retrospective.

The second is per-developer branches — a real database each, seeded once, forked on demand, with no coordination and no Slack thread titled "don't touch the DB this afternoon." The third is destructive-migration rehearsal, which is really the honest version of the first: `DROP TABLE` on a branch is a learning experience; `DROP TABLE` on production is a résumé event. The fourth is time travel — branching from an earlier snapshot to answer "what did this row look like on Tuesday, before the backfill ate it," without restoring anything or touching the live system.

Here is the shape of it against PandaStack's managed Postgres. Each managed database is its own Firecracker microVM with a durable volume; a create takes 30–90 seconds because it blocks until Postgres is genuinely accepting connections. You pay that once, for the trunk. Everything after is a fork.

from pandastack import Sandbox

# --- ONE TIME: build the trunk. This is the slow, boring part: a real
# managed Postgres (create is 30-90s, because it waits for the server to
# actually accept connections), your full migration chain, and a restore
# of production-shaped data. Do it nightly in CI and never think about it.
trunk = Sandbox.create(
    template="postgres-16",
    persistent=True,                  # durable volume, exempt from the idle reaper
    metadata={"role": "db-trunk", "schema": "head"},
)
trunk.exec("./migrate.sh up", timeout_seconds=1800)
trunk.exec("pg_restore -d pandastack /seed/production_shaped.dump",
           timeout_seconds=3600)

# Quiesce before freezing. A CHECKPOINT flushes dirty buffers so the
# on-disk state is coherent, which makes every branch's recovery a no-op
# instead of an adventure. Then warm the cache you want branches to inherit.
trunk.exec("psql -c 'CHECKPOINT'")
trunk.exec("psql -c 'SELECT count(*) FROM orders'")   # pull hot pages in

# Freeze the whole machine: rootfs AND RAM, including shared_buffers and
# the running postmaster. This is the artifact every branch forks from.
trunk_snap = trunk.snapshot()


def branch(name: str) -> Sandbox:
    """Fork the live database. Same-host: 400-750ms. Cross-host: 1.2-3.5s.

    The branch resumes with the trunk's buffer pool already warm, so it
    performs like the parent did -- which is the whole point when the
    thing you are measuring is how long a migration takes.
    """
    sbx = Sandbox.fork(trunk_snap.id, metadata={"branch": name})
    # Fence the branch before anyone connects: it inherited the trunk's
    # identity, so stop it from shipping WAL to the trunk's archive.
    sbx.exec("psql -c \"ALTER SYSTEM SET archive_mode = off\" "
             "&& pg_ctl reload")
    return sbx


# --- An agent gets a real database to run a real migration against.
b = branch("agent/0042-drop-legacy-columns")
b.filesystem.write("/work/0042.sql", agent_generated_migration)

out = b.exec("psql -v ON_ERROR_STOP=1 -f /work/0042.sql", timeout_seconds=900)
if out.exit_code != 0:
    # Ground truth, not a hallucinated review. Hand the real error back to
    # the agent and let it try again on a fresh branch.
    feedback = out.stderr

# Native TLS connection string, routed by SNI on the sandbox's own ID:
#   postgres://pandastack:<pw>@<id>.db.pandastack.ai:5432/pandastack
print(b.connection_url)

b.kill()   # the branch, its data, and its very bad ideas all disappear

And the part that actually earns its keep — what you run once you're inside a branch. Not "did the migration succeed," which your CI already tells you, but the questions you cannot ask anywhere else.

-- Running on a BRANCH. The distance between this and running it on
-- production is the distance between a learning experience and a
-- conversation with your VP.

\timing on

-- 1. How long does the rewrite take against production-SHAPED data?
--    Your fixtures answered this in 4ms and were lying to you.
ALTER TABLE orders DROP COLUMN legacy_customer_ref;

-- 2. What did it lock, and what was queued behind it? Run this from a
--    SECOND session while the ALTER is still in flight.
SELECT  a.pid, a.state, l.mode, l.granted,
        now() - a.query_start AS waited,
        left(a.query, 60) AS query
FROM    pg_locks l
JOIN    pg_stat_activity a USING (pid)
WHERE   l.relation = 'orders'::regclass
ORDER BY a.query_start;

-- 3. How much of the table did the migration actually rewrite? On a
--    copy-on-write branch this is also, almost exactly, the storage bill.
SELECT  pg_size_pretty(pg_total_relation_size('orders')) AS after_size,
        n_tup_upd, n_tup_hot_upd
FROM    pg_stat_user_tables WHERE relname = 'orders';

-- 4. Did anything downstream quietly break? The view that referenced the
--    column you just dropped would like a word.
SELECT  c.relname, pg_get_viewdef(c.oid, true)
FROM    pg_class c
WHERE   c.relkind = 'v'
  AND   pg_get_viewdef(c.oid, true) ILIKE '%legacy_customer_ref%';

-- No ROLLBACK needed. The branch has about ninety seconds to live.

The honest limits of branching a live database

Forking a machine that is running a database is the case where snapshot semantics stop being an abstraction and start being your problem. These are the things that bite, in roughly the order they bite.

A branch is not a backup. This is the most common confusion and the most expensive one. A branch's disk is a copy-on-write clone that shares blocks with its parent and lives on one host; it is not replicated, not archived, and — by design — about to be deleted. Its durability is the durability of a machine you have already decided to throw away. If someone starts keeping "the good branch" around because it has data they need, you no longer have a branching system, you have an unbacked-up production database with a casual name.

`fsync` inside a branch means something narrower than you think. When Postgres fsyncs in the guest, the write reaches the host's storage stack for that branch's CoW device — which is real durability against a guest crash, and no durability at all against the branch simply ceasing to exist. That distinction is fine, and correct, as long as nobody mistakes a branch's WAL for a durability guarantee that outlives the branch.

A forked Postgres inherits its parent's identity, and that identity was not designed to have two holders. The branch carries the same system identifier, the same `pg_control` state, the same timeline, and the same replication slots. If the trunk archives WAL or streams to a replica, the branch will cheerfully try to do the same thing — to the same archive, on the same timeline, with divergent content. That is how you corrupt an archive with a feature that was supposed to be safe. Fence the branch on resume: archiving off, replication slots dropped, any outbound connection strings the guest holds rewritten to point somewhere harmless.

Connections do not survive, and shouldn't. The branch's memory contains the parent's TCP sockets, but the network underneath was rebuilt for a new machine with a new address, so every one of those connections is a corpse holding a file descriptor. Fork from a quiesced trunk with no meaningful sessions open, and let clients dial the branch fresh on its own connection string. Similarly, anything the guest cached about its own hostname or advertised address is now wrong — the branch has a new identity in DNS, and pgBouncer or an application config baked into the image will happily keep pointing at the trunk.

A restored guest also wakes up believing it is still bake day, which for a database is not a cosmetic problem — `now()` feeds default values, partition routing, retention jobs, and TTL logic. The platform has to sync the guest clock on restore and resume; PandaStack does, but if you are building this yourself, put it on the list before you find it in a partition that got written into last month.

And the structural one: your branch points are discrete. A log-structured page store can branch from any position in the WAL within retention; a whole-machine fork can only branch from snapshots that exist. If you want hourly time travel you take hourly snapshots, and your recovery granularity is your snapshot cadence. It is an honest trade — you gain engine-independence and a warm machine, you lose continuous point-in-time — and pretending otherwise would be the kind of claim that gets found out on the day it matters.

The summary

Database branching is one property wearing several product names: the cost of having a private, writable copy of real data should not depend on how much data there is. Everything else follows from that. Copy-on-write is how you get it, and the only real design question is which layer you apply it at.

Branch the storage and you get continuous point-in-time branch points and a very elegant data structure, at the price of building an engine-specific distributed page store and handing every branch a cold cache. Branch the whole machine and you get the parent's warm buffer pool, its running postmaster, and total engine-independence — a reflink or dm-snapshot for the disk, `MAP_PRIVATE` for the memory, a resume on top — at the price of discrete branch points and a real obligation to quiesce before you freeze and fence after you fork. On PandaStack that's a 30–90 second managed Postgres create for the trunk, once, and 400–750ms per branch on the same host thereafter.

The thing worth ending on isn't the latency, it's what the latency changes. When a private copy of real data costs half a second, you stop rationing it. Every developer gets one. Every pull request gets one — that specific pattern is in /blog/per-pr-ephemeral-database-microvm. Every agent that wants to try a migration gets one, and gets to be wrong in a place where being wrong is free. The 400 GB restore that owned your afternoon was never the interesting part of the work. It was just the toll.