Testing LLM-Generated Database Migrations Safely in a Sandbox

AI coding agents write schema migrations now. You ask for "add a nullable phone column and backfill it," and the model hands you a SQL file — or worse, a raw `ALTER TABLE` it wants to run for you. The problem is that a migration is not like other generated code. A bad function call throws an exception; a bad migration mutates state, and some of those mutations are irreversible. "The migration is probably reversible" is a sentence people say exactly once. Running an unreviewed, model-authored migration against any real database — even staging — is how you get a `DROP TABLE users` at 2am because the model "cleaned up" a table it decided was unused.

The safe loop is simple and old: don't apply candidate migrations to real infrastructure. Apply them to a throwaway copy first. What's new is that ephemeral microVMs make that copy cheap enough to spin up per migration, per PR, or per agent iteration. This post walks the loop end to end with PandaStack: create a disposable VM with a Postgres seeded from your current schema, apply the candidate migration there, run smoke queries and your test suite, diff the resulting schema, and — only then — surface it to a human to approve against real infra. I'm Ajay, I built PandaStack, and I'll be honest about where the boundaries are.

Why not just run it against staging?

Staging feels safe because it's "not prod." But staging is shared, long-lived, and stateful — the three properties that make a bad migration expensive. Someone else is mid-test on it. Yesterday's migration already ran, so this one is being applied on top of drifted state you didn't reproduce. And when the model's migration locks a large table for eight minutes or drops a column another service still reads, you've now broken staging for the whole team, and rolling back means restoring a backup you hopefully took.

The thing you actually want is a database whose entire purpose is to be destroyed. It should look exactly like production's schema, hold no data anyone cares about, cost nothing when idle, and vanish the instant the check is done. Rollback isn't a `down` migration you're praying is correct — it's deleting the VM. That's the whole pitch for an ephemeral, per-migration microVM.

Three ways to get a throwaway database

Before the code, here's the honest comparison of how you get an isolated database to test against. All three work; they trade off fidelity, isolation, and speed differently. Verify competitors' behavior against their own docs — I'm describing the shape, not benchmarking their products.

• Run against staging directly — A: zero setup, real-ish data. B: shared and stateful, so a bad migration breaks everyone; rollback means a backup restore; you can't run twenty candidates in parallel; state drifts from prod. Correctness-of-rollback is the killer.
• Local Docker Postgres on the CI runner — A: cheap, fast, familiar. B: shares the runner's kernel (a resource-hungry or malicious migration can starve or affect neighbors on the box), containers on one host contend for the same page cache, and cleaning up a half-applied migration means tearing down and recreating the container — fine for one, awkward for many-in-parallel or truly untrusted SQL.
• Ephemeral microVM per migration (PandaStack) — A: each candidate gets its own hardware-isolated Firecracker VM with its own guest kernel; a runaway or destructive migration is contained to a disposable VM; fork a seeded snapshot to get many identical copies fast; rollback = delete the VM. B: you provision a VM (managed DB create is 30–90s, or run Postgres in a base sandbox), so it's heavier than `docker run` for a single one-off — the payoff is isolation and parallelism, not shaving milliseconds off a single local test.

The dividing line is trust and fan-out. If you're testing your own hand-written migration once, local Docker is genuinely fine. The moment the migration is model-authored (untrusted), or you want to test many candidates in parallel, or you want rollback to be "the VM is gone" rather than "I hope the down-migration is correct," the per-VM model earns its keep.

The safe loop, step by step

Here's the shape you're building. Each step is a gate; if any gate fails, the migration never reaches a human, let alone prod:

1. Provision a disposable Postgres — either a managed database VM, or a base sandbox running Postgres you start yourself.
2. Seed it from a schema snapshot — load your current production schema (structure only, or a scrubbed fixture set) so the target looks like prod.
3. Apply the candidate migration — the exact SQL the LLM produced, unedited, with a timeout.
4. Run checks — smoke queries, the app's test suite against this DB, and constraint/row-count assertions.
5. Diff the resulting schema — dump the post-migration schema and compare it to the pre-migration one; flag destructive operations (dropped tables/columns, type narrowing).
6. Throw the VM away — kill the sandbox. If the migration was bad, nothing survives. If it was good, you keep the diff + test report for a human to approve.

Step 1–2: a disposable Postgres, seeded from your schema

You have two ways to get Postgres. The managed database API gives you a real PostgreSQL 16 VM with a connection URL (create takes 30–90s because it blocks until Postgres is actually ready). For a tight test loop where you don't need durability, it's often simpler to create a `base` sandbox and run Postgres inside it — you control the lifecycle directly and the VM is gone the moment you kill it. This example takes the sandbox route: create the VM, write the current schema and the candidate migration into the guest, and initialize Postgres.

from pandastack import Sandbox

# Your current production schema (structure only) and the candidate migration
# the LLM produced. In practice you'd pg_dump --schema-only from prod once and
# cache it; the migration text comes straight from the model, unedited.
SCHEMA_SQL = """
CREATE TABLE users (
  id         bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  email      text NOT NULL UNIQUE,
  created_at timestamptz NOT NULL DEFAULT now()
);
CREATE TABLE orders (
  id       bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  user_id  bigint NOT NULL REFERENCES users(id),
  amount   numeric(12,2) NOT NULL
);
"""

# The migration under test — treat it as guilty until proven innocent.
CANDIDATE_MIGRATION = """
ALTER TABLE users ADD COLUMN phone text;
CREATE INDEX idx_orders_user_id ON orders(user_id);
"""

sbx = Sandbox.create(template="base", ttl_seconds=600)
try:
    # Bring up a throwaway Postgres inside the guest. No data anyone cares about.
    sbx.exec("apt-get install -y postgresql >/dev/null 2>&1 || true", timeout_seconds=120)
    sbx.exec("service postgresql start", timeout_seconds=30)
    sbx.exec(
        "sudo -u postgres createdb migration_test",
        timeout_seconds=30,
    )

    # Seed the throwaway DB so it looks like prod's schema.
    sbx.filesystem.write("/tmp/schema.sql", SCHEMA_SQL)
    seed = sbx.exec(
        "sudo -u postgres psql -v ON_ERROR_STOP=1 -d migration_test -f /tmp/schema.sql",
        timeout_seconds=60,
    )
    assert seed.exit_code == 0, seed.stderr
    print("seeded schema:", seed.exit_code)
except Exception:
    sbx.kill()
    raise

Step 3–5: apply the migration, then diff and test

Now capture the schema before, apply the candidate migration exactly as the model wrote it, capture the schema after, and diff. The diff is where destructive operations get caught: a dropped column, a narrowed type, a table that quietly disappeared. Pair it with smoke queries and, ideally, your app's test suite pointed at this database — a migration can be schema-valid and still break the application (renamed a column a query still selects).

    # --- Snapshot the schema BEFORE, then apply the candidate migration ---
    before = sbx.exec(
        "sudo -u postgres pg_dump --schema-only -d migration_test",
        timeout_seconds=60,
    )
    sbx.filesystem.write("/tmp/migration.sql", CANDIDATE_MIGRATION)

    applied = sbx.exec(
        "sudo -u postgres psql -v ON_ERROR_STOP=1 -d migration_test -f /tmp/migration.sql",
        timeout_seconds=120,  # a migration that locks forever is a finding, not a hang
    )
    migration_ok = applied.exit_code == 0

    # --- Snapshot the schema AFTER and diff structure ---
    after = sbx.exec(
        "sudo -u postgres pg_dump --schema-only -d migration_test",
        timeout_seconds=60,
    )

    # Cheap, effective destructive-op detector: dropped tables/columns show up as
    # lines present BEFORE but absent AFTER. A real pipeline would parse this into
    # a structured diff; a grep gets you 80% of the safety for 5% of the effort.
    sbx.filesystem.write("/tmp/before.sql", before.stdout)
    sbx.filesystem.write("/tmp/after.sql", after.stdout)
    diff = sbx.exec(
        "diff /tmp/before.sql /tmp/after.sql || true",
        timeout_seconds=30,
    )

    # --- Smoke query: does the app's hot path still work post-migration? ---
    smoke = sbx.exec(
        "sudo -u postgres psql -v ON_ERROR_STOP=1 -d migration_test -c "
        "\"INSERT INTO users(email, phone) VALUES ('[email protected]', '+15551234567'); "
        "SELECT count(*) FROM users;\"",
        timeout_seconds=30,
    )

    report = {
        "migration_applied": migration_ok,
        "migration_error": applied.stderr if not migration_ok else None,
        "schema_diff": diff.stdout,
        "smoke_ok": smoke.exit_code == 0,
        "smoke_error": smoke.stderr if smoke.exit_code else None,
        # Flag the scary verbs for the human reviewer up front.
        "looks_destructive": any(
            kw in CANDIDATE_MIGRATION.upper()
            for kw in ("DROP TABLE", "DROP COLUMN", "TRUNCATE", "DROP DATABASE")
        ),
    }
    print(report)
finally:
    # Step 6: rollback-by-deletion. The VM and everything in it is gone.
    sbx.kill()

That `report` dict is the artifact a human (or a gating step in CI) approves. It says: did the migration apply at all, what changed structurally, does a representative query still work, and does it contain destructive verbs. If `looks_destructive` is true or the diff shows a column vanished, you block and ask a person. If everything's green, a human still eyeballs the diff before it ever runs against real infrastructure — the sandbox proves the migration is applicable and non-crashing, not that it's correct for your business.

Fanning out: fork one seeded database into many

The seeding step (install Postgres, load the schema) is the slow part. If an agent is generating several candidate migrations — or you're testing one migration against several schema versions — you don't want to pay that setup cost N times. Set up one sandbox with Postgres seeded to your baseline, snapshot it, and fork the snapshot per candidate. Each fork is an identical, independent copy: a same-host fork lands in roughly 400–750ms (a cross-host fork is 1.2–3.5s), and forks share memory and disk copy-on-write, so ten forks don't cost ten times the RAM until they diverge.

This is the pattern behind running an agent's best-of-N migration attempts in parallel: fork the seeded DB once per candidate, apply a different migration in each, diff them all, and let the human compare. Each fork is its own microVM, so candidate three's `TRUNCATE` can't touch candidate one's data — the isolation boundary is the VM. For the deeper lifecycle of this pattern against per-branch and per-PR databases, see the companion post on the per-PR ephemeral database microVM.

The economics hold up because there's no warm pool of idle database VMs sitting around — a create restores a baked snapshot on demand (a create is p50 179ms, p99 ~203ms; ~49ms for the restore step itself, with a ~3s first cold boot before a snapshot exists). A single agent can allocate a lot of these: an agent host carries 16,384 pre-allocated /30 subnets, so per-VM networking isn't the bottleneck — memory is.

When to use a managed database instead

The sandbox-runs-Postgres approach above is best for a fast, fully-disposable test loop. But if you want the throwaway database to behave exactly like the managed PostgreSQL your app connects to in production — same template, same TLS connection URL, same durable-volume semantics — create a managed database instead and run the migration against its connection string. That path costs 30–90s per create (it blocks until Postgres is ready and healthy), so it's the right choice when fidelity to your real DB matters more than shaving the setup time, or when the migration tool you're testing expects a normal `postgres://` URL rather than a local socket. For how per-tenant managed databases isolate on their own microVMs, see the post on per-tenant database isolation.

Honest limits

A sandbox tests applicability and structure; it can't test what it can't reproduce. A few things to keep in mind:

• Data-dependent migrations need representative data. A backfill that's instant on an empty table can lock for minutes on 200M rows — seed synthetic data at production scale if migration duration is part of what you're checking.
• Concurrency and lock behavior differ under load. A throwaway DB with one connection won't surface a lock contention issue that only appears when live traffic is hitting the table. Test the migration's locking strategy (e.g. CONCURRENTLY) explicitly.
• The schema diff is only as good as its parser. A grep-level diff catches dropped tables and columns; subtle changes (a constraint's ON DELETE behavior, a default expression) deserve a structured schema-comparison tool.
• The human still approves. The sandbox's job is to make sure the human is only ever reviewing migrations that actually run and don't obviously destroy data — not to remove the human.

Within those bounds, the win is real and specific: an AI agent can now propose a migration, have it applied against a hardware-isolated throwaway Postgres, get a schema diff and a test report back in seconds, and never once put a `DROP TABLE` anywhere near a database that matters. Rollback is deleting a VM. That's a much better place to be at 2am than restoring a backup. For the closely related pattern of gating model-authored SQL queries (not just migrations), see sandboxing LLM SQL execution.