End-to-end tests with a real app and a real database

A test suite with mocked dependencies tells you your code does what you told the mock to expect. That's a real thing to know and it catches real bugs. It is also structurally incapable of catching the bugs that actually reach production, because those live in the gaps between components — the unique constraint your mock didn't have, the transaction that wasn't actually a transaction, the migration that's fine on ten rows, the connection pool that exhausts under concurrency.

So you want end-to-end tests against a real app and a real database. Everyone knows this. The reason they don't have them is operational: real environments are slow to create, expensive to keep, and shared ones become flaky in a way that erodes trust until people start re-running failed builds without reading them.

The shared environment is the root of the flakiness

Most teams end up with one staging environment and one test database, and it degrades in an entirely predictable way.

• Tests interfere. One test's leftover rows change another's results. Someone adds ordering assumptions that hold only when the table is empty.
• Cleanup is unreliable. Truncating between tests is slow and one forgotten table poisons everything after it. Transaction rollback per test is faster but breaks the moment your code commits internally, which real code does.
• Parallelism is impossible, so the suite takes forty minutes and people stop running it locally.
• Someone is always debugging in it. A person poking at staging while CI runs against it produces failures nobody can reproduce.
• Schema drift. The shared database has been migrated by hand three times. Nobody can recreate it from scratch and everybody is slightly afraid of it.

Each individually is fixable. Together they produce a suite people don't trust, and an untrusted suite is worse than no suite, because it still costs forty minutes and everybody ignores what it says.

One environment per run

The structural fix is to stop sharing: every test run gets its own app instance and its own database, created at the start and destroyed at the end. Every problem above disappears — not mitigated, gone. There's no interference because there's nothing to interfere with, no cleanup because you delete the whole thing, no drift because it's built from migrations every time, and parallelism is free because runs are independent.

The reason this isn't universal is that it's only practical if creating an environment is fast and cheap. Provisioning a VM used to take minutes; nobody spends four minutes per test run on setup. Containers made this tolerable and testcontainers-style tooling made it common for databases specifically.

With snapshot-restore the numbers change again: creating a microVM is about 179ms at p50. Creating a database with real Postgres is slower — 30 to 90 seconds, because it waits until Postgres genuinely accepts connections — but you can clone an existing one instead of building from scratch, and clone the already-migrated, already-seeded state rather than repeating that work per run.

import os
import subprocess
import requests
from pandastack import Sandbox

# 1. Clone a database that is ALREADY migrated and seeded. Cloning warm
#    state beats rebuilding it: no migration run, no fixture load per test.
API = "https://api.pandastack.ai/v1"
H = {"Authorization": f"Bearer {os.environ['PANDASTACK_API_KEY']}"}

clone = requests.post(
    f"{API}/databases/{os.environ['TEMPLATE_DB_ID']}/clone",
    headers=H, json={"label": f"ci-{os.environ.get('GITHUB_RUN_ID', 'local')}"},
).json()

# create/clone is async -- poll until Postgres actually accepts connections
db = poll_until_running(clone["id"])      # your helper; 30-90s
DATABASE_URL = db["connection_url"]

try:
    # 2. A fresh microVM for the app under test (~179ms p50 to create).
    with Sandbox.create(template="base", ttl_seconds=1800) as app:
        app.filesystem.write("/app/src.tar", open("build/src.tar", "rb").read())
        app.exec("cd /app && tar xf src.tar && npm ci --omit=dev")

        # Run migrations against the clone, so the test also PROVES the
        # migration applies cleanly to production-shaped data.
        app.exec(f"cd /app && DATABASE_URL='{DATABASE_URL}' npm run migrate",
                 timeout_seconds=300)

        app.exec(
            f"cd /app && DATABASE_URL='{DATABASE_URL}' "
            "setsid nohup npm start >/var/log/app.log 2>&1 &",
        )
        base_url = wait_for_http(app, port=3000, timeout_s=60)

        # 3. Point the real test suite at the real thing.
        subprocess.run(
            ["npx", "playwright", "test"],
            env={**os.environ, "BASE_URL": base_url},
            check=True,
        )

        # On failure, grab the app's own logs before the VM disappears.
finally:
    requests.delete(f"{API}/databases/{db['id']}", headers=H)
    # sandbox is destroyed by the context manager; ttl_seconds is the backstop
    # if this process is killed mid-run, which CI cancellation does often

Clone warm state instead of rebuilding it

The biggest speed win isn't creating things faster — it's not repeating expensive setup. A typical environment build runs migrations, loads fixtures, warms caches, and builds assets. That's most of the setup time, it's identical every run, and it's pure waste after the first time.

So do it once and clone the result. Keep a template database that's migrated and seeded, and clone it per run. Keep a snapshot of a VM with dependencies installed and the app built, and restore that. A fork inherits memory and disk copy-on-write — same-host forks land in 400 to 750 milliseconds — so twenty parallel test shards can each start from the same fully warm state rather than each doing the setup.

One caution worth stating: forking captures state the process considered unique at fork time, including random seeds and in-memory session keys. Forked processes producing identical 'random' values is a real and well-documented class of bug. Re-seed after fork; treat that as part of the pattern rather than an optimisation.

What belongs in an end-to-end test

Real environments are more expensive than unit tests even when they're fast, so spend them on what only they can find.

• Migrations against production-shaped data, timed. A migration that's instant on fixtures may hold a lock for forty minutes on real data. Failing the build when a migration exceeds a time budget converts an outage class into a red build.
• Transaction boundaries. Does a failure halfway through leave a half-written record? Mocks never model this because mocks don't have transactions.
• Constraint violations. Unique indexes, foreign keys, check constraints, and the error paths that fire when they're hit — which are usually the least-tested code in the system.
• Concurrency. Two requests racing for the same row, pool exhaustion, deadlocks. These need a real database with real locking.
• The full request path. Auth, middleware, serialisation, and the ten places a mocked HTTP client quietly differed from the real one.

What doesn't belong: business logic that a unit test covers faster, and anything you're testing end-to-end only because the unit test would have required refactoring. That second one is how a 40-minute suite grows, one reasonable-seeming decision at a time.

Parallelism, once isolation is real

The payoff of per-run environments is that the suite scales horizontally. Twenty shards, twenty environments, wall-clock time divided by twenty rather than by nothing.

Two things to get right. Shard by cost rather than by file count — one shard containing all the slow tests defeats the point, and most runners can shard by historical duration. And label every environment with the run and shard that created it, so an abandoned one can be traced to a job instead of sitting anonymously until a sweep deletes it.

The endpoint worth aiming for: an end-to-end suite fast enough to run on every pull request, isolated enough that a failure means something, and cheap enough that nobody argues about it in planning. That's achievable now in a way it wasn't a few years ago, and the thing that changed is simply how fast a real environment can be created.