Letting an AI Agent Run Integration Tests Against Real APIs

Letting a coding agent write and run unit tests is close to a solved problem. Unit tests are pure functions over fake data: the worst outcome is a red build and some wasted tokens. Run them on your laptop, in CI, in a container, in a thread — nobody cares, because a unit test with no network and no credentials cannot do anything interesting to the outside world.

Integration tests are a different animal, and this is where teams quietly stop letting the agent run things. An integration test is only useful if it talks to a real API with real credentials over a real network — precisely the two capabilities you least want to hand a system that plans at runtime and occasionally hallucinates a hostname. I'm Ajay, I built PandaStack. This post is about the shape that makes it safe: one disposable Firecracker microVM per test run, scoped short-lived credentials injected after boot, a hard egress allowlist, and a TTL that kills the whole thing whether or not your code remembers to.

Why integration tests break the usual agent sandbox story

The standard advice for running model-written code is "put it in a sandbox with no network." That's correct and completely useless here: the entire point of an integration test is the network. You can't assert that checkout works against the payments API without reaching the payments API. So the question isn't whether the agent's code gets network and secrets — it's how narrowly you scope them, and what happens when the model gets it wrong.

And it will. The failure mode of a good coding agent isn't sabotage, it's confident competence pointed at the wrong target: a perfectly idiomatic test that reads its base URL from an environment variable which happens to be set to production.

The threat model: four ways this ruins your afternoon

The test that hits production instead of staging

This is the big one, and it's almost never a security failure — it's a configuration failure with security-sized consequences. The agent reads your README, finds an example curl against the production host, and templates it into the test. Or staging is unreachable so it "helpfully" falls back to the URL that works. On a laptop with production credentials in the ambient environment (and they are, because you needed them last Tuesday), nothing stops this. The test passes. It passed against prod.

The teardown that deletes real records

Every well-written integration test cleans up after itself, every LLM knows this, so it writes a teardown. The teardown deletes the resources the test created — or the ones it thinks the test created: a list-then-delete-all over a collection, a DELETE against a filter broader than the fixture, a "reset the test tenant" helper aimed at a tenant that turned out to be somebody's. It's the most dangerous pattern in agentic testing precisely because it looks like good hygiene.

The API key that ends up somewhere it shouldn't

Secrets leak through boring channels. A test fails, the failure output includes request headers, the headers include the bearer token, and that output goes straight into the agent's context window — logged by your observability stack, sent to a model provider, maybe pasted into an issue by a human debugging it. Nobody attacked anything; the key walked out through a stack trace. Long-lived and broadly scoped, that's an incident. A 15-minute read-only staging token is a shrug.

The retry loop that eats your rate limit and your bill

Agents iterate. That's the feature. But an agent debugging a flaky integration test will happily re-run it forty times in five minutes, and if that test hits a metered third-party API you have just built a very expensive fuzzer. Worse is the test that itself contains a retry loop with no backoff, hammering a partner's endpoint until they rate-limit your whole organization. Your CI didn't do this. Your agent did, at 3am, unattended.

The pattern: one disposable microVM per test run

The fix isn't a better prompt. It's making the four failures above structurally impossible — or at least contained — by giving each test run its own machine that can't do much. A Firecracker microVM per test run buys you four properties at once:

• Credential blast radius = one run. The only secrets in the guest are the scoped, short-lived ones you injected. No ambient AWS profile, no ~/.netrc, no kubeconfig, no SSH agent socket. The agent cannot leak what was never mounted.
• Egress is a policy, not a hope. Each sandbox gets its own network namespace and tap device, so "this VM may reach api.staging.example.com and nothing else" is enforced at the network layer, not in a system prompt the model may or may not honor.
• Cleanup is guaranteed by the platform. A hard ttl_seconds kills the VM on a timer whether the test hangs, the agent wanders off, or your orchestrator crashes. Teardown that depends on your code running isn't teardown; it's an intention.
• Parallelism is cheap and independent. Ten runs are ten VMs with ten kernels, filesystems, and network namespaces — no shared fixtures, ports, or temp files, which is the classic source of "passes alone, fails in the matrix".

The historical reason nobody did VM-per-test-run is that VMs were slow to create. Snapshot-restore removes that objection: on PandaStack every create restores a baked template snapshot on demand at p50 179ms (p99 around 203ms; the restore step itself is roughly 49ms), versus about 3 seconds for a first-ever cold boot. At sub-200ms, a fresh VM per iteration of the agent's loop costs less than the pytest collection phase.

Teardown, and actually proving the VM died

"The sandbox is deleted when the context manager exits" is fine until the process holding it is OOM-killed. Belt and braces: set ttl_seconds so the platform reaps the VM independently of your code, then verify — list sandboxes filtered by the metadata tag you set at create time and assert zero. Deleting a Firecracker microVM is a real deletion: guest kernel, memory, and copy-on-write rootfs clone all go away. That's a stronger statement than "the container exited" on a host where layers, volumes, and anything written through a bind mount are still sitting there.

Credentials: inject after restore, never bake

This is the rule people get wrong most often: a snapshot of a VM that had secrets in memory is a secret-bearing artifact. Firecracker snapshots capture guest RAM. Bake a template with an API token in an env var, or snapshot a VM after the suite loaded credentials into the process heap, and that token now lives in a memory image in object storage — replicated, restored into every VM created from it. That's not a vault; it's a multi-gigabyte blob your whole fleet reads.

So the sequence matters. Restore the clean snapshot first, then mint a short-lived scoped credential, then write it into the guest — onto tmpfs, not the rootfs — and let it expire on its own. The template snapshot stays boring and secretless and shareable across every run. If a credential does escape, it's read-only, scoped to one staging resource, and dead in fifteen minutes.

The egress allowlist: what the test is allowed to talk to

Default-deny outbound, then allow exactly the API hosts under test. This is the control that turns "the agent wrote a test against production" from an incident into a connection timeout with a legible error message — which, as a bonus, is a signal the agent can read and correct itself from. The guest-side shape, run before any credential touches the VM:

#!/usr/bin/env bash
# /opt/net/allowlist.sh -- runs INSIDE the test VM, BEFORE any secret lands.
# Usage: allowlist.sh api.staging.example.com auth.staging.example.com
set -euo pipefail

RESOLVER="${DNS_SERVER:-10.200.0.1}"

# 1. Default-deny egress. The agent's test can talk to nothing by default.
iptables -P OUTPUT DROP
iptables -A OUTPUT -o lo -j ACCEPT
iptables -A OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A OUTPUT -p udp -d "$RESOLVER" --dport 53 -j ACCEPT

# 2. Allow exactly the hosts under test, pinned to the IPs we resolved now.
for host in "$@"; do
  for ip in $(getent ahostsv4 "$host" | awk '{print $1}' | sort -u); do
    iptables -A OUTPUT -p tcp -d "$ip" --dport 443 -j ACCEPT
    echo "allow $host -> $ip:443"
  done
done

# 3. Log what got denied -- this is how the agent learns it aimed at prod.
iptables -A OUTPUT -j LOG --log-prefix "EGRESS-DENIED " --log-level 4

# 4. Secrets live on tmpfs ONLY: never on the CoW rootfs, never in a snapshot.
mkdir -p /run/secrets
mount -t tmpfs -o size=1m,mode=0700,noexec,nosuid tmpfs /run/secrets

# 5. Keep tokens out of shell history and out of core dumps.
export HISTFILE=/dev/null
ulimit -c 0

echo "egress locked to: $*"

Two honest caveats. IP-pinned allowlists drift: if the API sits behind a CDN with rotating addresses, a long-running VM can find its rules stale. For a run measured in minutes that's fine; allowing by hostname via a filtering proxy is more correct but more machinery. And in-guest iptables is defense in depth, not the boundary — the real boundary is the per-sandbox network namespace and tap device on the host. The guest can rewrite its own firewall all it likes and still can't reach anything the host-side netns doesn't route. Do both: in-guest rules give the agent fast, readable feedback; the namespace is what holds.

The agent's write-run-read-fix loop

Now the loop itself. The agent writes a test, you write it into the guest, run pytest, and hand the model back stdout, stderr, and the exit code — nothing else. Not the env file, not the token, not the process environment. It gets the assertion failure and the traceback, which is all it needs to fix the test and all it should ever see.

from pandastack import Sandbox

ALLOWED = ["api.staging.example.com", "auth.staging.example.com"]


def agent_writes_and_runs_tests(agent, task: str, max_iters: int = 5) -> dict:
    """Let the model write integration tests and actually RUN them -- in a VM
    that can only reach staging, holds only short-lived creds, and self-destructs."""
    # Minted per run, from YOUR IdP -- read-only, staging-scoped, 15 min.
    token = mint_scoped_token(ttl_seconds=900, scopes=["read:orders"])

    with Sandbox.create(
        template="base",
        ttl_seconds=1800,  # hard kill, even if this process dies
        metadata={"purpose": "agent-integration-tests", "task": task},
    ) as sbx:
        # 1. Lock down egress BEFORE anything secret exists in the guest.
        net = sbx.exec("bash /opt/net/allowlist.sh " + " ".join(ALLOWED),
                       timeout_seconds=60)
        assert net.exit_code == 0, net.stderr

        # 2. Inject credentials AFTER restore, onto tmpfs, guest-only. These
        #    never enter the snapshot and never enter the model's context.
        env = f"API_BASE=https://{ALLOWED[0]}\nAPI_TOKEN={token}\n"
        sbx.filesystem.write("/run/secrets/test.env", env.encode())
        sbx.exec("chmod 600 /run/secrets/test.env", timeout_seconds=10)

        failure = None
        for i in range(max_iters):
            # 3. The model writes the test; we never exec its shell directly.
            src = agent.write_test(task, previous_failure=failure)
            sbx.filesystem.write("/work/tests/test_generated.py", src.encode())

            run = sbx.exec(
                "set -a; . /run/secrets/test.env; set +a; "
                "cd /work && timeout 300 pytest -q tests/test_generated.py",
                timeout_seconds=360,
            )
            if run.exit_code == 0:
                return {"status": "pass", "iterations": i + 1, "test": src}

            # 4. Feed back ONLY stdout/stderr, redacted. A denied egress shows
            #    up here as a timeout -- the agent reads it and retargets.
            failure = redact(run.stdout + "\n" + run.stderr)

        return {"status": "fail", "iterations": max_iters, "last_error": failure}
    # VM destroyed on block exit; the token dies with it (and expires anyway).

The important detail is what is not in this code: no docker run with a mounted socket, no subprocess on your host, no shared temp directory. The agent's output goes into a file inside a VM, and a command runs inside that VM. If the generated test decides to shell out and rm -rf its way to a green build, it's deleting a copy-on-write rootfs clone with thirty minutes to live.

Record once, fork forever: stop hammering the real API

Here's where the microVM model stops being purely defensive and starts being faster than the alternatives. Most of an agent's iterations don't need a live API at all — iterations two through twenty fix an assertion, a JSON path, a datetime format. Only the first one needed real responses.

So run once live with the recorder on (VCR.py, responses, or a local mitm-style proxy writing cassettes to disk), then snapshot the VM with those fixtures captured. Every later iteration forks that snapshot instead of creating a fresh VM: same-host forks land in 400–750ms (cross-host is 1.2–3.5s), and the forked guest wakes with the cassettes on disk and the client already pointed at replay. The agent iterates against recorded traffic at local-disk speed, and the real API sees one request instead of forty.

The same fan-out gives you parallel test matrices nearly free. Fork the recorded snapshot N times — one per Python version, region config, or feature-flag combination — and run them concurrently. Each fork has its own kernel, filesystem, and network namespace, so there's no port contention and no shared fixture state, killing the entire genre of matrix flake where two jobs fight over one temp file. Copy-on-write memory and a reflinked rootfs mean forks share pages until they diverge, so twenty forks cost nowhere near twenty times one. An agent host pre-allocates 16,384 /30 subnets, so the ceiling is host CPU and RAM, not networking.

Agent's host vs. container vs. ephemeral microVM

Three places you can run agent-written integration tests, from most convenient to most contained. Verify the specifics of any runtime or CI product against its own docs — behavior varies sharply by version, config, and how locked-down your particular setup is.

• Credential blast radius — Agent's host: everything the developer has. Cloud CLI profiles, SSH keys, kubeconfig, and every .env in every sibling repo are one os.environ away. Container: whatever you passed in, plus what the image bakes, plus anything reachable through a bind mount or a mounted docker socket. Ephemeral microVM: only what you injected after restore, on tmpfs, scoped and short-lived — the guest starts with no ambient identity.
• Egress control — Agent's host: whatever the developer's network allows, i.e. the internet plus the corporate VPN. Container: possible with custom networks and host firewall rules, but usually left wide open because it's fiddly and everything works without it. Ephemeral microVM: per-sandbox netns and tap device, so default-deny plus an allowlist is the normal configuration, not a project.
• Isolation from your data — Agent's host: none. A confused teardown can delete your working tree. Container: process and namespace isolation, but a shared host kernel — a container is a polite suggestion to the kernel, and escapes cross the boundary. Ephemeral microVM: hardware-virtualized guest kernel via KVM; escaping needs a hypervisor break, not a syscall trick.
• Parallel test matrix — Agent's host: serial, or a mess of port collisions. Container: workable, though shared kernel resources and port mapping leak between jobs. Ephemeral microVM: fork the recorded snapshot N ways at 400–750ms each, independent kernels and netns, copy-on-write keeping the marginal cost low.
• Cleanup guarantee — Agent's host: your teardown code, plus hope. Container: docker rm, if the orchestrator survives; volumes and images commonly outlive the run. Ephemeral microVM: TTL enforced by the platform independent of your code — deletion takes the guest kernel, its RAM, and its CoW disk with it.
• Setup cost — Agent's host: zero, which is why everyone does it. Container: a Dockerfile and some flags. Ephemeral microVM: an SDK call; create is p50 179ms via snapshot-restore, so it feels like "run this function" rather than "provision an environment".

When you don't need any of this

Plenty of agentic testing needs none of this, and pretending otherwise is how you end up with infrastructure nobody uses. If the agent writes pure unit tests with no network and no secrets, run them wherever is fastest — an in-process runner beats a VM and the isolation buys nothing. If your integration tests hit a hermetic local stack you spin up yourself (a Postgres container, a fake API server, testcontainers), you already have the boundary: the secrets are fake and the egress is loopback. And if you're one developer with a disposable staging account whose credentials genuinely can't hurt anything, disciplined environment variables are honest and cheap.

The microVM pattern earns its keep where three things overlap: the credentials are real, the endpoints are reachable from where the code runs, and the code was written by something that doesn't flinch at a hostname. That's when "the agent will be careful" stops being a control and starts being a wish. A disposable VM with scoped credentials, an egress allowlist, and a TTL turns the four scary failures — wrong environment, destructive cleanup, leaked keys, runaway loops — into a timeout, an expired token, and a machine that no longer exists.