Sandbox Mode for Your API: Give Every Customer a microVM

Every platform with integrators eventually ships a sandbox. Stripe has test mode, Plaid has its sandbox institutions, Twilio has test credentials — a place where a developer can create a fake charge, link a fake bank, send a fake SMS, and get their integration working without moving real money or waking anyone up at 3am. It is one of the highest-leverage things an API company builds, because it is where a developer decides whether your product is pleasant before anybody signs anything.

It is also, in most companies, one shared deployment called "sandbox" running a slightly older build, with a `test_` prefix on the IDs and an `is_test` boolean on every row. That works beautifully on day one. It rots by month six, and it rots in ways that are invisible to you and extremely visible to the customer trying to integrate.

I'm Ajay, I built PandaStack — this post is about the other design: give each customer a real environment. A microVM running your actual API image next to a real Postgres, forked from a snapshot you seeded once, handed over with credentials that are worthless outside the guest, resettable with a button, and destroyed when they stop showing up.

How a shared sandbox cluster rots

The failure modes are boringly predictable, which is the good news — you can name them all before you build anything. The bad news is that none of them are fixable inside the shared-cluster design; they're consequences of it.

• Test data collisions — everyone's fixtures live in the same tables. Customer A's integration test creates a customer named "Test User" and then asserts there is exactly one. Two months later there are forty, half of them created by your own QA. Every integrator eventually writes defensive code that only exists because your sandbox is a shared landfill, and that code ships to production.
• One customer's load test wrecks everyone's demo — sandbox is where people benchmark, because it's free and nobody told them not to. So somebody runs 5,000 requests per second against it on a Tuesday afternoon, and a different customer's live investor demo starts timing out. You find out from a support ticket, not a pager.
• Rate limits that don't mirror prod — the sandbox is usually either wide open (so integrators never build backoff and get surprised on their first production day) or aggressively throttled because of the load-test problem (so integrators build backoff against limits that don't exist in prod). Either way the thing they tested against is not the thing they'll run against.
• It isn't the same code path — this is the one that actually costs you money. `if is_test:` branches accumulate. The sandbox skips the fraud check, stubs the ledger write, short-circuits the webhook signer, and runs on a build that's two weeks behind. "It worked in sandbox" then means "it worked in a program that shares a name with yours."
• Nobody can reset it — because the data is shared, "reset my sandbox" cannot be a button. It's a ticket, an engineer, and a DELETE with a WHERE clause that someone will eventually get wrong. So integrators don't reset; they just accumulate garbage and lose confidence in their own test assertions.

What "a real environment" actually means

The alternative is to make the unit of isolation a machine instead of a column. A customer's sandbox is a microVM. Inside it runs the same API artifact you deploy to production — same binary, same config shape, same migrations — talking to a real PostgreSQL instance with real constraints, real transactions, and real foreign keys that will actually reject the malformed thing they just POSTed. The `is_test` branch disappears, because the environment is what makes it a test, not a flag on the row.

That single change kills most of the list above by construction. There is no test-data collision because there is no shared table. A load test saturates one guest with its own fixed vCPU and RAM ceiling, and the noise stays inside the blast wall. Rate limits are your production rate-limiter, running on their environment, configured with their plan's numbers. And "reset" becomes a lifecycle operation on a VM rather than a data-surgery procedure on a shared database.

From the integrator's side, the whole thing should be three calls and a base URL that looks like your production one.

# 1. Provision a private sandbox environment (returns in well under a second).
curl -sX POST https://api.example.com/v1/sandbox/provision \
  -H "Authorization: Bearer $LIVE_KEY" | jq
# {
#   "sandbox_id": "3f2a91c4-...",
#   "base_url":   "https://8080-3f2a91c4-....sandbox.example.com",
#   "api_key":    "sk_sbx_9f1c...",
#   "expires_at": "2026-09-02T11:04:19Z"
# }

# 2. Point the integration at it. Same routes, same errors, same rate limiter,
#    same migrations -- it is your production artifact, on their machine.
curl -sX POST "$SANDBOX_URL/v1/charges" \
  -H "Authorization: Bearer $SANDBOX_KEY" \
  -d amount=1999 -d currency=usd -d customer=cus_fixture_alice

# 3. Wreck it freely, then put it back exactly how it started.
curl -sX POST https://api.example.com/v1/sandbox/reset \
  -H "Authorization: Bearer $LIVE_KEY"
# { "status": "ready", "reset_in_ms": 612 }

Shared cluster vs container namespace vs microVM

The middle option — give each customer their own namespace or their own container on a shared cluster — is a genuine improvement and worth naming honestly, because for a lot of teams it's the right stopping point. It fixes data collisions. It does not fix the two things that hurt most: kernel-level isolation and instant reset.

• Test data — Shared sandbox cluster: one set of tables for everyone; collisions and defensive test code forever. Per-customer container namespace: a schema or database per customer, so collisions are gone but the reset path is still "delete rows in dependency order". Per-customer microVM: the customer's data is the only data in the guest, and throwing away the guest is the reset.
• Blast radius of a load test — Shared cluster: one integrator's benchmark degrades every other integrator's demo. Container: cgroup CPU and memory limits help, but you share a kernel, a page cache, and usually a database server. MicroVM: fixed vCPU and RAM baked into the guest; a runaway saturates only itself.
• Code-path fidelity — Shared cluster: `if is_test:` branches accumulate until sandbox is a different program. Container: usually the real artifact, though the database and dependencies are often shared services with test-only config. MicroVM: your production image plus a real Postgres in the same guest; nothing is stubbed because nothing needs to be.
• Reset to a known state — Shared cluster: a ticket and an engineer. Container: a TRUNCATE script you must maintain in lockstep with every migration, plus whatever state lives outside Postgres. MicroVM: fork the seeded snapshot again — 400-750ms same-host, 1.2-3.5s cross-host — and kill the old guest.
• Credential scope — Shared cluster: sandbox keys authenticate against a real multi-tenant service, so a leaked one is a real-ish credential. Container: narrower, but usually still holds a connection string to shared infrastructure. MicroVM: the key is only meaningful to one disposable guest and is worthless the moment it's destroyed.
• Idle cost — Shared cluster: always on, sized for the worst-behaved integrator, paid for at 3am. Container: cheaper per tenant, but the pool underneath still runs continuously. MicroVM: there is no warm pool; create is a snapshot restore at p50 179ms, so an idle customer costs nothing until they come back.
• Isolation strength — Shared cluster: application-level, i.e. your ORM's WHERE clause. Container: namespaces and cgroups over a shared host kernel. MicroVM: hardware-virtualized guest with its own kernel — the same model AWS Lambda uses to separate untrusted tenants.

Seed once, fork per customer

The naive version of this design is expensive: provision a VM, install Postgres, run migrations, load fixtures, start the API, wait for health. That's minutes, and if you put it on the hot path your "instant sandbox" button spins for so long that customers assume it's broken. Creating a managed PostgreSQL instance per customer is worse for the same reason — that's a 30-90s create, because a real Postgres has to bootstrap and pass a readiness check before you can hand out a connection string. You do not want any of that between a developer clicking a button and seeing a base URL.

So don't do it per customer. Do it once, at release time, and snapshot the result. Bake a golden environment: your API artifact, a real Postgres, migrations applied, deterministic fixtures loaded, the server warm and answering on its port. Snapshot that. Every customer sandbox after that is a fork of the snapshot — copy-on-write memory pages shared until something writes, a reflinked rootfs clone instead of a copy — which lands in 400-750ms on the same host. The slow, boring, error-prone part happens in CI where it belongs.

"Deterministic" is doing real work in that sentence. Fixtures must have fixed IDs, fixed timestamps, and a fixed clock relationship, because your integrators are going to write assertions against them, and those assertions have to survive a reset. `cus_fixture_alice` should be the same Alice with the same three invoices in every sandbox, forever, until you cut a new golden snapshot and version it.

import json, hashlib
from pandastack import Sandbox

FIXTURES = open("fixtures/seed.sql").read()


def bake_golden_env(release: str):
    """Runs once per API release, in CI. Everything slow happens HERE,
    never on the path between a customer clicking a button and getting a URL."""
    sbx = Sandbox.create(template="base", ttl_seconds=1800)

    # Your real production artifact + a real Postgres in the SAME guest, so a
    # single fork clones both and they can never drift out of sync.
    sbx.filesystem.write("/srv/api/release.tar.gz", read_release_artifact(release))
    sbx.exec("tar -xzf /srv/api/release.tar.gz -C /srv/api", timeout_seconds=120)
    sbx.exec("apt-get install -y postgresql-16 && service postgresql start",
             timeout_seconds=600)

    # Deterministic fixtures: fixed ids, fixed timestamps. Integrators will
    # write assertions against these, and the assertions must survive a reset.
    sbx.filesystem.write("/srv/api/seed.sql", FIXTURES)
    sbx.exec("cd /srv/api && ./bin/migrate up && psql -f seed.sql", timeout_seconds=300)

    # Start the API and wait until it actually answers, so the snapshot is WARM.
    # A fork of a warm process comes back warm -- no cold start on the hot path.
    sbx.exec("cd /srv/api && ./bin/serve --port 8080 &", timeout_seconds=30)
    sbx.exec("until curl -sf localhost:8080/health; do sleep 0.2; done",
             timeout_seconds=120)

    snap = sbx.snapshot()          # <- the golden environment, versioned by release
    sbx.kill()
    return snap


def provision_customer_sandbox(golden, customer_id: str) -> dict:
    """Hot path: fork the golden env. No migrations, no seeding, no waiting."""
    env = golden.fork(
        ttl_seconds=14 * 24 * 3600,          # abandoned sandboxes die on their own
        metadata={"customer": customer_id, "kind": "sandbox-env"},
    )

    # Per-environment credentials. Worthless outside this guest, by construction.
    key = "sk_sbx_" + secrets_token()
    env.filesystem.write("/srv/api/tenant.json", json.dumps({
        "customer_id":  customer_id,
        "api_key_hash": hashlib.sha256(key.encode()).hexdigest(),
        "webhook_url":  lookup_webhook(customer_id),
        "rate_limit":   plan_limits(customer_id),   # their PROD limits, not a guess
    }))
    env.exec("kill -HUP $(cat /run/api.pid)", timeout_seconds=30)

    # Sandbox previews are host-routed: https://<port>-<sandbox-id>.<your-suffix>
    return {
        "sandbox_id": env.id,
        "api_key":    key,
        "base_url":   f"https://8080-{env.id}.sandbox.example.com",
    }

"Reset my sandbox" is a button, not a ticket

In the shared-cluster world, reset means writing a teardown script: truncate every table in foreign-key dependency order, reset every sequence, re-run the seed, and remember to clear whatever lives outside Postgres — the Redis keys, the S3 objects, the queued jobs, the idempotency-key cache that will now happily swallow the customer's next request because it thinks it already saw it. That script has to be maintained in lockstep with every migration forever, and the day it drifts, reset silently leaves debris and your integrator's tests start failing for reasons neither of you can explain.

With a per-customer environment there is no teardown script, because you never clean anything. You fork a fresh guest from the same golden snapshot, re-apply the customer's identity, flip their route to the new one, and kill the old guest with all its accumulated mess still inside it. Whole-machine rollback beats selective deletion every time, and it's fast enough to be synchronous behind a button.

def reset_customer_sandbox(golden, customer_id: str) -> str:
    """The 'Reset my sandbox' button. We do not clean anything -- we replace it.
    Fork the golden snapshot again, swap the route, destroy the old guest."""
    old = current_env_for(customer_id)          # may be None if it TTL'd out

    fresh = golden.fork(
        ttl_seconds=14 * 24 * 3600,
        metadata={"customer": customer_id, "kind": "sandbox-env"},
    )

    # Identity is re-applied, not preserved: same key hash, same webhook target,
    # same plan limits -- so the customer's existing config keeps working.
    rehydrate_identity(fresh, customer_id)
    fresh.exec("until curl -sf localhost:8080/health; do sleep 0.2; done",
               timeout_seconds=60)

    swap_route(customer_id, fresh.id)           # atomic flip of their base URL

    if old:
        old.kill()   # orphaned rows, half-finished jobs, poisoned idempotency
                     # cache, that one 4GB test upload -- all gone with the guest

    return fresh.id

Two details worth stealing. First, reset should be idempotent and cheap enough that customers use it constantly — put it in your CI docs and let integrators call it in a test fixture's setup step. Second, keep the customer's stable identity outside the guest (in your control plane) and re-apply it on every fork, so a reset changes the data but never the API key or the webhook URL. Nothing sours a reset button faster than making the developer go re-copy a credential afterwards.

Webhooks, credentials, and blast radius

Webhook testing is where shared sandboxes get genuinely embarrassing. The integrator wants your platform to POST a signed event to their laptop or their staging endpoint, then wants to trigger that event on demand, and wants to see exactly what got sent when it didn't arrive. On a shared cluster this is a queue everyone contends for and a delivery log you have to carefully filter by tenant before you dare show it to anyone.

In a per-customer environment, the webhook sender is inside the guest. It signs with the same code as production, retries with the same backoff as production, and writes its delivery log to a file that belongs to exactly one customer — so "show me the last 50 deliveries with request and response bodies" is a `filesystem.read` away, with no cross-tenant filtering to get wrong. Triggering a synthetic event is an exec into their own machine. And because each guest gets its own network namespace with its own routing, you can enforce sane egress policy per environment rather than fleet-wide: allow outbound HTTPS so deliveries reach the customer's endpoint, deny link-local so nothing can reach a cloud metadata service, deny your internal ranges so a sandbox can never talk to production infrastructure.

Credentials get better for the same structural reason. A sandbox API key doesn't authenticate against a real multi-tenant service — it authenticates against one disposable guest whose entire contents are fixture data. A customer can paste it into a public repo, a screenshot, or a support ticket (they will do all three) and the worst outcome is that a stranger creates fake charges in a machine that expires in two weeks. That is a much better Tuesday than the shared-cluster version, where a leaked sandbox key is a live credential against shared infrastructure and you get to have a conversation with your security team about it.

Scale to zero, and TTLs for the ones nobody comes back to

The predictable objection is cost: a whole VM per customer sounds worse than one shared cluster. It's usually the reverse, and the reason is idle. The shared cluster is sized for peak, over-provisioned specifically so that one integrator's load test can't take it down, and paid for continuously — including all night, and including every month for the customers who evaluated you in March and never came back.

Per-customer environments invert that. There is no warm pool to keep fed, because create is a snapshot restore rather than a boot: the restore step is around 49ms and end-to-end create is p50 179ms, p99 around 203ms. (A genuine cold boot is ~3s, and only happens the first time a template's snapshot gets baked.) So a sandbox nobody is using does not need to exist. Hibernate it on idle and restore it when the next request arrives; the developer notices nothing, and you stop paying for six months of an abandoned evaluation.

The shared sandbox's real cost isn't the traffic it serves. It's the capacity you keep idle to survive your worst-behaved integrator.

Then set TTLs, and be unsentimental about it. Most sandbox environments are created during an evaluation and abandoned within a fortnight. A `ttl_seconds` on creation means the abandoned ones reap themselves without a cleanup cron you'll forget to monitor, and because provisioning is a sub-second fork, an expired sandbox is not a loss — if the customer comes back, they get a fresh one from the current golden snapshot, which is better than the stale one they left. Density is not the constraint you'd expect either: copy-on-write means the hundredth fork of a snapshot is not the hundredth copy of your API and Postgres, and each agent pre-allocates 16,384 network slots, so you run out of RAM long before you run out of plumbing.

None of this is free. You now version golden snapshots alongside releases, you run a router that maps customers to guests, and you own a fixture set that has to stay deterministic across migrations. If you have six integrators and a small API, the shared cluster with an `is_test` column is genuinely the right call — build the boring thing. But if sandbox mode is how developers evaluate your platform, then a sandbox that's slow, dirty, shared, and subtly not-your-real-code is a first impression you're choosing to make. A per-customer microVM makes "it worked in sandbox" mean something again.

For the closely related patterns: spinning up a full environment per pull request is covered in /blog/preview-environments-on-microvms, and the database-side argument for giving every tenant a real instance instead of a shared table is in /blog/per-tenant-database-isolation.