Your API Mock Is a Server Running Customer Code

Every company that sells an API eventually ships a fake one too. Your customers need somewhere to integrate before they're allowed near production, so you stand up a sandbox: same routes, same schemas, plausible responses, no real money moves. Internally the same thing happens under a different name — service virtualization — because the payments team can't run integration tests against a mainframe that bills per transaction and is only reachable on Tuesdays.

I'm Ajay; I build PandaStack, a Firecracker microVM platform, and I've watched a specific failure play out enough times to write it down. It starts with the most reasonable feature request in the world — "can we make the sandbox return OUR test data?" — and ends, eighteen months later, with a shared mock server that evaluates customer-authored matchers, templates, and callback scripts, in one process, on one host, with one filesystem. Nobody ever decided to host untrusted code. They decided to support configuration, and configuration grew a Turing machine.

The escalation: fixtures, then templates, then a scripting engine

Nobody builds an RCE sandbox on purpose. They build it one ticket at a time, and each ticket is individually correct.

1. Static fixtures. GET /v1/orders/123 returns a canned JSON blob. Delightful. Ships in an afternoon. Survives about two weeks.
2. Parameterised fixtures. "The ID in the response should match the ID in the URL, obviously." Now you need path capture and interpolation, so you add a template language.
3. Helpers. "Our tests assert created_at is within the last hour." So the template language grows `now`, then date math, then `randomInt`, then a fake-data library with 300 locales. Your config file has expressions in it now.
4. Conditionals and loops. "Return 20 line items if the request asks for 20." Handlebars-style block helpers, iteration, comparison operators. Your config file has control flow now.
5. State. "After a POST to /payments, the GET should say captured." You add a scenario state machine, then a key-value store customers can read and write from a template.
6. Scripts. Someone needs a checksum, or a JWT signed with their test key, or a response that depends on three previous calls. You expose a callback hook: a snippet of JavaScript, Lua, Groovy, or a Velocity/Mustache extension. Ticket closed. Sprint velocity excellent.

Step six is where the product changes category. Up to step five you were interpreting a data structure; at step six you're executing a program written by someone whose entire relationship with you is a signup form and a credit card that may or may not be theirs. The mainstream tools all live somewhere on this ladder — WireMock's response templating with Handlebars helpers and JVM extensions, MockServer's callbacks and templated responses, mountebank's `inject` behavior which is literally a JavaScript function you hand the server, Hoverfly's middleware which shells out to a script, Mockoon and Beeceptor with their own helper sets, Prism generating from OpenAPI schemas. That's not a criticism of any of them; several are excellent and I use them. It's a statement about what the feature IS. Check each project's current docs before you assume where its sandbox boundary sits — these engines evolve, and "we disabled scripting in v3" is exactly the kind of thing that changes between minor versions.

{
  "id": "orders-get",
  "tenant": "acme-corp",
  "request": {
    "method": "GET",
    "urlPattern": "^/v1/orders/[0-9a-f-]{36}$",
    "headers": { "Authorization": { "matches": "^Bearer .+$" } }
  },
  "response": {
    "status": 200,
    "headers": { "Content-Type": "application/json" },
    "bodyTemplate": {
      "id": "{{request.pathSegments.[2]}}",
      "created_at": "{{now offset='-3 days' format='iso8601'}}",
      "total_cents": "{{randomInt lower=100 upper=99999}}",
      "status": "{{#if (state.get 'captured')}}captured{{else}}pending{{/if}}",
      "lines": "{{#repeat (request.query.limit)}}{{> line_item}}{{/repeat}}"
    }
  },

  "__comment": "Everything above is DATA. Everything below is a PROGRAM.",

  "hooks": {
    "afterMatch": "js:(ctx) => { ctx.state.set('hits', (ctx.state.get('hits') || 0) + 1); if (ctx.request.query.fail === '1') ctx.response.status = 402; }"
  }
}

Even the matchers are code (this one surprises people)

Suppose you hold the line at step four: no scripts, no callbacks, just declarative matchers and templates. You are still running customer-supplied code, because a regular expression is a program and most regex engines will happily execute a pathological one until the heat death of the universe. A matcher like `^(a+)+$` against a 40-character non-matching input is the classic catastrophic-backtracking case; it pins a core, and on a shared mock server with a thread-per-request or event-loop model, one tenant's bad regex is every tenant's outage.

The same is true of the template side. Nested `#repeat` blocks generating a 900MB response body is not an attack, it's a typo — someone bound the loop count to a query parameter and a customer's load test passed `limit=100000`. In a shared process, that typo is an OOM kill for everyone. This is why "we don't allow scripting" is a weaker safety claim than it sounds: you don't have to allow scripting to allow unbounded CPU and unbounded allocation. You just have to allow expressiveness, which is the entire point of the feature.

Stateful mocks: a database with worse ergonomics

The single most-requested mock feature after templating is memory. A customer POSTs an order and wants the subsequent GET to return it. They want a payment to move from `pending` to `captured` after a webhook. They want to test their idempotency keys, which by definition requires the mock to remember that it has seen a key before. A stateless mock can't test any of the integration logic that actually breaks in production.

So you add state, and the moment you do, three properties you were getting for free evaporate. First, isolation between tenants stops being automatic — you now need a partition key threaded through every read and write, and the interesting bug is not "the key is missing," it's "the key is present in nine of ten code paths." Second, isolation between a tenant's own test runs disappears: two CI jobs from the same customer hitting the same mock will interleave writes and produce flakes that look like race conditions in their code. Third, cleanup becomes your problem forever — TTLs, reset endpoints, a nightly truncate job, and the incident where the truncate job's tenant filter was wrong.

The honest way to describe a stateful multi-tenant mock is: you have accidentally built a small multi-tenant database, with a query language you invented on a Thursday, and none of the twenty years of hardening that a real database has. The alternative isn't better isolation code. It's not sharing the process.

Record-and-replay is a PII pipeline in a trench coat

The most powerful service-virtualization feature is recording. Point a proxy at the real dependency, run traffic through it, capture request/response pairs, replay them forever. It's the only way to virtualize a service whose behavior nobody can fully describe — which is most legacy services, because the person who could describe it retired in 2019.

It is also, mechanically, a system whose job is to write production traffic to disk. That capture contains Authorization headers with live bearer tokens, session cookies, API keys in query strings that someone swore were removed in 2021, full names and addresses in response bodies, card BINs and last-four, national ID numbers if you operate anywhere with KYC, and free-text fields where a support agent pasted something they shouldn't have. Recordings are usually stored as HAR or a similar JSON-ish blob, in a directory, next to the other tenants' recordings.

The standard mitigation is scrubbing rules, and scrubbing rules are regexes, and regexes miss. They miss the token in a nested JSON field nobody enumerated. They miss the base64 body. They miss the new endpoint shipped last week. Scrubbing is worth doing and it is not a boundary — it's a filter with a false-negative rate, deployed against a data set you can't fully enumerate. Meanwhile the retention question stacks on top: under GDPR-style regimes, a recording of a real customer transaction is personal data, and "we keep every capture forever because someone might need to replay it" is a sentence your DPO will want to discuss. If recordings live in one shared mock server's filesystem, the blast radius of one path-traversal bug in the admin API is every tenant's captured production traffic.

Deterministic latency and fault injection

The other half of service virtualization — the half that justifies the whole exercise to engineering leadership — is misbehaving on purpose. Real dependencies fail, and the only way to know whether your retry logic, circuit breaker, and timeout budget actually work is to make something fail on command. A good virtual service injects latency distributions, 5xx bursts, connection resets, truncated responses, and the genuinely nastiest one: the hang. Not a slow response, no response, socket open, until your client's timeout fires — or doesn't, because someone left it at the library default of "never."

# fault-profile.yaml -- per-tenant, seeded, and therefore reproducible.
# Random chaos is unfalsifiable: a test that fails 3% of the time teaches
# nobody anything. Seed it and the same run produces the same disasters.
seed: 4172

rules:
  - match: { method: POST, path: /v1/payments }
    latency:
      distribution: lognormal      # not a flat sleep -- real p99s have tails
      p50_ms: 120
      p99_ms: 2400
    faults:
      - { every_nth: 7,  status: 503, body: '{"error":"upstream_unavailable"}' }
      - { every_nth: 23, action: reset_connection }   # RST mid-handshake
      - { every_nth: 97, action: hang, hold_ms: 60000 }

  - match: { method: GET, path: /v1/orders/* }
    faults:
      # Truncated body: content-length lies. Finds every JSON parser that
      # treats a partial read as an empty object and carries on.
      - { every_nth: 31, action: truncate_body, after_bytes: 512 }

  # The 'everything is on fire' profile, on demand, per tenant.
  - match: { header: { X-Chaos-Mode: full } }
    latency: { distribution: fixed, p50_ms: 30000 }
    faults:
      - { every_nth: 2, status: 500 }

Now look at that config from the mock server's point of view. A `hang` holds a connection for sixty seconds. A `fixed: 30000` latency profile holds one for thirty. If your mock is one shared server, those held connections come out of a shared pool — a shared thread pool, a shared event loop's socket budget, a shared file-descriptor limit. Tenant A testing their timeout handling is, from tenant B's perspective, an outage of your sandbox. And you cannot even fix it by capping concurrency per tenant, because holding connections open IS the feature they're paying for.

One big mock server vs. one microVM per tenant

Here's the same workload in two topologies. The shared column isn't a strawman — it's what almost everyone runs, and for a while it's correct. It stops being correct at the exact moment stub definitions become expressive enough to be useful.

• Isolation boundary — Shared mock server: tenants are rows in a table and a partition key threaded through your code; a bug in one code path crosses tenants. Per-tenant microVM: separate guest kernel behind KVM, so crossing tenants means breaking the hypervisor, not missing a WHERE clause.
• Customer-authored scripts — Shared mock server: an eval sandbox inside your process, which is a category of security control with a long and unhappy CVE history. Per-tenant microVM: the script runs as a normal process in a VM that holds nothing but that tenant's own mock; full RCE inside it is an unremarkable Tuesday.
• CPU-burning matchers — Shared mock server: one catastrophic regex or unbounded template loop pins a core or OOMs the process for everyone. Per-tenant microVM: it burns that tenant's own vCPU allocation inside their own VM, and the blast radius is a support ticket from the person who wrote it.
• Stateful mocks — Shared mock server: you build a multi-tenant key-value store, plus TTLs, reset endpoints, and a cleanup job with a tenant filter you'd better get right. Per-tenant microVM: state is a file on that VM's own disk; deletion is destroying the VM.
• Recorded traffic and PII — Shared mock server: every tenant's captures in one filesystem, so one path-traversal bug in the admin API exposes all of them. Per-tenant microVM: captures live only in the tenant's own VM and its own storage prefix; there is no shared directory to traverse.
• Hang and latency injection — Shared mock server: held connections consume shared threads, sockets, and file descriptors; tenant A's timeout test degrades tenant B. Per-tenant microVM: held sockets are that VM's own file descriptors, and its slow clock is nobody else's problem.
• Idle cost — Shared mock server: one always-on process sized for peak, paid for at 3am when nobody is integrating. Per-tenant microVM: scale to zero when idle, restored on demand from a snapshot at p50 179ms.
• Blast radius of a bad deploy — Shared mock server: one process, one config reload, everyone's stubs. Per-tenant microVM: roll the template forward tenant by tenant; a broken stub set affects exactly the tenant who wrote it.

What per-tenant provisioning actually looks like

The reason nobody ran a VM per tenant historically is that VMs were expensive and slow to start, so "one per customer" meant a fleet of idle instances and a capacity planning spreadsheet. Snapshot-restore is what changes the arithmetic. On PandaStack a sandbox isn't cold-booted; it's created by restoring a pre-baked snapshot of an already-running machine — p50 179ms, p99 203ms, with the restore step itself around 49ms. The first-ever boot of a template takes about 3 seconds, once, and every instance after that is a restore. Bake your mock engine, its dependencies, and its warm JIT into the snapshot and a tenant's mock environment costs roughly one round trip to exist.

import { Sandbox } from "@pandastack/sdk";

type MockEnv = { tenantId: string; sandboxId: string; baseUrl: string };

// One microVM per tenant mock environment. The stub definitions, the state
// store, the recordings, and the fault profile all live inside it -- so
// "delete this customer's data" is one API call, not a migration.
export async function provisionMockEnv(
  tenantId: string,
  stubs: unknown,          // customer-authored: matchers, templates, hooks
  faultProfile: string,    // customer-authored: the YAML above
): Promise<MockEnv> {
  const sbx = await Sandbox.create({
    template: "mock-engine",       // baked snapshot: engine + deps, pre-warmed
    ttlSeconds: 86_400,            // backstop; the idle reaper usually wins first
    metadata: { tenant: tenantId, kind: "service-virtualization" },
  });

  // Config crosses as DATA. We never eval a customer's stub file on our side --
  // the whole point is that it is only ever executed inside their VM.
  await sbx.filesystem.write("/etc/mock/stubs.json", JSON.stringify(stubs));
  await sbx.filesystem.write("/etc/mock/faults.yaml", faultProfile);

  const boot = await sbx.exec("systemctl restart mock-engine", { timeoutSeconds: 30 });
  if (boot.exitCode !== 0) throw new Error(`mock engine failed: ${boot.stderr}`);

  // Tokenless preview host: <port>-<sandbox-id>.<suffix>. The customer points
  // their integration tests here instead of at your production API.
  return {
    tenantId,
    sandboxId: sbx.id,
    baseUrl: `https://8080-${sbx.id}.sandbox.pandastack.ai`,
  };
}

// Teardown is the whole GDPR deletion story: state, recordings, and any
// process the customer's afterMatch hook decided to leave running.
export async function destroyMockEnv(env: MockEnv) {
  const sbx = await Sandbox.connect(env.sandboxId);
  await sbx.kill();
}

The pattern that surprised me most in practice is branching. Because a running VM can be forked copy-on-write — same-host forks land in 400-750ms, cross-host 1.2-3.5s — "a mock environment per pull request" becomes a reasonable thing to offer. The customer's team gets a fork of their mock, complete with its current state, recordings, and stub set, tears it down when the PR merges, and never touches the shared one. It's the same primitive that makes per-branch database environments work, applied to the fake side of the integration.

from pandastack import Sandbox

# The tenant's long-lived mock env, with recordings and state already in it.
base = Sandbox.connect(tenant_mock_id)

# Fork it per PR: copy-on-write disk, so a 40GB recording corpus doesn't get
# copied 40GB at a time. The branch can be mutated freely -- new stubs, a
# nastier fault profile, a wiped state store -- without touching the original.
branch = base.fork(metadata={"pr": "4711", "purpose": "contract-tests"})
try:
    branch.filesystem.write("/etc/mock/faults.yaml", chaos_profile)
    branch.exec("systemctl restart mock-engine", timeout_seconds=30)

    # CI points at the branch's preview URL and runs the suite against a mock
    # that fails on purpose, deterministically, for this PR only.
    run_contract_suite(f"https://8080-{branch.id}.sandbox.pandastack.ai")
finally:
    branch.kill()   # the PR is merged; the fake payments processor can go

The economics: mocks are idle almost all the time

Here's the thing about a customer's sandbox environment: it is used during onboarding, during a CI run, and during the two weeks somebody is actually building the integration. The rest of the time — nights, weekends, the eleven months after go-live — it serves nothing. A shared mock server hides this by amortising: one always-on process, sized for peak, and the idle cost is a rounding error because there's only one of it. That's a real advantage, and it's why the shared design wins early.

It stops winning when the shared server has to be sized for the worst-behaved tenant. One customer's chaos profile holding a hundred connections for sixty seconds each, one catastrophic regex, one 900MB templated response — and now you're provisioning headroom for everyone's peak simultaneously, on a box you can't autoscale cleanly because it's stateful. The per-tenant model inverts it: each environment is small, each one is idle most of the day, and idle environments can be snapshotted and torn down entirely, then restored on the next request. Sub-200ms restore is what makes wake-on-request tolerable — a first request that pays roughly a fifth of a second, and subsequent ones that pay nothing. You are paying for mock environments that are actually in use, and the density ceiling stops being "how many tenants fit in one process" and becomes host memory (each PandaStack agent host pre-allocates 16,384 /30 subnets, so networking is nowhere near the binding constraint).

Be honest about what the first request costs, though. Restore is fast, but it isn't zero, and if the mock engine reloads a large recording corpus afterwards, that's your latency, not the platform's. A snapshot restores a machine that already finished starting up, so anything you do before the snapshot is free on every restore after it.

When this is overkill

Most mock servers should stay one shared mock server. If your virtual service is internal, the stub definitions are written by your own engineers, and the mock returns static or lightly-templated fixtures with no scripting and no recorded production traffic, then you have a config-driven HTTP server and the isolation story is "it's all one team's code" — which is a completely legitimate isolation story. Running a VM per tenant there buys you operational complexity and nothing else. Same if your sandbox is genuinely read-only and schema-generated: a Prism-style mock derived from an OpenAPI document has no customer-authored logic in it at all, so there's nothing to isolate.

The per-tenant microVM earns its keep at a specific intersection, and it's worth checking whether you're actually at it. You are if: customers (or other teams you don't control) author matchers, templates, or scripts; the mock is stateful, so a partition-key bug is a cross-tenant data bug; you record real traffic, so recordings contain data with a retention policy attached; or fault injection means one tenant can deliberately exhaust a shared resource. Two or more of those and the shared design is running on the honor system. All four and you are one bad regex from a status page.

And the costs are real. You take on VM lifecycle: provisioning, idle reaping, snapshot management, and a fleet view of environments where you used to have one process and one dashboard. Debugging is a step removed — you can't tail one log file for all tenants any more, so you'll want log shipping out of each guest from day one. Per-tenant config drift becomes possible in a way it wasn't when everyone shared a binary, which means you need a rollout story for engine upgrades rather than a deploy. Those are genuine trade-offs. The reason I still land on per-tenant is that the alternative asks you to build a correct multi-tenant script sandbox, a correct multi-tenant state store, and a correct multi-tenant PII store, all inside one process, and to keep all three correct forever — whereas a hypervisor boundary is a thing you configure once and it holds while you're asleep.