Building a load-testing fleet on microVMs

Every team eventually needs to answer "what happens at 20,000 requests per second?" and the first attempt always looks the same: a couple of long-lived EC2 boxes with k6 installed, or — more often now — a Deployment in the shared Kubernetes cluster, because that's where the CI runners already live. Both work. Both also quietly corrupt the number you came to produce, in a way that never surfaces as an error, just as a p99 that's 40ms worse than reality and a graph nobody can reproduce next quarter.

I'm Ajay — I build PandaStack, which runs Firecracker microVMs as a service, so I have an obvious stake here. This post is about the pattern rather than the product: one ephemeral microVM per load-generator worker, created for the test and destroyed the moment it's done. Why load generators are a terrible container neighbour, what per-VM network namespaces buy you in accounting terms, why boot latency decides whether this is practical, the fan-out/fan-in shape in code, and the cases where a plain VM fleet is fine and you should just do that.

A load generator is the worst neighbour on the host

Most workloads are polite. A load generator is designed — explicitly, as its entire purpose — to consume every resource it can reach until something breaks. Putting one on a shared host is like inviting someone to a dinner party whose stated goal is to find out how many chairs the floor supports.

Three resources go first, and none of them are the ones your dashboards are watching.

The host NIC and its queues

Bandwidth is the obvious one, but packets-per-second usually breaks first. A generator firing small requests is a PPS machine: tiny packets, huge counts, softirq processing pinned to whichever CPUs the NIC's receive queues hash to. Once ksoftirqd saturates those cores, every other container on the host pays a scheduling tax on its own network I/O — including containers whose owners have no idea a load test is running. Cloud instances also enforce throughput ceilings at the hypervisor, so you can hit a limit that appears nowhere in your guest metrics and looks, from inside, exactly like the target getting slower.

Conntrack and ephemeral ports

This one bites hardest and gets diagnosed last. A container on a bridged or NAT'd network shares the host's connection-tracking table and, depending on setup, its ephemeral port range. Both are finite. Open enough short-lived outbound connections fast enough and you fill `nf_conntrack`, at which point the kernel drops packets and logs `nf_conntrack: table full, dropping packet` — if you happen to be reading dmesg on the node, which you aren't, because you're watching a Grafana dashboard.

The ephemeral port range is roughly 28,000 ports by default, and every closed connection holds its four-tuple in `TIME_WAIT` for a minute afterwards. At a few thousand new connections per second with keep-alive disabled — exactly what a naive load script does — you exhaust the range in under half a minute. The generator starts failing to allocate sockets, reports connection errors, and you file a bug against the service under test. The service is fine. Your client ran out of ports, and in a shared namespace so did everything else on the node.

CPU, and the timers that measure the thing

Load generators measure time: when a request left, when the response arrived, difference becomes your latency distribution. That's only as good as the generator's ability to get scheduled promptly. When the host is CPU-contended, the measuring thread waits in the run queue and that wait is recorded as target latency — coordinated omission's less famous cousin. Not "we stopped sending while the server was slow" but "we noticed the response late because the CPU was busy." Either way the number is wrong in the direction that makes your service look bad.

The real cost: you invalidate the measurement

This is the argument I actually care about, and it's not a security argument. Noisy-neighbour effects in normal workloads are an annoyance — some tenant's p99 gets a bit worse. Noisy-neighbour effects in a load test destroy the artifact you spent the afternoon producing.

A load test is an experiment. If the instrument perturbs the system it measures — and it also perturbs itself — you don't have a slow result. You have no result.

Concretely, the shared-namespace load test gives you a number that depends on which CI jobs landed on the same node, whether another team was also testing, how full the conntrack table already was, and which NIC queues your pods hashed onto. None of that is in the test report. So when the number moves 15% next month you can't tell whether the service regressed or whether Tuesday was busier than Thursday. You will spend two days finding out. I have spent those two days.

The reverse direction is funnier and more expensive. A generator co-located with the service under test doesn't just measure it, it competes with it, so your capacity number is pessimistic by however much CPU the client stole. And if the test environment shares an egress path with production, congratulations: you're now load-testing your own NAT gateway, your own DNS resolver, and — my favourite genre of incident — your payment provider's sandbox endpoint, right up until they rate-limit the whole account and finance asks why checkout is down. "We DDoSed ourselves from a shared CI runner" is nearly always a load generator with no boundary around it.

One microVM per worker

The fix is structural: give each worker its own machine, and make that machine cheap enough to create that you do it per test run rather than per quarter. On PandaStack each sandbox is a real Firecracker microVM — own guest kernel, own network namespace, own TAP device, own filesystem — which maps onto the failure modes above almost one-for-one.

• Own kernel, own conntrack, own ephemeral port range — the ~28k-port ceiling is per worker, not per node, so adding workers adds connection capacity instead of subdividing a fixed pool.
• Own network namespace and TAP — per-worker throughput and drop accounting is a property of the infrastructure, not something you infer from application logs.
• Own vCPU allocation — the measuring thread isn't competing with a neighbour's build job for the run queue, so the timestamps mean what they say.
• Blast radius of one — a worker that OOMs mid-ramp takes down one worker; the other twenty-three keep generating load and fan-in records that one slice came back short.
• Dies on a timer — `ttl_seconds` means a forgotten fleet reaps itself, rather than being discovered six weeks later via the bill.

On per-VM addressing and source diversity

Each agent host pre-allocates 16,384 /30 subnets — a NATID pool, where each slot is a fully built network namespace with a veth pair and a TAP device waiting for a VM. A worker drops into a slot and gets its own address on its own subnet. That's what makes per-worker accounting clean: traffic is separable in the kernel, not merged into one bridge and untangled afterwards by parsing logs.

Be precise about what this does and doesn't give you. Distinct in-host addressing is not automatically distinct public source addressing — what your target sees depends on how egress is NAT'd or routed, which you should verify from the target's side, not from inside the VM. It matters whenever the system shards, rate-limits, or load-balances on client IP: hammer a per-IP-limited endpoint from one source and you've measured the rate limiter, not the service. And this is for systems you own or are authorised to test — source diversity to evade someone else's protections isn't load testing, it's the other thing.

Why boot latency matters more than you'd think

The reason teams keep long-lived load-generator boxes is provisioning friction. If standing up 24 workers takes five minutes of instance boot plus cloud-init plus a package install, a 10-minute test costs 15 minutes — and a failed test costs 15 minutes again. So somebody keeps the fleet warm, and now you pay for idle machines whose k6 version drifts three minors behind the one on your laptop.

Snapshot-restore removes the friction rather than amortising it. Every create restores a baked snapshot instead of cold-booting: p50 179ms, p99 around 203ms, with the underlying restore step about 49ms. The first cold boot of a template is roughly 3 seconds, once, and then everything after that rides the snapshot path. Twenty-four workers created concurrently land in well under a second of wall clock. At that point provisioning stops being a thing you plan around and becomes a line in a loop.

The other half is what's in the image. Bake the generator binary, the scenario, and the fixtures into a snapshot once, and every worker starts warm instead of running `apt-get` twenty-four times in parallel against a package mirror that is now, briefly, also being load-tested. To warm a machine interactively and branch it instead, `snapshot()` then `fork()` — same-host forks land in 400–750ms, cross-host 1.2–3.5s since the memory image has to move first.

Fan-out, fan-in: the code

The shape: create N sandboxes concurrently, write the scenario in, run the generator with a per-worker id and VU budget, read back a machine-readable summary from each, aggregate. Here's the k6 side. The detail that matters is `handleSummary` — it writes a JSON file inside the guest that the host reads back over the filesystem API, so you never parse a human-formatted summary out of stdout.

// loadtest.js -- one worker's slice of the scenario.
// Each microVM runs this with its own worker id and VU budget, so the
// fan-in step can attribute every number back to a specific worker.
import http from 'k6/http';
import { check, sleep } from 'k6';
import { Trend } from 'k6/metrics';

const WORKER = __ENV.WORKER_ID || 'w0';
const TARGET = __ENV.TARGET_URL;
const VUS = Number(__ENV.VUS || 50);

const ttfb = new Trend('worker_ttfb', true);

export const options = {
  // Ramp shape is PER WORKER. Offered load = this x N workers, so keep
  // the arithmetic somewhere a human can see it before you run 24 of these.
  scenarios: {
    ramp: {
      executor: 'ramping-vus',
      startVUs: 0,
      stages: [
        { duration: '30s', target: VUS },
        { duration: '4m',  target: VUS },
        { duration: '30s', target: 0 },
      ],
      gracefulRampDown: '10s',
    },
  },
  // Fail loudly rather than producing a pretty green run nobody reads.
  thresholds: {
    http_req_failed:   ['rate<0.01'],
    http_req_duration: ['p(95)<800'],
  },
  // Don't spend the worker's CPU decoding bodies you're going to discard.
  discardResponseBodies: true,
};

export default function () {
  const res = http.get(TARGET + '/api/products?page=' + (__ITER % 50), {
    tags: { worker: WORKER },
  });
  ttfb.add(res.timings.waiting);
  check(res, { 'status is 200': (r) => r.status === 200 });
  sleep(0.2);
}

// k6 writes whatever this returns. One machine-readable artifact per
// worker, at a known path, for the host to collect after the run.
export function handleSummary(data) {
  return {
    '/work/summary.json': JSON.stringify({ worker: WORKER, vus: VUS, data }),
  };
}

And the host side. A thread pool is more than enough concurrency — creates are sub-200ms, so wall-clock time is the test duration plus a rounding error. Set `PANDASTACK_API_KEY` in your environment and the SDK picks it up.

import json
import uuid
from concurrent.futures import ThreadPoolExecutor, as_completed
from pandastack import Sandbox

TARGET = "https://staging.acme.example"   # a system you own / are authorised to test
WORKERS = 24
VUS_PER_WORKER = 50                        # => 1,200 VUs offered in total
RUN_ID = uuid.uuid4().hex[:8]

with open("loadtest.js", "rb") as f:
    SCENARIO = f.read()


def run_worker(i: int) -> dict:
    name = f"w{i:02d}"
    # Fresh microVM per worker: own kernel, own conntrack table, own
    # ephemeral port range, own netns. ttl_seconds is the backstop --
    # if this process dies mid-run, the fleet still reaps itself.
    with Sandbox.create(
        template="base",
        ttl_seconds=1800,
        metadata={"role": "loadgen", "run": RUN_ID, "worker": name},
    ) as sbx:
        sbx.filesystem.write("/work/loadtest.js", SCENARIO)

        res = sbx.exec(
            "cd /work && "
            f"WORKER_ID={name} TARGET_URL={TARGET} VUS={VUS_PER_WORKER} "
            "k6 run --quiet loadtest.js",
            timeout_seconds=600,
        )

        # Always try to collect the summary: k6 exits non-zero when a
        # threshold is breached, and THAT run is the interesting one.
        try:
            summary = json.loads(sbx.filesystem.read("/work/summary.json"))
        except Exception as exc:
            return {"worker": name, "error": f"no summary: {exc}",
                    "stderr": res.stderr[-2000:]}

        # Client-side health check. If the GENERATOR was saturated, the
        # latency numbers below are fiction -- see "the client must not
        # be the bottleneck".
        health = sbx.exec(
            "ss -s | head -3; "
            "cat /proc/sys/net/netfilter/nf_conntrack_count 2>/dev/null || true",
            timeout_seconds=15,
        )

        return {
            "worker": name,
            "exit_code": res.exit_code,          # non-zero => threshold breach
            "wall_ms": res.duration_ms,
            "client_health": health.stdout.strip(),
            "summary": summary,
        }
    # sandbox destroyed on block exit -- nothing left running, nothing left billing


# Fan out. 24 creates at ~179ms p50 means the fleet is live in well under a second.
results = []
with ThreadPoolExecutor(max_workers=WORKERS) as pool:
    futures = [pool.submit(run_worker, i) for i in range(WORKERS)]
    for fut in as_completed(futures):
        r = fut.result()
        print(f"{r['worker']}: exit={r.get('exit_code')} {r.get('error', '')}")
        results.append(r)

# Fan in. Aggregate ACROSS workers -- never average the per-worker p95s,
# that's not how percentiles compose. Sum the counters, and treat the
# per-worker tails as a distribution of tails.
ok = [r for r in results if "summary" in r]
total_reqs = sum(r["summary"]["data"]["metrics"]["http_reqs"]["values"]["count"]
                 for r in ok)
p95s = sorted(r["summary"]["data"]["metrics"]["http_req_duration"]["values"]["p(95)"]
              for r in ok)

print(f"run {RUN_ID}: {len(ok)}/{WORKERS} workers reported")
print(f"total requests: {total_reqs:,}")
print(f"per-worker p95 spread: {p95s[0]:.1f}ms .. {p95s[-1]:.1f}ms")

with open(f"run-{RUN_ID}.json", "w") as f:
    json.dump(results, f, indent=2)

Two details worth stealing. Collect the summary even when the exec exits non-zero — k6 returns a failure code on a breached threshold, which is precisely the run you want data from. And that per-worker p95 spread line is a cheap sanity check: if one worker's p95 is triple the others', you have a sick worker, not a sick service, and a fleet average would have hidden it.

Three ways to run the fleet

Honest comparison across the dimensions that actually decide this. Cloud and Kubernetes behaviour varies by configuration, so verify the specifics for your own setup.

• Provisioning time — Long-lived VM fleet: zero, it's always on; you paid in advance. Shared k8s namespace: seconds to schedule, if capacity exists. Ephemeral microVM fleet: p50 179ms per create via snapshot-restore, run concurrently, so 24 workers are live in under a second.
• Conntrack and ephemeral ports — Long-lived VM fleet: per-instance, reused by every test that box ever runs. Shared k8s namespace: typically shared with the node, so one aggressive test exhausts it for unrelated pods. Ephemeral microVM fleet: per-VM guest kernel — own table, own ~28k port range.
• Measurement fidelity — Long-lived VM fleet: good, if nothing else is on the box and the last run's TIME_WAIT sockets have drained. Shared k8s namespace: the weakest link — neighbour CPU contention lands straight in your histogram. Ephemeral microVM fleet: own vCPUs and own network stack, fresh every run.
• Blast radius of a bad test — Long-lived VM fleet: your instances, though shared egress is still a risk. Shared k8s namespace: the whole node, plus quite possibly cluster egress, CoreDNS, and everyone's CI. Ephemeral microVM fleet: one VM.
• Reproducibility — Long-lived VM fleet: drifts, accumulating package versions and tunables somebody set during an incident. Shared k8s namespace: depends on who else was scheduled there that day, recorded nowhere. Ephemeral microVM fleet: every worker restores the same baked snapshot, so build and scenario are identical by construction.
• Cost shape — Long-lived VM fleet: 24/7 spend for machines used hours a month. Shared k8s namespace: looks free because it's someone else's budget line. Ephemeral microVM fleet: you pay for the test window, and `ttl_seconds` closes it even if your script crashes.

Paying for the test window, not the quarter

The economics are about duty cycle, not unit price. A load-generator fleet runs in bursts: a few runs before a release, a monthly soak, a panicked capacity check the day someone tweets about you. Sizing persistent infrastructure for peak offered load and leaving it idle 99% of the time is the exact problem that pushed everyone to on-demand compute — we just kept exempting test infrastructure because provisioning it was annoying. When creation is sub-second, the fleet is sized per test rather than per quarter. And because `ttl_seconds` is enforced by the platform rather than your script's `finally` block, the fleet still disappears when a CI job is cancelled mid-run — the most common way load-generator instances become permanent.

Honest caveats: the client is also a system under test

None of this helps if you don't measure the generator. Three caveats I'd want a reviewer to hold me to.

A microVM's vCPU allocation caps its requests per second

There's no free lunch: a worker with 2 vCPUs generates what 2 vCPUs of that generator can generate. Isolation doesn't create throughput, it stops other people stealing yours. So you scale out — more workers, not bigger ambitions per worker — and you need to know your per-worker ceiling before trusting a run. One wrinkle specific to snapshot-restore: Firecracker can't change vCPU or RAM at restore time, so a restored worker's size comes from the baked snapshot, not a per-request parameter. A different worker shape means a different baked template, not a flag.

Measure the generator, every run

Before you believe any latency number, check the client wasn't the bottleneck. Was a worker CPU-saturated? Did socket errors appear? Did conntrack fill? Did the achieved request rate match the offered rate, or did the generator quietly fall behind schedule? Most tools will tell you if you ask — k6 exposes dropped iterations, and any generator worth using distinguishes "the server was slow" from "I couldn't send on time." That distinction is coordinated omission, and ignoring it is how you report a p99 off by an order of magnitude in the flattering direction. Ramp the fleet until throughput stops scaling linearly with worker count: where it flattens is your client ceiling.

The network path is part of the experiment

Where your workers run relative to the target changes the answer. Same region, cross region, over the public internet, through your own NAT gateway — each adds latency and its own ceilings. A test from another region measures a different system than one from next door; neither is wrong, but the report has to say which you did. And point the fleet at something you own: third-party APIs in your critical path, payment sandboxes especially, will notice.

When a plain VM fleet is completely fine

The useful version of this post includes the part where you don't need it. If you run one load test a month from two dedicated instances that nothing else shares, and the numbers have been stable for a year — do that. It works, you understand it, and swapping it buys you nothing. Same for long soaks measured in days: the provisioning cost you'd optimise away is a rounding error against a 72-hour run.

The ephemeral fleet earns its keep at a specific inflection: when tests run often enough that provisioning friction is why people skip them, when you need enough workers that per-node conntrack and port limits are a real ceiling, when run-to-run reproducibility matters, or when the generators share infrastructure with the thing whose latency you're measuring. That last one is the killer — the shared-namespace load test isn't just impolite to your neighbours, it's the reason your p99 graph has an unexplained step change nobody can account for.

So: give every worker its own kernel, its own port range, and its own vCPUs; bake the scenario into the image so all of them run the same thing; create them in a loop and let them die on a timer. Then measure the generator too, because the client is always the first system to fall over and always the last one anybody suspects.