Synthetic Monitoring Is a Hostile Workload: One microVM Per Check

Synthetic monitoring has the most harmless-sounding job description in infrastructure: every minute, load a page and tell me whether it worked. Nobody puts it on the threat model. It gets a box labelled "checks" on the architecture diagram, usually drawn in a soothing colour, usually next to the word "just".

Now describe the same system without the marketing: it runs a full Chromium against URLs that arbitrary customers typed into a form, it holds those customers' real login credentials so it can click through an authenticated checkout flow, it executes check scripts those same customers wrote, and it does all of that in a worker pool shared by every tenant you have. That is a scraper fleet with a nicer dashboard, and it is one of the more hostile workloads in a typical SaaS company — it's just hostile in a way that never shows up until it does.

I'm Ajay; I built PandaStack, a Firecracker microVM platform, and monitoring teams keep bringing me the same conversation: the checks work fine, the isolation is "a fresh browser context per check," and someone in the room has started to feel uneasy about that sentence. This post is why the unease is correct, what the per-check microVM shape looks like, how snapshot-restore makes it affordable, and — the part people forget — what one-VM-per-check does to your measurements, which is not automatically good.

What a synthetic check actually is

Strip a check down and it has four properties, each of which would individually make a security engineer ask questions. It renders untrusted content: the target URL is whatever the customer configured, and whatever that page loads is whatever it loads — third-party scripts, ad tech, an injected payload from a supply-chain compromise the customer doesn't know about yet. It executes untrusted code: not just the page's JavaScript but often the check script itself, because "write your own Playwright flow" is table stakes for anything past a plain HTTP ping.

It holds real secrets: authenticated flows need a real username and password, sometimes a TOTP seed, sometimes a session token minted against production. And it produces numbers people trust — TTFB, LCP, full-flow duration, pass or fail — which wake people at 3am and land in SLA reports, so anything that quietly biases them is a correctness bug wearing a performance costume.

Why the shared browser pool leaks

Almost every homegrown monitoring fleet lands on the same sensible architecture: a pool of long-lived workers, each running a persistent browser, checks dispatched onto them, a fresh incognito context per check. Browsers are expensive to start and contexts are cheap, so you amortise. That instinct is right, and it is also where the leaks live — a browser context is an isolation boundary for cookies and a good deal less than that for everything else.

• Renderer escapes are a normal Tuesday. Chromium's own sandbox is excellent and it is also the most attacked piece of software on earth, with a steady flow of in-the-wild renderer and GPU-process exploits. A page that escapes the renderer is on the worker; if the worker is a container, the next stop is a shared host kernel that every one of your tenants' checks is also sitting on.
• Credential residue outlives the context you closed. Cookies go with the context, sure. Saved passwords, HSTS pins, certificate exceptions, downloaded files in /tmp, service worker registrations, an IndexedDB write, a crash dump containing heap memory from the authenticated flow you just ran — those live in the browser's profile and cache directories, on disk, on that worker, until something deliberately removes them.
• Network state is shared and sticky. DNS caches, keep-alive connection pools, TLS session tickets, proxy auth state: a check can be affected by — and can observe the timing of — what the previous tenant did to the resolver and connection pool on that worker.
• The cache is a measurement channel and a correctness one. Shared HTTP cache and shared service workers mean check N can be served an asset that check N-1 pulled, which makes tenant A's latency number a function of tenant B's traffic. That is a data leak and a metric corruption in the same bug.
• Runaway scripts are load-bearing chaos. A customer's Playwright script with a busy loop, a page with a memory-leaking animation, a Chromium that wedges instead of exiting: the process eats the worker, and every check assigned to that worker for the next few minutes fails or times out. You now have a monitoring outage that looks exactly like a customer outage.
• Cleanup is a script you wrote once. "We clear the profile dir between runs" holds until a browser release starts persisting state somewhere you didn't enumerate — and that list is long, vendor-specific, and grows every version.

The standard mitigation is one container per check rather than one context, which genuinely helps: it kills the profile residue and the process wedging in one move. But a container is a polite suggestion to the kernel — namespaces and cgroups over one shared host kernel, holding only as long as that kernel has no reachable bug in the syscalls a full Chromium needs, which is most of them. And browser containers get weakened in practice for the same reasons build containers do: someone adds `--no-sandbox` because seccomp fought Chromium, mounts a shared font and cache volume for speed, or bumps `/dev/shm` and hands over more than they meant to.

The shape: one microVM per check run

The alternative is boring, which is how you know it's right. Every check run gets its own Firecracker microVM: its own guest kernel, its own memory, its own virtual disk, its own network namespace, separated from the host by hardware virtualization. It boots from a snapshot with the browser already installed and warm, runs exactly one check, hands back a result as data, and is destroyed. The next check run for that tenant starts from the same clean baked snapshot, as does the next check run for a completely different tenant.

Map that onto the leak list and it goes quiet. A renderer escape lands in a guest kernel that exists for the next forty seconds and belongs to no one else; going further means breaking the hypervisor, a categorically harder problem than the container escapes that ship every year. Credential residue has nowhere to persist — the disk that held the profile dies with the VM. DNS and connection state start empty. Cache state is whatever you deliberately baked into the snapshot. A runaway script burns its own VM and gets reaped by a TTL, taking down exactly one check instead of a worker's queue.

The historical objection is cost: a VM per check sounds like trading a 200ms context creation for a 30-second boot, times 5,000 checks, times every minute, forever. That is the thing snapshot-restore removes. On PandaStack a sandbox isn't cold-booted — it's created by restoring a pre-baked snapshot on demand, at p50 179ms and p99 203ms, with the restore step itself around 49ms. Only the first-ever boot of a template costs about 3 seconds, once. Against a browser check that spends three to ten seconds doing real navigation and waiting on the network, a sub-200ms create is inside the noise of your target site's TTFB variance.

What it looks like in code

Here's the runner. It creates a VM per check run, writes the check script into the guest, passes the credentials as environment for that one process, runs Playwright under a hard wall-clock cap, and pulls back a structured result plus an artifact bundle. Note what it does not do: reuse anything, trust anything the guest produced as code, or leave the VM alive if our own process dies.

import { Sandbox } from "@pandastack/sdk";
import { readFileSync } from "node:fs";

type CheckSpec = {
  id: string;
  url: string;
  region: string;
  scriptPath: string;   // the Playwright flow, ours or the customer's
  budgetMs: number;     // how long the CHECK may take, excluding setup
};

type Creds = { user: string; pass: string };

export async function runCheck(spec: CheckSpec, creds?: Creds) {
  const dispatchedAt = Date.now();

  // One VM per run. It has never seen another tenant's cookies, and it
  // will not exist in two minutes whether or not our code behaves.
  const sbx = await Sandbox.create({
    template: "browser",
    ttlSeconds: 120,                 // backstop: the reaper wins if we crash
    metadata: { check: spec.id, region: spec.region, kind: "synthetic" },
  });

  const readyAt = Date.now();        // create cost -- reported, never billed
                                     // to the customer's latency number
  try {
    await sbx.filesystem.write(
      "/work/check.mjs",
      readFileSync(spec.scriptPath, "utf8"),
    );

    // Credentials cross the boundary as process env for ONE run. There is
    // no next run on this machine to inherit them, and no profile dir that
    // outlives the process to leave them in.
    const run = await sbx.exec("node /work/check.mjs", {
      env: {
        TARGET_URL: spec.url,
        BUDGET_MS: String(spec.budgetMs),
        CHECK_USER: creds?.user ?? "",
        CHECK_PASS: creds?.pass ?? "",
      },
      timeoutSeconds: Math.ceil(spec.budgetMs / 1000) + 15,
    });

    // The guest writes DATA. We parse it; we never execute it.
    const raw = await sbx.filesystem.read("/work/result.json");
    const result = JSON.parse(new TextDecoder().decode(raw));

    return {
      ...result,
      setup_ms: readyAt - dispatchedAt,
      total_ms: Date.now() - dispatchedAt,
      runner_exit: run.exitCode,
      stderr_tail: run.stderr.slice(-2000),
    };
  } catch (err) {
    // Distinguish OUR failure from THEIR failure. This is the single most
    // important line in a monitoring system and it is usually missing.
    return { status: "infra_error", error: String(err), check: spec.id };
  } finally {
    await sbx.kill();  // disk, memory, zombie chromium, saved passwords
  }
}

And the guest side: a plain Playwright script that classifies its own failure, captures artifacts on the way down, and writes a result file. The classification matters more than it looks — the difference between "DNS didn't resolve" and "the login button moved" is the difference between paging an SRE and filing a ticket with the customer.

// /work/check.mjs -- runs INSIDE the microVM, which is why it can be
// this trusting about the machine it's on.
import { chromium } from "playwright";
import { writeFileSync } from "node:fs";

const t0 = Date.now();
const out = { status: "unknown", phase: "launch", timings: {}, artifacts: [] };

const browser = await chromium.launch();
const ctx = await browser.newContext({
  // Cache posture is DECLARED, not inherited from whoever ran last.
  // "cold" is the default here because the VM is genuinely cold.
  userAgent: "PandaStack-Synthetic/1.0 (+https://status.example.com/probes)",
});
const page = await ctx.newPage();

try {
  out.phase = "navigate";
  const nav = await page.goto(process.env.TARGET_URL, {
    waitUntil: "domcontentloaded",
    timeout: Number(process.env.BUDGET_MS),
  });
  out.http_status = nav?.status() ?? 0;
  out.timings.nav_ms = Date.now() - t0;

  if (process.env.CHECK_USER) {
    out.phase = "login";
    await page.fill("#email", process.env.CHECK_USER);
    await page.fill("#password", process.env.CHECK_PASS);
    await page.click("button[type=submit]");
    await page.waitForSelector("[data-testid=dashboard]", { timeout: 15000 });
    out.timings.login_ms = Date.now() - t0;
  }

  out.phase = "assert";
  out.timings.nav_timing = await page.evaluate(() =>
    JSON.parse(JSON.stringify(performance.getEntriesByType("navigation")[0])),
  );
  out.status = out.http_status >= 200 && out.http_status < 400 ? "pass" : "fail";
} catch (err) {
  // Failing IN a phase is the diagnosis. Failing generically is a ticket.
  out.status = "fail";
  out.error_class = err.name === "TimeoutError" ? "timeout" : "assertion";
  out.error = String(err).slice(0, 500);
  await page.screenshot({ path: "/work/failure.png", fullPage: true });
  out.artifacts.push("failure.png");
} finally {
  out.timings.total_ms = Date.now() - t0;
  writeFileSync("/work/result.json", JSON.stringify(out, null, 2));
  await browser.close();
}

Keeping the numbers honest: cold cache is a decision, not a default

Here is the part nobody warns you about when they sell you per-run isolation. A fresh VM has an empty HTTP cache, an empty DNS cache, no TLS session to resume and no warm connection pool. Excellent for isolation — and it systematically biases your latency numbers pessimistic relative to a returning human, who already has most of your static assets.

The old shared pool biased them the other way, and worse: it biased them randomly, because how warm your cache was depended on which worker you landed on and what the previous tenant happened to fetch. Cold-and-consistent beats warm-and-arbitrary every time. But you still have to be deliberate about it, and the fix is to make cache posture an explicit property of the check rather than an accident of the topology.

• Declare the posture per check. A "cold visitor" check runs as-is on the fresh VM. A "returning visitor" check does an unmeasured warm-up navigation first, then measures the second load. Store which one you did in the result, because a cold number compared against a warm SLO is a false alarm generator.
• Bake warmth into the snapshot, don't accumulate it. If you want a warm profile — a primed HTTP cache, a resolved DNS entry, a logged-in session — put it in the template at bake time so every VM starts from the identical warm state. Deterministic warmth is a measurement; accidental warmth is noise.
• Fork when you need shared warm state across many checks. A same-host fork lands in 400-750ms (1.2-3.5s cross-host), so you can warm one VM, fork it per check, and have every run start from a byte-identical cache. It costs more than a plain create; use it where the determinism actually buys you something.
• Never fold setup into the reported number. The VM create, the script upload, the browser launch — those are your infrastructure's latency, not the target's. Report them separately (the runner above keeps `setup_ms` distinct from the in-guest timings) so a slow scheduler never looks like a slow customer site.

Egress identity: probes want to be recognisable

Every sandbox gets its own network namespace, which means egress policy is a host-side rule the guest cannot argue with. On PandaStack that's backed by 16,384 pre-allocated /30 subnets per agent host, so per-check network identity is the default shape rather than something you bolt on. What you do with that capability is the interesting question, and monitoring wants the opposite of what a scraping fleet wants.

A scraper rotates egress IPs to avoid being recognised — that fleet shape is /blog/ephemeral-scraper-fleet-egress-rotation. A monitoring probe wants the opposite: to be recognised, permanently, from a small stable set of addresses customers can allowlist in their WAF and exclude from their analytics. Rotate instead and you will spend your life debugging false positives from bot detection challenging your own probe, while customers explain why their conversion rate has a robot in it. Publish the ranges, set an honest User-Agent, keep it boring.

Regional probes are the other half. "Is the site up" is regional in practice — a broken CDN PoP, a GeoDNS misconfiguration, a regional cert rollout, a country-level block — so you run the same check from several regions and treat disagreement as signal. Per-check VMs make that cheap to reason about, because the check is the unit of placement: same snapshot, same browser build, different host, different egress. The only thing that varies is the thing you're measuring.

When 5,000 checks all fire at :00

Monitoring load is not smooth. Humans configure checks at one-minute and five-minute intervals, cron-shaped schedulers fire them on the boundary, and so a large fraction of your entire fleet's work is requested in the same second, every minute, forever. The naive implementation queues 5,000 VM creates at :00, starves, and reports a latency spike that is entirely your own scheduler looking at itself in a mirror.

Three fixes, none exotic. Spread deterministically: hash the check ID into its interval so check X always runs at second 37 of its minute — stable per check, flat across the fleet. Cap concurrency on real resources: a browser VM is memory-shaped, so the ceiling is host RAM, not a made-up worker count. And make lateness explicit: if a check can't start inside its own interval, recording it as "skipped, capacity" is honest, where letting it drift into the next window slowly corrupts the series.

import asyncio, hashlib, time

def offset_for(check_id: str, interval_s: int) -> int:
    """Deterministic per-check offset. Same check, same slot, forever --
    so the time series stays comparable, and :00 stops being a stampede."""
    h = hashlib.sha256(check_id.encode()).digest()
    return int.from_bytes(h[:4], "big") % interval_s


async def dispatch_window(checks, sem, now=None):
    """Fire only the checks whose slot is this second. Never queue the fleet."""
    now = int(now or time.time())
    due = [c for c in checks if now % c.interval_s == offset_for(c.id, c.interval_s)]

    async def one(check):
        # Concurrency is capped by HOST MEMORY, not by an invented worker
        # count. A browser VM is a memory-shaped object.
        if sem.locked() and sem._value == 0:
            return record_skipped(check, reason="capacity")   # honest, not late
        async with sem:
            first = await run_check(check)
            if first["status"] != "fail":
                return record(check, first)

            # Confirm on a DIFFERENT fresh VM in a DIFFERENT region before
            # paging anyone. Because every run starts from the same clean
            # snapshot, this retry is genuinely independent -- it cannot
            # inherit the state that caused the first failure.
            second = await run_check(check, region=check.backup_region)
            agree = second["status"] == "fail"
            return record(check, second, confirmed=agree,
                          classification="outage" if agree else "flaky")

    await asyncio.gather(*(one(c) for c in due))

Flaky versus real, and why fresh VMs make retries mean something

Every monitoring system eventually implements "retry before alerting," and on a shared pool that retry is weaker than it looks: it often lands on the same worker, with the same poisoned DNS cache, the same wedged browser, the same exhausted connection pool. You retried the symptom, not the check. When each run is a fresh VM from an identical snapshot, a retry is actually an independent trial, which is the only condition under which "it failed twice" means anything statistically.

Pair that with failure classification and confirm-in-another-region, and your alert quality improves more than any dashboard change will. Two fresh VMs in two regions both failing at the TCP-connect phase is an outage. One failing on a selector timeout while the other passes is a flaky check or a slow deploy, and it belongs in a report, not on a pager. The general principle — that a check is only as good as the failures it can tell apart — is the subject of /blog/how-to-write-a-health-check-that-catches-real-failures, and it applies just as hard from the outside as it does to the `/healthz` endpoint being probed.

The result envelope is where all of this gets written down: flat, with the phase and the classification in it, your infrastructure's timings kept separate from the target's, and an artifact list so the on-call engineer sees the page at the moment it broke instead of reconstructing it from a stack trace.

{
  "check_id": "acme-checkout-flow",
  "run_id": "9f4c1e02-3d77-4a10-b8de-2c6f0b9a1d55",
  "tenant": "acme",
  "region": "eu-west",
  "dispatched_at": "2026-08-28T14:03:37Z",
  "status": "fail",
  "phase": "login",
  "error_class": "timeout",
  "error": "TimeoutError: waiting for selector [data-testid=dashboard]",
  "http_status": 200,
  "cache_posture": "cold",
  "browser": "chromium-141.0.0",
  "egress_ip": "203.0.113.44",
  "timings": {
    "nav_ms": 812,
    "login_ms": 15417,
    "total_ms": 15982
  },
  "platform": {
    "sandbox_id": "c1b7a4e8-5f21-4d90-9a3c-77e0d2f4b118",
    "setup_ms": 186,
    "template": "browser",
    "reused": false
  },
  "confirmation": {
    "required": true,
    "second_run_region": "us-east",
    "second_run_status": "pass",
    "classification": "flaky"
  },
  "artifacts": ["failure.png", "trace.zip", "console.log"]
}

Shared browser pool vs. one microVM per check

Same workload, two topologies. As always, verify anything load-bearing about a specific browser automation stack, monitoring vendor, or container runtime against its own current docs — the details differ by version and they move.

• Isolation boundary — Shared pool: browser contexts and, at best, containers over one host kernel, frequently weakened by --no-sandbox or shared volumes. Per-check microVM: its own guest kernel behind hardware virtualization, so a renderer escape has nowhere useful to go.
• Credential residue — Shared pool: profile dirs, saved passwords, HSTS pins, cert exceptions, /tmp downloads and crash dumps persist until a cleanup script you maintain removes them. Per-check microVM: credentials exist as process env for one run, on a disk that is deleted with the VM.
• Cache and network state — Shared pool: HTTP cache, service workers, DNS cache and keep-alive pools are shared across tenants, so one tenant's traffic changes another's numbers. Per-check microVM: empty by default, or identically warm from the snapshot — either way a decision, not a leak.
• Runaway scripts — Shared pool: one busy loop or wedged Chromium takes out every check queued on that worker, which reads as a customer outage. Per-check microVM: burns its own VM's cgroup, gets reaped by TTL, blast radius of exactly one check.
• Egress identity — Shared pool: workers share an IP, so a WAF challenge or rate limit hits every tenant on that box. Per-check microVM: its own network namespace with host-enforced rules; stable allowlistable ranges by design, backed by 16,384 pre-allocated /30 subnets per host.
• Startup cost — Shared pool: near-zero, because the browser is already running (which is the entire problem). Per-check microVM: snapshot-restore at p50 179ms / p99 203ms, inside the noise of a browser check that runs for seconds.

When this is overkill, and what it costs

I would rather you skip this than cargo-cult it. If you are monitoring your own handful of properties with your own credentials, and no third party can configure a check or supply a URL, then multi-tenancy is not your threat model and the leaks above are all leaks between you and yourself. A container per check with a fresh profile is a perfectly respectable answer, it is cheaper, and the engineering time is better spent making your checks assert something meaningful about your product instead of loading the homepage and squinting at it.

The per-check microVM earns its place at a specific intersection: multiple tenants, customer-supplied URLs or customer-authored scripts, and real credentials in the run. That is where a renderer zero-day stops being an abstract CVE and becomes a cross-tenant credential incident. It also earns its place when the measurements are the product — if customers reason about your latency numbers contractually, per-run determinism is a feature, not hygiene — and for anyone who has to explain to an auditor where customer credentials are processed and how long they persist, because "in a VM that existed for 40 seconds and was destroyed" is an unusually easy answer to give.

Be honest about the costs. You take on capacity management: browser VMs are memory-shaped, so a region's check volume converts fairly directly into host RAM, and you size for the peak rather than the average. You lose accidental cache warmth, so the numbers shift the day you migrate and you must decide, per check, what posture you actually meant. Debugging is a step removed — the worker is gone, so your artifact bundle (screenshot, trace, console log, HAR) has to be good enough to diagnose from. And a pinned browser is correct and is also a thing you now update on purpose.

The reason I still think it's the right shape is the asymmetry. The cost is measured in engineering hours, some capacity planning, and roughly two hundred milliseconds per check. What it buys is that a compromised page, a leaked cookie jar, a customer's malicious check script, and a wedged Chromium all become the least interesting events in your monitoring fleet — which is exactly what you want from the system whose entire job is to tell you when something interesting has happened somewhere else.