Building an ephemeral web-scraper fleet on microVMs

If you run a data-collection platform, the shape of the problem is fan-out: you have a queue of crawl jobs, and you want each one to run in a clean, isolated environment, produce a result, and vanish. The tempting design is a long-lived pool of scraper workers that chew through the queue forever. It works on day one and rots by month three — state leaks between jobs, memory bloats, and one pathological page can wedge a worker that's also handling two hundred others. This post is about the opposite design: one short-lived microVM per crawl job, spun up on demand, killed the moment it's done.

I'm Ajay — I built PandaStack, a Firecracker microVM platform. This is a use-case walkthrough of the ephemeral-fleet pattern: why fresh-state-per-job beats a long-lived pool, how per-VM network namespaces give you real egress control, how to fan out N crawlers in parallel over the create fast path, and how to collect results back. To be clear up front about scope: this is about legitimate data collection — respecting robots.txt, honoring rate limits and Terms of Service, and identifying your crawler honestly. The value here is operational hygiene and isolation, not evading anyone's detection. If a site says no, the answer is no.

Why ephemeral scrapers beat a long-lived pool

A long-lived scraper worker accumulates. Cookies, local storage, DNS caches, a browser profile that's now three thousand tabs of history deep, leaked file descriptors, and the slow memory creep of a headless browser that was never designed to run for a week straight. Job N+1 inherits whatever job N left behind. Most of the time that's invisible; occasionally it's a cross-job contamination bug that takes you a day to reproduce because it only happens when a specific pair of jobs land on the same worker in a specific order.

Then there's fate-sharing. In the shared-process design, a scraper that OOMs after ten thousand pages doesn't just fail its own job — it takes its two hundred in-flight neighbors down with it, because they were all Python coroutines in the same interpreter. You lose the crash, the results in flight, and the ability to attribute the failure to a single job. Ephemeral VMs make the blast radius exactly one job: the VM that OOMs dies alone, its neighbors never notice, and the queue simply reschedules the one job that failed.

The classic objection to per-job VMs is startup cost — nobody wants to pay three seconds of boot per crawl. That's the constraint PandaStack is built to remove: every create restores a baked snapshot rather than cold-booting, so a create is p50 179ms (p99 around 203ms), with the underlying snapshot restore around 49ms. The first cold boot of a template is about 3 seconds, once, and then every subsequent create rides the snapshot path. At that latency, one-VM-per-job stops being a luxury and becomes the default.

One microVM per crawl job: what you actually get

Each PandaStack sandbox is its own Firecracker microVM: its own guest kernel, its own network namespace, its own TAP device, its own filesystem. That's a stronger boundary than a container, which shares the host kernel with every neighbor. For a scraper fleet the practical wins line up like this:

• Fresh fingerprint per job — a brand-new browser/HTTP-client profile every time, with no cookie, cache, or history carried over from the previous crawl. State starts empty and dies empty.
• No cross-job contamination — a bug, a wedged browser process, or a filled-up disk in one job cannot leak into the next, because the next job is a different VM entirely.
• No fate-sharing on crash — a VM that OOMs or panics takes down exactly one job. Its neighbors are separate kernels and separate memory; they keep running.
• Per-VM egress control — because each VM has its own network namespace, you can enforce outbound rules (allow-lists, DNS pinning, rate policy) at the namespace layer, per job, not just globally.
• Trivial fan-out — creating a hundred VMs is a hundred independent create calls over the fast path; there's no shared pool to contend on.

Threads vs container-per-job vs microVM-per-job

• Threads/coroutines in one process — cheapest and fastest to start, zero isolation. Shared memory, shared file descriptors, shared fate: one OOM or segfault kills every concurrent job. Fine for a handful of trusted, well-behaved crawls; a liability at fleet scale.
• Container-per-job — real process isolation and its own filesystem view, but all containers share the host kernel and, by default, host network plumbing. A kernel bug or a resource-exhaustion crash can still reach neighbors, and network isolation takes deliberate setup. Good middle ground; not a hard boundary.
• MicroVM-per-job — separate guest kernel, separate network namespace, separate TAP per VM. Hardware-virtualized isolation, so a crash or exhaustion is contained to one job, and egress rules are enforceable per VM. Slightly more memory per job than a container, but the create-via-snapshot path keeps startup in the sub-200ms range.

Per-VM networking and egress control

The networking model is what makes this pattern more than "containers with extra steps." Each agent host pre-allocates 16,384 /30 subnets (a NATID pool), each a fully built network namespace with a veth pair and a TAP device ready to hand to a VM. When a scraper VM is created it drops into one of these pre-built namespaces, so it has its own isolated egress path rather than sharing one big NAT with every other job on the box.

Because the boundary is a real Linux network namespace, egress policy lives at the namespace layer — below the scraper code, where the crawl job can't disable it. That's exactly where you want to enforce the boring-but-important controls of a well-behaved fleet: an allow-list of destination hosts a given batch is permitted to reach, blocking traffic to internal/metadata ranges so a redirect or an SSRF in fetched content can't pivot inward, and pinning DNS so name resolution is predictable and auditable. The scraper asks to reach a host; the namespace decides whether it may.

Warm the image once, fork the fleet

You don't want each of a thousand scrapers to cold-start a headless browser and warm its caches from scratch. The move is to warm one VM to a known-good state — browser launched, HTTP client configured, common libraries paged in — snapshot it, and then fork that snapshot per job. A fork shares the parent's memory copy-on-write, so each child starts from the warmed state and only diverges on the pages it actually touches. Same-host forks land in roughly 400–750ms; cross-host forks are 1.2–3.5s because the memory image has to move over the network first.

Practically: bake a scraper template (or snapshot a configured sandbox once), then create-or-fork per job with a tight ttl_seconds so any VM you forget to kill reaps itself. The warmed state is shared read-only until a child writes to it, which is what keeps a large fan-out cheap on memory.

Fanning out a batch with the Python SDK

Here's the core loop: take a list of crawl jobs, spin up a sandbox per job concurrently, run the crawl inside each, read the result back through the filesystem API, and kill the VM. A thread pool is plenty — the create call is sub-200ms, so the wall-clock time is dominated by the crawls themselves, not by provisioning. Set PANDASTACK_API_KEY in your environment and the SDK picks it up.

import json
from concurrent.futures import ThreadPoolExecutor, as_completed
from pandastack import Sandbox

# A batch of legitimate crawl jobs. Each dict is one URL to collect,
# on your own or an approved, robots-respecting target.
jobs = [
    {"id": "j1", "url": "https://example.com/catalog/1"},
    {"id": "j2", "url": "https://example.com/catalog/2"},
    {"id": "j3", "url": "https://example.com/catalog/3"},
]

CRAWL = """
import json, sys, urllib.robotparser as rp
import httpx

URL = sys.argv[1]
UA = "AcmeDataBot/1.0 (+https://acme.example/bot; [email protected])"

# Respect robots.txt before fetching anything.
robots = rp.RobotFileParser()
robots.set_url(URL.rsplit('/', 3)[0] + '/robots.txt')
try:
    robots.read()
except Exception:
    pass
if not robots.can_fetch(UA, URL):
    print(json.dumps({"url": URL, "skipped": "disallowed by robots.txt"}))
    sys.exit(0)

r = httpx.get(URL, headers={"User-Agent": UA}, timeout=20.0)
result = {"url": URL, "status": r.status_code, "bytes": len(r.content)}
with open("/workspace/result.json", "w") as f:
    json.dump(result, f)
print("ok")
"""

def run_job(job: dict) -> dict:
    # Fresh microVM per job: own kernel, own netns, own TAP. Dies when done.
    with Sandbox.create(template="scraper", ttl_seconds=180) as sbx:
        sbx.filesystem.write("/workspace/crawl.py", CRAWL)
        res = sbx.exec(f"python3 /workspace/crawl.py {job['url']}",
                       timeout_seconds=60)
        if res.exit_code != 0:
            return {"id": job["id"], "error": res.stderr.strip()}
        raw = sbx.filesystem.read("/workspace/result.json")
        return {"id": job["id"], **json.loads(raw)}
    # sandbox is destroyed here — no state survives the job

# Fan out: one VM per job, up to 32 in flight at once.
with ThreadPoolExecutor(max_workers=32) as pool:
    futures = {pool.submit(run_job, j): j for j in jobs}
    for fut in as_completed(futures):
        print(fut.result())

Every job gets a clean VM, a timeout on the exec (your circuit breaker against a crawl that hangs), and a ttl_seconds backstop on create so nothing leaks if your process dies mid-batch. Results come back as structured JSON on the guest filesystem — read it, parse it, and the VM is gone. Scale the batch by raising max_workers; the create fast path is what makes a wide fan-out cheap.

Getting results out and into a store

The cleanest collection pattern is the one above: the crawl writes structured JSON to a known path, the host reads it back over the filesystem API, and the VM is discarded. Don't scrape stdout for anything load-bearing — have the guest write a result file and read that. For larger payloads, have the crawl push directly to object storage or your database from inside the VM (subject to the egress allow-list), and return just a pointer. Either way, from the fleet's perspective a job is a pure function: URL in, result out, VM gone.

# One-shot crawl-and-collect from the CLI, useful for smoke-testing a job.
SBX=$(pandastack create --template scraper --ttl 120 --json | jq -r .id)

# Run the crawl inside the fresh microVM.
pandastack exec "$SBX" -- python3 /workspace/crawl.py "https://example.com/page"

# Pull the structured result back out, then tear the VM down.
pandastack fs read "$SBX" /workspace/result.json > result.json
pandastack kill "$SBX"

cat result.json | jq .

When a VM per job is the wrong call

This pattern earns its keep when jobs are numerous, short, untrusted-in-content, or prone to memory blowups — exactly the profile of a broad crawl. It's overkill when you have a handful of long, well-behaved crawls of your own sites that you fully trust: a single well-instrumented worker is simpler. And it's not a license to crawl harder — the isolation is about keeping your fleet clean and your egress controlled, not about extracting data faster than a site permits. Within those bounds, one microVM per crawl job gives you fresh state every time, a real network boundary per job, blast-radius-of-one on failure, and fan-out that costs you sub-200ms of provisioning per VM. The scraper that OOMs after ten thousand pages still dies — it just dies alone now.