The best web scraping and crawling platforms in 2026

Ask ten teams which scraping platform they use and you get ten answers that are not comparable to each other. One says Bright Data. One says Firecrawl. One says Apify. One says Browserbase. One says we run Scrapy on three boxes and a Postgres table. Those are not five vendors competing in one market. They are four different layers of one stack, plus a team that built the whole thing themselves, and the reason most comparison articles about this category read like a shopping list is that they line up products which do not do the same job and then rank them.

So this is not a leaderboard. It is a map. I want to sort the field into the four layers it actually contains, say what each layer is genuinely good at, and be explicit that the right answer for most teams is a combination of two or three of them rather than one vendor who claims to cover everything.

I'm Ajay, and I build PandaStack — Firecracker microVMs you run your own code in. That puts me squarely at layer four below, which also makes me the wrong answer for a large fraction of the people who type best web scraping platform into a search box. I have tried to write the layer-four section so it is easy to skip when it is not your problem. You will find no competitor prices, no success rates and no proxy pool sizes anywhere in this post: this category reprices and repositions constantly, and a number I half-remember is worse than no number at all. Everything about other vendors here is qualitative and taken from their public documentation. Check current pricing and terms against their own docs before you commit to anything.

The four products hiding behind one phrase

Every scraping problem decomposes into the same sequence. Something has to make the request from an address the site will answer. Something has to render the page if the content only exists after JavaScript runs. Something has to turn a rendered page into a record with fields in it. And something has to orchestrate all of that across a queue of a hundred, or a hundred million, URLs, while remembering what it has already done.

Products in this market pick a contiguous slice of that sequence and sell it. There are four common slices:

1. Proxy and unblocking networks. You keep your own HTTP client and your own crawler; they sell you the egress path — the address the request comes from, and increasingly a managed request pipeline that handles retries, headers and session stickiness for you. The interface is usually a proxy endpoint or a single fetch API.
2. Browser infrastructure. You keep your own automation code; they run the Chromium. The interface is a remote CDP or WebSocket endpoint your existing Playwright or Puppeteer script connects to instead of launching a local browser. What you are buying is the operational misery of running browsers at scale, removed.
3. Turnkey scraping and extraction APIs. You send a URL, they send back content — HTML, JSON, or increasingly clean markdown shaped for a language model to read. The browser, the proxy and the parsing are all implementation details you never see. This layer now includes an LLM-flavoured wing that will also do the extraction, so you send a URL and a schema and get back typed fields.
4. Raw compute. Nobody's abstraction. A machine, a container or a microVM, and your own crawler running on it — Scrapy, Crawlee, Playwright, a Go program, whatever. You own the frontier, the retries, the parsing and the browser lifecycle. In exchange you own the data path end to end and pay for compute rather than per page.

Once you have these four in your head, a lot of confusing product positioning resolves. Zyte sells across layers one and three. Apify is mostly three but with a real four-shaped platform underneath it. Firecrawl is three with an opinion about who is going to read the output. Browserbase and Browserless are two. Bright Data is one, with layer-three products bolted on top of its own network. A vendor that appears to compete with another vendor is often selling a different layer to a different buyer inside the same company.

And the useful consequence: the question is not which platform is best, it is which layers you should buy and which you should own. Almost every serious pipeline I have seen buys one or two and owns the rest.

The unit you are buying, which is never a scrape

Before the vendors, the pricing shapes, because they differ more than the products do and they are what makes a cheap-looking option expensive in month three. There are three shapes in this market and they reward completely different workloads.

Per page, or per successful request

The extraction APIs mostly price this way, sometimes with a multiplier for pages that needed a browser or a harder unblocking path. It is the easiest model to reason about and the easiest to forecast: pages times rate, done.

Two things distort it. First, a page you fetch and discard still costs. Crawls have terrible yield — you walk category listings and pagination to find the twenty product pages you actually wanted, and every one of those navigational fetches bills the same as a useful one. Measure your real ratio of fetched pages to kept records before you extrapolate anything. It is frequently five or ten to one and people budget as though it were one to one.

Second, retries. Whether a failed attempt bills, and what counts as failure, is the single most consequential line in a per-page contract. Ask specifically. A 404 is a legitimate answer to a request; a timeout is not, but somebody has to have burned resources to produce it.

Per gigabyte

Proxy networks conventionally price on traffic, and it catches people out badly, because a modern page is not the HTML. Fetch a product page with a real browser and you pull the document, then fonts, then a few hundred kilobytes of JavaScript, then a hero image, then analytics beacons, then whatever a tag manager decides to load. The record you extract might be eight hundred bytes. You paid for three megabytes.

This is why request interception is a cost control and not just a speed trick. Blocking images, media, fonts and third-party analytics at the browser level before they leave the machine can change a per-GB bill by an order of magnitude, and it usually does not change the extracted data at all. If you are on a per-GB plan and you are not intercepting, that is the first thing to fix, ahead of any vendor change.

Concurrency, which is the real unit under all of them

Here is the thing most pricing pages obscure. Whatever the invoice is denominated in, the resource you are actually consuming is concurrent slots held over time. A crawl is not compute-bound, it is wait-bound: the overwhelming majority of a scrape's wall clock is a socket sitting open waiting for someone else's server, or a browser idling while a page settles. Throughput equals concurrency divided by average latency, and latency is set by the target, not by you.

That has a practical consequence people discover late. Doubling your crawl rate against a slow target means doubling concurrency, and concurrency is what every layer of this stack meters, formally or informally. Browser platforms sell it by name. Proxy networks limit it per account. Your own fleet has it as a memory ceiling. And a target that gets slower under load — which is what a target does when you crawl it hard — silently increases your concurrency requirement for the same page rate, which is how a crawl that was fine on Tuesday saturates its plan on Thursday without anyone changing a line of code.

The legal and terms-of-service part, stated plainly

I am going to be direct about this because a lot of writing in this category is evasive. Scraping is not one legal question, it is at least four, and they have different answers.

• Contract. The site's terms of service may prohibit automated access. Whether those terms bind you can depend on whether you agreed to them and how, which varies by jurisdiction and by how the site presents them. This is the question a proxy does not change: routing a request differently does not alter what you agreed to.
• Computer-misuse law. Different jurisdictions and different courts have reached genuinely different conclusions about when accessing a public web page without permission crosses a statutory line, particularly around whether a technical block or a cease-and-desist changes the answer. This area has moved and will move again.
• Copyright and database rights. The page content is somebody's work. What you may do with a copy — index it, quote it, train on it, republish it — is a separate question from whether you were allowed to fetch it, and the answer differs across regions.
• Personal data. If any field you extract is personal data, a whole separate regime applies regardless of whether the page was public. Under GDPR and similar laws, public availability is not a lawful basis on its own. This is the one that most often turns a technical project into a compliance project.

The engineering practices that keep you on the right side of most of this are boring and cheap. Read and honour robots.txt. Set a truthful User-Agent with a contact address, and answer the mail when it arrives. Rate-limit per host and back off hard on 429 and 503 rather than treating them as noise. Cache aggressively so you are not re-fetching what you already have. Do not collect personal fields you do not need. And if a site tells you to stop, stop — that is a conversation with a lawyer, not a problem to route around.

On the unblocking category specifically, I will describe it factually and leave it there. Products in that space exist to make requests succeed against sites that are actively trying to distinguish automated traffic from human traffic, using residential and mobile exit addresses, fingerprint management and challenge handling. That is what they sell and it is legal to sell. Whether using it on a given target is appropriate for you is a question about that target and your purpose, not about the tool. I am not going to write a how-to for it, and I would treat any vendor who frames the capability as a way to ignore a site's stated wishes as a risk you are taking on rather than a service you are buying.

One diligence item worth naming because it is a real supply-chain question: ask any residential proxy vendor how their network is sourced and what consent the endpoint owners gave. The answers vary across this industry and some of them have been the subject of significant public scrutiny. It is a reasonable thing to ask in a procurement call and the quality of the answer is informative.

Layer 1: proxy and unblocking networks

What you are buying: an egress path, and increasingly a managed request pipeline in front of it. You keep your crawler. The interface is either a proxy endpoint you point an HTTP client at, or a single-call API that takes a URL and handles the whole request lifecycle.

Bright Data is the largest and broadest name here. It sells access to several kinds of network — datacenter, ISP, residential, mobile — and layers products on top, including managed unblocking, search-results retrieval and pre-collected datasets. If your problem is genuinely that requests do not succeed, this is the category and Bright Data is where most evaluations start. Zyte comes at the same problem from the crawler side: it is the company behind Scrapy, and its API bundles proxy selection, browser rendering and retry policy behind one call, with a bias toward being driven from a real crawler framework rather than one-off fetches.

The tradeoffs of this layer are consistent whichever vendor you pick. Pricing is usually traffic-shaped, which drags in everything from the per-GB section above. Debugging becomes harder, because a failure could be your parser, the target, or the exit address, and you can no longer reproduce it locally. And you have introduced a dependency in the hot path of every single request you make, which means their incident is your incident.

The strongest argument for this layer is also simple: if you are being blocked and it matters commercially, the maintenance of an unblocking capability is a full-time job for a team, and buying it is nearly always cheaper than staffing it. The strongest argument against is that many teams buy it before establishing that they need it. A large fraction of scraping targets are ordinary sites that will serve a politely-rate-limited, honestly-identified crawler from a datacenter address indefinitely. Test that first; it costs you an afternoon.

Layer 2: browser infrastructure

What you are buying: somebody else runs the Chromium. You connect your existing Playwright or Puppeteer code to a remote endpoint instead of launching locally, and your script is otherwise unchanged.

Browserbase and Browserless are the two names most people arrive with, and they differ in a way worth knowing. Browserless is source-available and designed to be run yourself in a container, which is why teams with data-residency or single-cloud requirements gravitate to it. Browserbase is hosted-only and has aimed itself increasingly at agent workloads, where session recording and replay matter because you need to see what an autonomous thing actually did. Steel occupies a middle position with an open-source core and a hosted offering. I have written separately about both fields, linked at the bottom, so I will not re-run those comparisons here.

What this layer genuinely removes is real. A headless browser is one of the worst-behaved long-lived processes in ordinary use: memory climbs forever, crashed tabs hold slots until something kills them, zombie processes accumulate, and the version needs updating on a treadmill. Somebody has to own that. Paying somebody to own it is a completely defensible decision.

What it does not remove is the two things people expect it to. It does not, by itself, solve blocking — several vendors in this layer bundle proxy and stealth features, but that is them reselling or reimplementing layer one, and you should evaluate it as such. And it does not give you a machine. You have a browser, not a Linux box: you cannot install a system package next to it, run a custom binary in the same environment, or do meaningful work in the same process space as the page.

The other thing to check in this layer is pricing shape against your traffic. Browser work is bursty and full of waiting, and a session parked on a login form consumes a slot exactly like one grinding through a render. If your peak concurrency is ten times your median, a reserved-concurrency plan means buying the peak and paying for it all month.

Layer 3: turnkey scraping and extraction APIs

What you are buying: content. You send a URL, you get back something structured. The proxy, the browser and the parsing are all hidden. This is the highest-leverage layer for most teams and the one I would default to for anything under a few hundred thousand pages a month.

ScrapingBee is the clean archetype: an HTTP API that takes a URL, optionally renders it in a browser, handles the proxying, and gives you back the page. Small surface area, easy to reason about, one call from anywhere. Zyte's API sits in the same space with the crawler-framework heritage described above and a heavier emphasis on running as part of a large managed crawl. Both are the correct answer to the question do I need the content, or do I need control of a browser, when the answer is the former.

The markdown-for-agents wing

Firecrawl is the clearest example of the newer flavour: same fetch-and-return shape, but the output is deliberately cleaned and converted to markdown, with boilerplate stripped, so that it can go straight into a model's context without you writing a readability pass. It also crawls — give it a site and it will walk it — and it will do schema-driven extraction, where you supply a description of the fields you want and get typed JSON back. The open-source core means you can run the same thing yourself if you would rather.

Be clear-eyed about what that last feature is. Schema-driven extraction means a language model is reading the page and filling in your fields, and it is priced accordingly and fails accordingly. It is superb for the long tail — two hundred sites you will each hit once, where writing two hundred parsers is absurd. It is the wrong tool for the twelve sites you hit a million times each, where a CSS selector is faster, cheaper, deterministic, and fails loudly and immediately when the site changes rather than quietly returning a plausible wrong answer.

That distinction — long tail versus hot path — is the single most useful way to decide whether the LLM extraction layer belongs in your pipeline. Most mature pipelines end up using both, with selectors on the sites that matter and model extraction as the fallback for everything else.

Apify, which is its own shape

Apify does not fit the layering cleanly and that is the interesting thing about it. At the surface it is a marketplace: a store of prebuilt scrapers, called Actors, that other people have written and maintain, which you run against your targets without writing code. Underneath it is a platform for running containerised scraper jobs with scheduling, storage, proxy integration and queueing, plus Crawlee, their open-source crawler library, which is genuinely good and which you can use entirely off their platform.

So it is layer three when you use somebody else's Actor and layer four when you write your own and deploy it there. The marketplace is the differentiator and it is a real one: if a maintained Actor already exists for your target, you have skipped the entire build. The corresponding risk is ordinary marketplace risk — an Actor is somebody else's code with somebody else's maintenance commitment, and the one you depend on may or may not be updated when the target site redesigns. Check the update history before you build a business process on one.

Layer 4: raw compute and your own crawler

What you are buying: nothing above the machine. Scrapy or Crawlee or Playwright or your own code, running on VMs, containers, functions or microVMs, with you owning the frontier, the deduplication, the retry policy, the browser lifecycle and the parsing.

This is the option people either dismiss instantly or choose reflexively, and both reflexes are wrong. Here is the honest case for it. There are five conditions, and you need at least two:

• Volume. Above some threshold, per-page pricing multiplied by your page count exceeds the cost of a few machines and an engineer's time by enough to be a line item somebody notices. Where that threshold sits depends on your yield ratio, and it is usually higher than enthusiasts claim and lower than vendors imply. Do the arithmetic with your own numbers.
• Custom logic in the loop. Your crawl decides what to fetch next based on what it just parsed, maintains state across pages, calls your own models mid-crawl, or fetches something that is not HTTP. Extraction APIs are stateless URL-to-content functions and this shape does not fit through them.
• Data residency. The content cannot transit a third party, or the extracted records cannot leave your cloud account or your jurisdiction. This one is binary and it settles the question by itself.
• You already own the crawler. Ten years of Scrapy spiders with your parsing rules in them is an asset. Rewriting it against someone's API to save operational overhead is often a net loss.
• Unusual runtime needs. A specific browser build, a native binary in the pipeline, a headful browser, a system dependency, GPU-adjacent post-processing. The moment you need a package installed next to the browser, layers two and three are out.

And the honest case against, which is longer than people expect. A crawler is not the hard part. The hard part is everything around it: a frontier that survives a restart without re-fetching a million pages, URL canonicalisation and deduplication, per-host politeness that actually holds under concurrency, retry classification that distinguishes transient failures from permanent ones, and a browser fleet that does not slowly eat itself. Every one of those is a week you did not budget, and layers one to three exist because those weeks are real.

Where compute choice matters within this layer is isolation and cost shape. A crawler is, by construction, executing content chosen by other people — and if there is a browser in the loop, it is executing their JavaScript on your machine, which is the browser's entire job. Running that alongside your other workloads on a shared kernel is a decision, whether or not anyone made it deliberately. And the cost shape matters because crawls are bursty: a fleet sized for a nightly window sits idle for twenty-two hours, and whether that idle time is billed is often a bigger factor than the hourly rate.

The four interfaces, side by side

The clearest way to feel the difference is to look at what you type. These are illustrative shapes with placeholder hostnames, not real endpoints for any vendor:

# Layer 1 -- proxy / unblocking network.
# Your HTTP client, your crawler, their exit address.
curl -x http://user:pass@proxy.vendor.example:8000 https://target.example/p/1

# Layer 2 -- browser infrastructure.
# Your Playwright script, their Chromium. One line changes: connect, not launch.
#   browser = playwright.chromium.connect_over_cdp(
#       "wss://browser.vendor.example/?token=" + TOKEN)

# Layer 3 -- extraction API.
# Their everything. You supply a URL and, optionally, a schema.
curl -X POST https://api.vendor.example/v1/scrape \
  -H "Authorization: Bearer $VENDOR_KEY" \
  -d '{"url":"https://target.example/p/1","format":"markdown"}'

# Layer 4 -- raw compute.
# Your crawler, your dependencies, an isolated machine to run them on.
pandastack exec "$SBX" -- python3 /workspace/worker.py

Notice how the amount of your own code goes down and the amount of somebody else's judgement goes up as you move down the list. That is the whole tradeoff, and it is why the answer is usually not one layer.

Almost nobody buys one layer

Here are four combinations I see repeatedly, all of them sensible:

• Own crawler, bought egress. Scrapy or Crawlee on your own compute, pointed at a proxy or unblocking API for the requests that need it and at plain HTTP for the ones that do not. The most common serious-scale shape, because the crawl logic is where your value is and the egress is where the specialist expertise is.
• Extraction API for the long tail, own pipeline for the hot path. Twelve sites with hand-written selectors on your own machines; three hundred one-off sources through a layer-three API. Costs are dominated by the hot path, which you control, and the long tail never turns into three hundred maintenance burdens.
• Browser infrastructure plus your own orchestration. You keep the queue, the frontier and the parsing; you rent the Chromium. Good when browser operations were the only genuinely painful part and you did not want the rest abstracted away.
• Everything bought, on purpose. Small volume, no differentiation in the collection itself, and an engineering team whose time is worth more elsewhere. Buy the extraction API, write no infrastructure, and revisit when the invoice becomes interesting. This is the correct answer far more often than infrastructure people like to admit.

Five things that bite in production

These are the items that do not appear on comparison tables and that determine whether the thing you built is still running in six months.

1. Session and cookie persistence across a crawl

Anything behind a login turns the crawl into a stateful problem, and most of this market is architected around stateless requests. You need the same cookie jar, and often the same exit address, across a sequence of pages — because a session that suddenly appears from a different country is exactly the pattern fraud systems are built to notice, and legitimately so.

Two consequences. First, sticky sessions are a feature you must check for by name at layer one, along with how long stickiness lasts and what happens when it expires mid-crawl. Second, measure how much of your wall clock is authentication rather than collection. On a lot of pipelines the login, the two-factor prompt and the cookie banner are the majority of the runtime, and the entire pipeline gets faster and cheaper if you can do that once and reuse the resulting state instead of repeating it per job.

2. The frontier, and what happens when it dies

The frontier is the set of URLs you have seen, fetched, failed on and deferred. Teams start with an in-memory set, which is correct for a first version and then quietly becomes the most important database in the system. When a process dies mid-crawl, the frontier is what stops you re-fetching four hundred thousand pages you already paid for.

Get canonicalisation right early. Query-parameter ordering, tracking parameters, trailing slashes, case in the path, session identifiers in the URL — a crawl without canonicalisation will happily fetch the same page eleven times and bill you eleven times. Put the frontier in a real database with a unique index on the canonical URL, and make claiming a URL atomic so two workers cannot take the same one.

3. Egress, the sneaky line item

Bandwidth shows up three times and people usually budget for one of them. It appears in your proxy bill if you are on a per-GB plan. It appears in your cloud provider's egress charges if your crawler talks to the internet from inside a cloud, which is where crawlers live. And it appears again when you ship the collected data somewhere else — to object storage in another region, to a warehouse, to a partner.

Ingress is usually free and egress usually is not, which is mildly perverse for a crawler, whose traffic is overwhelmingly inbound. But every request you make has an outbound component, and every rendered page pulls resources you did not ask for. The controls are the same ones from earlier: intercept and block what you do not need at the browser, prefer plain HTTP over a browser wherever the data allows, compress at rest, and do not move the corpus between regions casually. I run a platform that meters egress, so treat that as a disclosed interest — but the advice holds regardless of who bills you.

4. Failure classification, which is where retries go wrong

A naive crawler treats every non-200 the same and retries three times. That is wrong in both directions. A 404 should never be retried; retrying it wastes a request and, on a per-page plan, actual money. A 429 must be retried but only after the interval the server asked for, and hammering through it is the behaviour that gets a crawler blocked and deserves to. A 403 that suddenly appears across every URL on one host is not a per-URL failure at all — it is a signal that something about your access has changed, and the right response is to stop that host entirely and alert a human, not to burn through the queue generating four hundred thousand identical errors.

Write the classification table before you write the retry loop. Transient, permanent, rate-limited, and blocked are four different states with four different responses, and conflating them is the most common way a crawl turns into an incident.

5. Selectors rot, silently

The site redesigns and your parser keeps returning 200 with an empty price field. Nothing errors. The pipeline is green. The data is quietly worthless and you find out from a downstream consumer three weeks later.

The fix is unglamorous and takes an afternoon: assert on extraction, not on fetching. If the price field is null on more than some small percentage of a host's pages in a batch, fail the batch loudly. Keep a handful of golden pages per source with known-correct expected output and run them on a schedule as a canary. This is the single highest-value hour of work in any scraping project and it is almost always skipped.

A fan-out crawl, in code

Here is the layer-four pattern in the form I would actually write it: shard a URL list across N isolated machines, one shard each, with the dependencies installed exactly once. The trick is that the workers are forks of a single warmed parent, so the pip install and the worker script are paid for once and every child starts from that state. Same-host forks land in roughly 400 to 750 milliseconds and share the parent's memory copy-on-write, which is what keeps a wide fan-out cheap.

"""Shard a crawl across N isolated microVMs, one URL shard per VM.

    pip install pandastack
    export PANDASTACK_API_KEY=...

Targets below must be ones you are permitted to crawl. The worker honours
robots.txt and rate-limits itself per host; keep both if you adapt this.
"""
import json
from concurrent.futures import ThreadPoolExecutor, as_completed

from pandastack import Sandbox

SHARDS = 8
URLS = [f"https://target.example/catalogue/page-{i}.html" for i in range(1, 201)]

# This runs INSIDE each microVM. It reads its shard, fetches politely, and
# writes one JSON file back. No shared state with any other shard.
WORKER = r'''
import json, re, time
import urllib.parse as up
import urllib.robotparser as rp
import httpx

UA = "AcmeResearchBot/1.0 (+https://acme.example/bot; crawler@acme.example)"
PER_HOST_DELAY = 1.0     # seconds between requests to the same host
TITLE = re.compile("<title[^<]*>(.*?)</title>", re.S | re.I)

shard = json.load(open("/workspace/shard.json"))
robots, last_hit, rows = {}, {}, []

def robots_ok(url):
    parts = up.urlsplit(url)
    origin = parts.scheme + "://" + parts.netloc
    if origin not in robots:
        parser = rp.RobotFileParser()
        parser.set_url(origin + "/robots.txt")
        try:
            parser.read()
        except Exception:
            parser = None     # unreachable robots.txt: skip, do not assume yes
        robots[origin] = parser
    parser = robots[origin]
    return bool(parser and parser.can_fetch(UA, url))

with httpx.Client(headers={"User-Agent": UA}, timeout=20.0,
                  follow_redirects=True) as client:
    for url in shard:
        if not robots_ok(url):
            rows.append({"url": url, "state": "skipped_robots"})
            continue

        host = up.urlsplit(url).netloc
        wait = PER_HOST_DELAY - (time.monotonic() - last_hit.get(host, 0.0))
        if wait > 0:
            time.sleep(wait)

        try:
            r = client.get(url)
        except Exception as exc:
            rows.append({"url": url, "state": "transient",
                         "error": type(exc).__name__})
            continue
        finally:
            last_hit[host] = time.monotonic()

        # Four states, four responses -- never one blanket retry rule.
        if r.status_code in (429, 503):
            rows.append({"url": url, "state": "rate_limited",
                         "retry_after": r.headers.get("retry-after")})
            continue
        if r.status_code == 403:
            rows.append({"url": url, "state": "blocked"})
            break                      # stop this host; a human decides next
        if r.status_code >= 400:
            rows.append({"url": url, "state": "permanent",
                         "status": r.status_code})
            continue

        m = TITLE.search(r.text)
        rows.append({"url": url, "state": "ok", "status": r.status_code,
                     "bytes": len(r.content),
                     "title": m.group(1).strip() if m else None})

with open("/workspace/out.json", "w") as f:
    json.dump(rows, f)
print("collected", len(rows))
'''


def shard_urls(urls, n):
    """Round-robin, so every shard sees a mix of hosts rather than one host
    getting all its pages from a single worker at full speed."""
    return [urls[i::n] for i in range(n)]


# 1. Warm ONE machine. Create is a snapshot restore, p50 around 179ms --
#    nothing is booting from cold here.
parent = Sandbox.create(template="code-interpreter", ttl_seconds=1800)
parent.exec("pip install --quiet httpx", timeout_seconds=300, check=True)
parent.filesystem.write("/workspace/worker.py", WORKER)

# 2. Fork the warm parent N ways. Each child inherits the installed deps and
#    the worker script, and gets its own kernel and its own network namespace.
children = parent.fork_tree(SHARDS)


def run_shard(child, urls):
    try:
        child.filesystem.write("/workspace/shard.json", json.dumps(urls))
        res = child.exec("python3 /workspace/worker.py", timeout_seconds=900)
        if res.exit_code != 0:
            return {"failed": res.stderr[-2000:]}
        return json.loads(child.filesystem.read("/workspace/out.json"))
    finally:
        child.kill()          # the VM is disposable; the data is not


collected = []
with ThreadPoolExecutor(max_workers=SHARDS) as pool:
    futures = [pool.submit(run_shard, c, s)
               for c, s in zip(children, shard_urls(URLS, SHARDS))]
    for fut in as_completed(futures):
        out = fut.result()
        if isinstance(out, list):
            collected.extend(out)
        else:
            print("shard failed:", out["failed"][:200])

parent.kill()

ok = sum(1 for r in collected if r["state"] == "ok")
print(f"{ok} ok / {len(collected)} attempted across {SHARDS} shards")

Three details in there are the ones that matter in practice. The shard split is round-robin rather than contiguous, so a single host's pages are spread across workers instead of one worker hammering one host as fast as it can — combined with the per-host delay inside each worker, that keeps you polite without central coordination. Every worker has a hard exec timeout and every sandbox has a ttl_seconds backstop, so a hung crawl cannot leak a machine even if your orchestrating process dies. And results come back as a structured file read over the filesystem API, not scraped from stdout, which stops a stray print statement from corrupting your dataset.

If your targets need a real browser, the same structure holds — swap the template for one with Chromium and Playwright preinstalled, and have the worker drive a page instead of an httpx client. The warm-parent-then-fork trick is worth more in that version, not less, because a browser's startup and cache warm-up is exactly the kind of expensive setup you want to pay for once.

Where PandaStack fits, and where we are the wrong answer

The wrong-answer part first, because it applies to more readers. We do not sell a proxy network. We do not sell an unblocking service, managed stealth, fingerprint management or CAPTCHA handling. We do not sell an extraction API and we have no marketplace of prebuilt scrapers. If your blocker is that requests are failing, or your requirement is a URL-in-JSON-out endpoint with a support contract behind it, buy that from a specialist in layer one or layer three. Building it on raw compute means signing up to maintain a capability against adversaries who do it full-time, and I would rather tell you that than sell you a disappointing quarter.

What we are is layer four with better properties than a generic VM for this specific workload. Three of them.

Isolation. Every sandbox is a Firecracker microVM with its own guest kernel and its own network namespace — KVM-level separation, not a shared kernel with namespacing on top. A crawler is executing content that other people wrote, and with a browser in the loop it is executing their JavaScript by design. When a renderer is compromised, what it lands in matters: a VM boundary rather than a container next to your other workloads. Because the boundary is a real Linux network namespace beneath the crawl code, egress policy — destination allow-lists, blocking internal and metadata address ranges so a redirect in fetched content cannot pivot into your network, pinned DNS — is enforced somewhere the crawl cannot switch off.

Startup, which changes what designs are affordable. Every create restores a baked snapshot rather than cold-booting, so a create is p50 around 179 milliseconds. That makes one-VM-per-shard, or even one-VM-per-page for genuinely hostile content, a default rather than an extravagance. Forking a warmed parent is 400 to 750 milliseconds on the same host and shares memory copy-on-write, so the browser you warmed up and logged in once becomes the starting state for every worker in the fan-out.

Cost shape. Crawls are bursty. Sandboxes scale to zero, so between crawl windows there is nothing to bill: one rate card at $0.054 per vCPU-hour and $0.0162 per GiB-hour, with CPU billed on the CPU-seconds actually burned rather than on slots reserved. For a nightly window that is a much better fit than a fleet sized for peak and idle for twenty-two hours a day. It is a worse fit than a per-page API if your volume is small, and I would say so: below some threshold, the invoice from a layer-three vendor is smaller than the engineering time you will spend, and the right move is to buy the abstraction and revisit later.

Two more honest limits. We do not run browsers as a service — we give you a machine with a browser template on it, which means you own the browser lifecycle, the memory management and the version treadmill that layer two would have taken off your hands. And the IP address you get is an ordinary datacenter address. Bring your own proxy layer if your targets need one.

Two measurements to take before you buy anything

Both take under an hour and both have changed procurement decisions I have watched.

# 1. How many bytes does a page on YOUR targets actually cost?
#    This is what a per-GB plan bills, and it is not the size of the HTML.
for u in https://target.example/p/1 https://target.example/p/2; do
  curl -s -o /dev/null -w '%{url_effective}  %{size_download} bytes  %{time_total}s\n' "$u"
done

# 2. Do you need a browser at all? Compare the raw HTML against the field
#    you are trying to extract. If it is already there, you have just deleted
#    the most expensive component in the stack.
curl -s -A 'AcmeResearchBot/1.0 (+https://acme.example/bot)' \
     https://target.example/p/1 | grep -c 'data-price'

# And check what the site says about crawling before either of the above.
curl -s https://target.example/robots.txt

The second one is the important one. An enormous amount of money in this category is spent rendering pages whose data was present in the initial HTML the whole time, or available from a JSON endpoint the page itself calls. Server-side rendering came back into fashion and a lot of crawlers never noticed. Check before you buy a browser layer.

Choosing, in about ten minutes

1. Read the target's robots.txt and terms, and decide whether you should be crawling it at all. This step is not a formality; it eliminates some projects entirely and it is much cheaper to do now than after a letter arrives.
2. Fetch three representative pages with plain curl and check whether the data you want is in the HTML. If it is, you do not need layer two and possibly not layer three either, and you have just removed the most expensive component from your architecture.
3. Compute your concurrency requirement: pages times measured latency divided by your time window. Under about ten, optimise for convenience and buy the abstraction. In the hundreds, read the concurrency and overage clauses first and the feature list second.
4. Establish whether you are actually being blocked, from a plain datacenter address, while identifying yourself honestly and rate-limiting politely. Many teams buy layer one before confirming they need it. If you are not blocked, skip that layer and save the per-GB bill.
5. Decide whether your extraction is hot-path or long-tail. Hot path — a dozen sites, high volume — wants deterministic selectors you own. Long tail — hundreds of one-off sources — is where schema-driven LLM extraction earns its cost. Most pipelines want both.
6. Ask where the data is allowed to live. If the answer is only in our account or only in this region, layers one to three narrow sharply and layer four may be forced regardless of the economics.
7. Only then compare vendors, and only within the layers you decided you are buying. Verify every price and limit against current documentation, because this category moves faster than any article about it.

The short version

If you are being blocked and it matters commercially, buy layer one from a specialist and keep your own crawler. If you need a real browser and running browsers is the part that hurts, buy layer two and keep your orchestration. If you want content and not control, buy layer three, and use the LLM-extraction flavour for the long tail while keeping deterministic selectors on the sites you hit constantly. If your volume is large, your logic is custom, or your data cannot leave your account, own layer four — and be honest that you are signing up for the frontier, the retry classification and the browser lifecycle along with it.

Most teams should end up buying one or two of these and owning the rest. A vendor selling you all four is selling you a stack, which is fine, but you should know that is what you are buying and price it against the alternative of assembling it yourself from specialists.

The scraping decision people get wrong is not which vendor. It is buying an unblocking network for a site that would have happily served a polite crawler, and then wondering why the pipeline costs what it does.

And whatever you assemble, spend the extra hour on the boring parts: canonicalise the URLs, classify the failures into four states rather than one, assert on extracted fields instead of on HTTP status, and keep a handful of golden pages as a canary. None of that is a purchasing decision, and all of it determines whether the thing is still producing correct data next spring.