Running Per-Tenant Blockchain RPC Nodes in MicroVMs

There is a specific product shape that keeps showing up: you sell blockchain data. Maybe you're a node provider renting dedicated RPC endpoints. Maybe you run an indexer that ingests blocks and serves your customers a nicer API on top. Maybe you're a wallet, an exchange, or an analytics product that decided the public endpoints were too flaky to build on. Whatever the label, the engineering problem underneath is identical: you have to run chain node software — a Geth/Reth/Erigon-class execution client, a consensus client next to it, or a Solana or Bitcoin node — and you have to make it serve more than one customer without those customers destroying each other.

I'm Ajay; I build PandaStack, a Firecracker microVM platform, and blockchain infrastructure teams are a recurring flavor of question in my inbox. The question is almost always the same one, phrased differently: "we run one big node behind an API gateway and it keeps falling over — what's the actual fix?" This post is the long answer. What a chain node actually is as a workload, the three distinct ways the shared-node design fails, why a microVM per tenant is a natural fit, the snapshot trick that makes it economically sane, and the parts where a microVM does absolutely nothing for you. That last section matters most, because chain state is measured in a unit that no amount of clever virtualization deflates.

What a chain node is, as a workload

Strip the crypto off and a full node is one of the least cloud-native workloads in common production use. It is stateful in the strongest sense — the value of the process is entirely in the directory on disk, not in the binary. It is disk-hungry: chain state runs from hundreds of gigabytes to multiple terabytes depending on the chain, the client, and whether you're pruned or archival, and it grows monotonically forever. Check the current client docs before you size anything, because those numbers move every quarter and every number you memorized is already wrong.

It is also chatty in a way normal services aren't. An execution client maintains a live peer set over devp2p with UDP discovery alongside it; a consensus client gossips over libp2p; a Bitcoin node holds long-lived connections on its P2P port; Solana's networking is famously bandwidth-hungry. This traffic is continuous, bidirectional, and completely unrelated to whether any customer is calling your API. Your egress bill has a floor set by peers, not by users.

And its performance profile is bimodal in the worst way. The overwhelming majority of RPC calls are trivial — `eth_blockNumber`, `eth_getBalance`, a receipt lookup — served from cache in a blink. The rest can pin a core and thrash your page cache. There is no middle, so capacity planning against a mean is meaningless: you are planning against a tail that a single customer can summon on demand.

Why "one big node behind a gateway" breaks

The shared design is the obvious one and I don't blame anyone for starting there. One well-provisioned node, an API gateway in front doing auth and rate limiting, tenants multiplexed onto the same upstream. It amortizes the expensive thing — the synced datadir — across everyone. It fails in three distinct ways, and it's worth separating them because they have different fixes.

One tenant's query is everyone's outage

The classic is `eth_getLogs` with a block range that someone typed optimistically — from genesis, say, with a loose topic filter. The node dutifully starts scanning. Every heavy call in the tracing namespace is worse, because `debug_traceTransaction`, `debug_traceBlockByNumber`, and the `trace_*` family don't read state, they re-execute it, allocating aggressively while they do. Point one of those at a block full of complex contract interactions and you've bought a long CPU burn and a page cache that just evicted everything your other tenants were relying on.

Here's the part people miss: your API gateway cannot save you from this. A gateway enforces requests per second. It has no idea that request number four is going to cost ten thousand times more than requests one through three. Once the call is admitted, the gateway is a spectator. You can be well under every published rate limit and still be the reason your platform is paging. Rate limiting the number of expensive calls helps a little, but the unit you actually need to cap is CPU-seconds and memory, and an HTTP proxy does not have that lever.

Tenants don't want the same node

The second failure is that a shared node forces one configuration on everyone. An analytics customer wants full historical state and the tracing namespaces enabled. A wallet backend wants a pruned node, low latency, and would be horrified to learn that a stranger can enable tracing on the machine serving their balance checks. One wants a specific client because its trace format matches their pipeline; another wants a different client for its sync behavior. Some want an older release pinned because a hardfork-adjacent upgrade broke their assumptions and they'd like to schedule that pain themselves.

On a shared node these are irreconcilable, so you end up running the union of everyone's requirements — archive mode with every namespace exposed, because someone needs it. Every tenant then pays the archive storage cost, and every tenant is one gateway misconfiguration away from the expensive namespaces. Per-tenant method allowlists are the right idea, but implemented at a shared upstream they are a filter in front of one machine that is still, physically, capable of everything.

Blast radius: reorgs and corrupted datadirs

The third failure is the one that actually wakes you up. A deep chain reorganization arrives and the node spends a while reorganizing state — during which every tenant sees stale or inconsistent answers, and every indexer downstream has to unwind and replay. A client bug or an unclean shutdown corrupts the embedded key-value store, and the datadir is now a very large paperweight. A memory leak in a release you upgraded to on a Friday takes the process down at 3am.

In all three cases the shared design converts one machine's bad day into a platform-wide incident, and the recovery path is the worst part: re-syncing is not a restart, it's an expedition. Every customer is down for the duration. You cannot roll one tenant back to a known-good state, because there is no "one tenant" — there is one datadir and everyone is in it.

# The shared-node approach, done as carefully as it can be done.
# Namespaces are the union of every tenant's needs -- so `debug` is on,
# because ONE analytics customer asked for it, for EVERYONE forever.
geth \
  --datadir /var/lib/geth \
  --http --http.addr 127.0.0.1 --http.port 8545 \
  --http.api eth,net,web3,debug \
  --gcmode archive          # archive because one tenant needs history;
                            # every other tenant pays for the disk

# ...and the gateway in front, doing what a gateway can do:
#   limit_req_zone $tenant_key zone=rpc:10m rate=50r/s;
#
# It counts REQUESTS. It cannot count CPU-seconds. This request is
# request #1 of 50 this second, perfectly within policy, and it will
# occupy a core while your other tenants time out:
curl -sS -X POST http://127.0.0.1:8545 \
  -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"eth_getLogs","params":[{
        "fromBlock":"0x0","toBlock":"latest",
        "topics":["0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef"]
      }]}'

# Verify the flags above against the current client docs before copying --
# pruning and namespace flags drift between releases and between clients.

Why a microVM per tenant fits this shape

A microVM is a real virtual machine — its own guest kernel, isolated by CPU hardware virtualization through KVM — stripped down to a minimal device model so it boots fast and costs little. For a chain node, four properties line up almost suspiciously well with the three failures above.

• Hard CPU and memory ceilings. vCPU count and guest RAM are fixed at the VM boundary by the hypervisor, not requested politely by a cgroup you hope is configured. A tenant's genesis-to-latest `eth_getLogs` saturates their own allocation and stops there. The noisy-neighbor problem becomes the tenant's own problem, which is exactly where it belongs.
• Its own kernel and page cache. This is the underrated one. The reason a heavy trace call hurts everyone on a shared box is cache eviction, and page cache is a kernel-level resource. Separate kernels means separate caches — one tenant's full-history scan cannot evict another tenant's hot state.
• Its own disk. Each tenant gets their own datadir on their own block device. A corrupted key-value store is one customer's incident and one customer's restore. It also means per-tenant pruning: the wallet tenant runs pruned and small, the analytics tenant runs archival and expensive, and nobody subsidizes anybody.
• Its own network namespace. P2P peering, discovery traffic, and egress are attributable and enforceable per tenant at the host, not guessed at from application logs. If you want a tenant's node to peer but never reach your internal network, that's a host-side rule about their namespace rather than a firewall exception you hope holds. The general mechanics are in /blog/controlling-network-egress-untrusted-code.

The method allowlist story also improves in a way that's easy to undersell. On a shared node, "tenant B may not call `debug_*`" is a rule enforced by a proxy in front of a node that has `debug` enabled. Per-tenant, it's a launch flag: their node was never started with the namespace at all. The difference between a filter and an absence is the difference between a bug and a non-event.

• Noisy-neighbor blast — Shared node + gateway: one admitted heavy call burns shared CPU and evicts the shared page cache; every tenant degrades. MicroVM per tenant: the call saturates that tenant's own vCPU and RAM ceiling and their own page cache; neighbors don't notice.
• Per-tenant configuration — Shared node + gateway: one client, one pruning mode, one namespace set, chosen as the union of everyone's needs. MicroVM per tenant: client, version, pruning mode, and enabled namespaces are per-tenant launch parameters.
• Method restriction — Shared node + gateway: a proxy filter in front of a node that is fully capable of the restricted methods. MicroVM per tenant: the namespace was never enabled in that guest, so there is nothing to filter.
• Corrupted datadir — Shared node + gateway: one datadir, everyone down, re-sync for the duration. MicroVM per tenant: one tenant's disk, one tenant's restore, everyone else untouched.
• Egress and P2P accounting — Shared node + gateway: peer traffic is one blended bill you allocate by guesswork. MicroVM per tenant: traffic is per-namespace and host-enforced, so it is measurable and cappable per customer.
• Cost profile — Shared node + gateway: one expensive datadir amortized across all tenants; genuinely cheaper at low tenant counts. MicroVM per tenant: N datadirs and N running processes; you pay for isolation in storage and RAM, and that is the real trade.

The expensive thing is the synced datadir, not the VM

Here's the reframe that makes per-tenant nodes practical. Booting a machine is cheap. Installing a client is cheap. The expensive, slow, unglamorous thing is getting a node from nothing to "synced to head," which is a long, network-bound, disk-bound process that you do not want to repeat once per customer. If provisioning a new tenant means syncing from genesis, per-tenant nodes are economically dead on arrival and the shared node wins on cost alone.

So don't re-sync. Sync once, then snapshot the warm machine — memory and disk together — and provision every subsequent tenant by restoring that snapshot. On PandaStack this is the default create path rather than a special mode: a sandbox is created by restoring a pre-baked snapshot at p50 179ms and p99 203ms, with the restore step itself around 49ms. Only the first-ever cold boot of a template takes roughly 3 seconds. The restored guest comes back with its process already running and its caches already warm, which for a chain node is the difference between "ready" and "ready in an hour."

The honest caveat is that a restored node is behind head by however long the snapshot has been sitting. It has to catch up from the snapshot height, which is dramatically cheaper than syncing from genesis but is not zero. The operational shape that follows: re-bake your baseline snapshot on a schedule so the catch-up gap stays small, and treat the gap as a first-class readiness signal rather than assuming a restored VM is immediately serviceable. Long-lived per-tenant nodes should also run on persistent volumes, with the snapshot used for provisioning and disaster recovery rather than as the everyday storage layer.

from pandastack import Sandbox
import json, time

WARM_SNAPSHOT = "snap_geth_mainnet_baseline"   # re-baked nightly by a cron job


def provision_tenant_node(tenant_id: str, plan: dict) -> dict:
    """Give one tenant their own node VM, forked from a warm-synced baseline."""
    # Same-host fork of the warm baseline: ~400-750ms, and the datadir is a
    # copy-on-write clone rather than a copy. Cross-host lands at 1.2-3.5s.
    with Sandbox.fork(WARM_SNAPSHOT, ttl_seconds=3600,
                      metadata={"tenant": tenant_id, "kind": "rpc-node"}) as sbx:

        # Per-tenant node config. The allowlist is a LAUNCH FLAG here, not a
        # proxy filter -- a tenant without `debug` has no debug namespace to
        # call, because their client was never started with it.
        cfg = {
            "http.api": ",".join(plan["namespaces"]),   # e.g. ["eth","net","web3"]
            "gcmode": plan["gcmode"],                   # "full" or "archive"
            "maxpeers": plan["maxpeers"],
        }
        sbx.filesystem.write("/etc/pandastack/node.json", json.dumps(cfg))

        started = sbx.exec("systemctl restart geth-tenant", timeout_seconds=60)
        if started.exit_code != 0:
            return {"ok": False, "stage": "start", "stderr": started.stderr[-4000:]}

        # A restored node is behind head by however stale the snapshot is.
        # Catch-up is cheap compared to genesis, but it is NOT zero -- gate
        # readiness on it instead of assuming a fast restore means "synced".
        for _ in range(60):
            probe = sbx.exec(
                "curl -sS -X POST http://127.0.0.1:8545 "
                "-H 'Content-Type: application/json' "
                "-d '{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"eth_syncing\",\"params\":[]}'",
                timeout_seconds=10,
            )
            if probe.exit_code == 0 and json.loads(probe.stdout).get("result") is False:
                break            # eth_syncing == false means caught up to head
            time.sleep(5)
        else:
            return {"ok": False, "stage": "catchup", "hint": "re-bake the baseline"}

        head = json.loads(sbx.filesystem.read("/var/lib/node/head.json"))
        return {"ok": True, "tenant": tenant_id, "head": head}
    # The `with` block kills the VM on exit. For a real tenant node you would
    # hold the handle instead -- and call sbx.kill() only on deprovision.

Copy-on-write is what makes "your own chain" cheap

The second thing snapshots buy you is more interesting than fast provisioning, and it's the feature your customers will actually ask for by name: give this tenant their own fork of chain state. A testnet that starts from real mainnet data. A simulation environment where they can replay a proposed transaction bundle against live state without touching anything. A staging indexer running against a frozen height so their diffs are reproducible.

Done naively, that's a full copy of the datadir per fork, which is absurd at chain-state sizes. Done with copy-on-write, the fork shares the parent's blocks until someone writes, and only the divergence costs disk. A fork lands at 400–750ms on the same host — where the reflink is local — and 1.2–3.5s cross-host, where the artifacts have to move first. The mechanics of reflink versus device-mapper CoW are in /blog/dm-snapshot-vs-reflink-cow, and the general model is in /blog/snapshot-and-fork-explained.

This is the same primitive that makes forked-chain contract simulation practical, covered separately in /blog/microvm-smart-contract-simulation-isolation. The difference here is duration: a simulation fork lives for seconds, while a tenant's private testnet fork might live for weeks and accumulate enough divergence that its disk cost creeps toward a full copy. Plan for that rather than being surprised by it.

What a microVM does not fix

Now the part that should temper all of the above, because I would rather you skip this architecture than adopt it and be angry at me in a quarter.

A microVM does not make chain state small. Archive data for a busy chain runs into the terabytes and grows forever. Isolation is a compute and boundary property; it does nothing to storage volume, and per-tenant isolation actively multiplies your storage bill unless CoW sharing is doing real work for you. If every tenant needs a genuinely independent archive dataset, you are buying N archives, and no hypervisor feature changes that arithmetic. Storage is where the money goes, and it's the number to model before you write any code.

Sync time still dominates first provisioning. Snapshot-restore is fast, but the snapshot had to come from somewhere, and that somewhere was a node that synced the slow way. You've moved the cost from per-tenant to per-baseline, which is a large win, not an elimination. And restored nodes always have a catch-up window proportional to snapshot staleness — the fix is disciplined re-baking, which is now an operational chore you own and should automate on day one.

#!/usr/bin/env bash
# nightly-rebake.sh -- keep the baseline snapshot close to head, so every
# tenant provisioned tomorrow has a short catch-up instead of a long one.
set -euo pipefail

BASELINE_ID="${BASELINE_SANDBOX_ID:?set me}"   # the long-lived syncing node
API="https://api.pandastack.ai/v1"
AUTH=(-H "Authorization: Bearer $PANDASTACK_API_KEY")

# 1. Refuse to bake a node that is behind. A snapshot of a lagging node is
#    a lagging snapshot, and you will not notice until a tenant complains.
syncing=$(curl -sS -X POST "$API/sandboxes/$BASELINE_ID/exec" "${AUTH[@]}" \
  -H 'Content-Type: application/json' \
  -d '{"command":"curl -sS -X POST http://127.0.0.1:8545 -H '\''Content-Type: application/json'\'' -d '\''{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"eth_syncing\",\"params\":[]}'\''","timeout_seconds":15}' \
  | jq -r '.stdout | fromjson | .result')

if [ "$syncing" != "false" ]; then
  echo "baseline is still syncing ($syncing) -- skipping bake" >&2
  exit 0
fi

# 2. Snapshot captures memory AND disk, so the restored guest comes back
#    with the client running and its caches warm -- not booting from zero.
curl -sS -X POST "$API/sandboxes/$BASELINE_ID/snapshots" "${AUTH[@]}" \
  -H 'Content-Type: application/json' \
  -d '{"label": "geth-mainnet-baseline"}' | jq -r '.id'

# The baseline node keeps running and keeps syncing. Tonight's snapshot just
# became tomorrow's provisioning path.

Persistent volumes and host pinning matter more than they do for stateless workloads. A node with terabytes of state is not freely reschedulable — moving it means moving the data, and cross-host operations land in that 1.2–3.5s class only because the artifacts are far smaller than a full chain archive. Nodes are pets. A microVM makes them well-fenced pets with a clean lifecycle; it does not make them cattle. The same tension shows up for managed databases, and I've written about the density-versus-isolation trade in /blog/per-tenant-database-isolation.

Finally, idle cost is real. A per-tenant node runs continuously whether or not the tenant is calling it, because it has to stay at head. There is no scale-to-zero for something whose entire job is keeping up with a chain that does not pause for your billing model. The hybrid most teams land on is honest about this: shared nodes for the long tail of small, cheap, read-mostly customers, and dedicated per-tenant VMs for the tier that needs archive access, tracing namespaces, custom clients, forked state, or a contractual isolation guarantee. Sell the isolated tier at a price that reflects a standing machine, because that's what it is.

The shape that works

Put it together and the design is unremarkable, which is how you know it's right. Sync a baseline node once and snapshot it warm. Provision each isolated tenant by restoring or forking that snapshot into their own microVM — own kernel, own page cache, own disk, own network namespace — with their client, pruning mode, peer count, and enabled namespaces as launch parameters rather than proxy rules. Gate readiness on actual sync status, not on the VM being up. Re-bake the baseline on a schedule so catch-up stays short. Keep the long tail on shared infrastructure and don't pretend otherwise.

What you get is that the failure modes stop being platform events. A tenant's genesis-to-head log scan is their own CPU ceiling and their own cache. A corrupted datadir is one restore. A bad client release is one tenant's rollback, on a schedule they picked. And "can I get a private fork of mainnet state?" — the request that used to mean provisioning an entire second archive — becomes a copy-on-write fork measured in hundreds of milliseconds. The chain is still enormous, the sync is still slow, and none of that is going away. But at least the blast radius stops being everybody.