Running EDA and Chip-Design Workloads in MicroVMs

Most infrastructure arguments are about the cost of compute. Chip design inverts that. The compute for an overnight regression is a rounding error next to what is being processed: an RTL tree that represents years of engineering, a foundry PDK that arrived with a contract specifying who may touch it and on what machine, a netlist that a competitor would pay a great deal to read. The infrastructure question is not "how do I run 4,000 simulations cheaply". It is "how do I run 4,000 simulations such that nobody has to have an uncomfortable conversation with the foundry".

That framing changes which isolation boundary you accept. A container is a polite suggestion to a kernel that every other job on that host is also making suggestions to. For a CI job compiling a Go binary, fine. For a job that has a PDK mounted and a design database open, the shared kernel is the thing your NDA reviewer will ask about, and "we use namespaces" is not the answer they want.

Why the workload shape actually fits microVMs

Set the security argument aside for a moment, because the operational fit is what makes this worth building. EDA verification has a shape that is unusually well matched to a fleet of short-lived VMs, and it is not a coincidence — it is the same shape as a CI farm, just with worse tools and better funding.

• Embarrassingly parallel. A regression suite is thousands of independent simulations that share nothing but the design under test. There is no cross-talk to preserve, which means the scheduler's only job is to place work, not to coordinate it.
• Huge fan-out of short-to-medium jobs. A directed test is seconds. A constrained-random seed is minutes. A block-level synthesis run is tens of minutes. The distribution is wide, but the mode is well under an hour, which is exactly the regime where per-job provisioning overhead starts to dominate if you are booting a full VM per job.
• Every job wants a pristine, identical environment. Simulation results have to be reproducible, and the classic reason they are not is that job 4,001 inherited a stale temp directory, a half-written work library, or an environment variable from job 4,000. A fresh guest per job makes that failure mode structurally impossible rather than a thing you write cleanup scripts for.
• The inputs are read-mostly and the outputs are small. The design, the PDK and the testbench come in; a pass/fail, a coverage database and a waveform go out. The authoritative copy of everything that matters lives outside the guest, which is what makes a guest disposable.
• Failure is normal and cheap. A failing test is the point of the exercise. Retrying it costs a retry, not an apology — so the platform is allowed to lose a job occasionally, which is a very different reliability budget from a database.

The one place the shape works against you is the tail. A full-chip gate-level simulation or a top-level DRC run is neither short nor small, and it will happily eat more memory than your entire fleet has per host. We will get to that, because memory — not CPU — is the constraint that decides what you can and cannot run this way.

Per-project isolation, and the contractor problem

Here is the scenario that makes this concrete. You have contractor A doing verification on project Orion and contractor B doing physical design on project Vega. Both are under NDA, both NDAs are to different parties, and neither is allowed to see the other's design. They are also both running your standard flow on your standard build farm, which means their jobs are two processes on the same Linux kernel, separated by uid, cgroups and the hope that nobody has a local privilege-escalation exploit.

The honest description of that arrangement is that the boundary between two contractual obligations is a kernel you did not write and cannot audit. Every container escape in the last decade has been a kernel bug reached through a shared syscall surface. You do not need to believe an escape is likely; you need to be able to describe the boundary to somebody who will ask what happens if one occurs.

A microVM gives you an answer that fits in a sentence: contractor A's job runs on its own kernel, with its own memory, behind a virtualisation boundary, and the only shared surface is a hypervisor whose device model is deliberately tiny. Firecracker exists precisely because AWS wanted that sentence for Lambda. It is the same argument, and the chip-design version of it has more money riding on it.

Shared build farm vs container-per-job vs microVM-per-job

The three models people actually choose between, compared on the axes that matter for this workload rather than the ones that matter for a web service.

• Blast radius of a runaway job — Shared build farm: one simulation that allocates until the host swaps takes every other job on that machine down with it, and the OOM killer picks its victim by heuristic, not by whose deadline is tomorrow. Container-per-job: cgroup limits cap the memory, so the runaway dies alone, which is a genuine improvement. MicroVM-per-job: the guest has a fixed RAM ceiling it physically cannot exceed, and it dies inside its own kernel — the host never even sees memory pressure from it.
• Separation between two NDAs — Shared build farm: uid separation on one kernel, which is an access-control story, not an isolation story. Container-per-job: namespaces and seccomp on one kernel; better, but the syscall surface is still shared and the escape history is real. MicroVM-per-job: separate kernels, separate memory, a hypervisor boundary you can name in a document.
• Environment drift between runs — Shared build farm: the classic failure. Stale work libraries, leftover temp files, an env var from the previous job, and a reproducibility bug that takes a week to find. Container-per-job: fresh filesystem per job, which solves most of it, though shared kernel state (sysctls, /dev, kernel version) still leaks in. MicroVM-per-job: fresh kernel and fresh filesystem, so the only thing shared between run N and run N+1 is the image you deliberately baked.
• Tool cold start — Shared build farm: amortised, because the tool is already installed and the file cache is warm; this is the model's one real advantage. Container-per-job: image pull is cached, but the tool still initialises from scratch on every job, which for commercial EDA binaries is the dominant cost. MicroVM-per-job: worst case if you boot cold, best case if you snapshot after tool init and restore — the guest wakes up with the tool already loaded.
• Licence seat on a crashed job — Shared build farm: the seat is orphaned until the daemon times it out, and you find out when somebody's synthesis run queues behind a licence held by a process that died an hour ago. Container-per-job: same problem, because the licence daemon does not know or care that a container exited. MicroVM-per-job: same problem again unless you make the release explicit — no isolation model fixes this for you, which is why it gets its own section below.
• Cost of the 400th parallel job — Shared build farm: you bought the machines; the marginal job is free until you hit the wall, and then it is catastrophic. Container-per-job: cheap, milliseconds, this is what containers are good at. MicroVM-per-job: cheap if and only if your create path is a snapshot restore rather than a boot — on PandaStack a create is a snapshot restore at a p50 of 179ms and a p99 of about 203ms, which is close enough to container-cheap that the isolation stops costing you throughput.
• What an auditor can verify — Shared build farm: process listings and file permissions, checked by hand, true at the moment you looked. Container-per-job: image provenance plus a runtime config that is one privileged flag away from meaningless. MicroVM-per-job: a per-job VM boundary and a per-job network namespace, both of which are structural rather than configurational — much easier to argue is still true next quarter.

Snapshot-restore, or: commercial EDA tools take forever to start

Anyone who has run a large regression knows the number that quietly dominates it. The simulation takes ninety seconds. The simulator takes forty to be ready to simulate — licence checkout, library elaboration, reading a design database that was serialised by a tool from 2009, initialising a Tcl interpreter that will be used for four commands. Multiply forty seconds by four thousand jobs and you have spent a day and a half of machine time on startup.

This is the thing microVMs do that containers cannot. You can freeze a running guest — RAM, vCPU registers, device state — and restore it later at exactly the instruction it was on. So the flow inverts: boot one VM, start the tool, let it finish everything expensive and idempotent, and snapshot at the moment it is ready but has not yet been told what to do. Every job then restores that snapshot instead of repeating the initialisation.

On PandaStack this is not an optional optimisation path, it is how create works. Templates are baked snapshots and every create is a restore, which is where the 179ms p50 comes from — the snapshot load itself is around 49ms of that, with the rest going to network setup, a reflink of the rootfs, and probing that the guest is actually up. The first spawn of a template that has never been baked does a genuine cold boot at around 3 seconds and captures the snapshot on the way through, so you pay the boot once per template rather than once per job.

For fan-out from a state that is more specific than the template — a guest that has already loaded your design, not just your tool — forking is the mechanism. A same-host fork lands in 400 to 750 milliseconds; cross-host is 1.2 to 3.5 seconds because the memory image has to come over the network. The child inherits the parent's memory and filesystem, which for a regression sweep means every seed starts from an elaborated design rather than elaborating it again.

The licence server problem, which nobody warns you about

Every serious EDA tool checks out a floating licence over the network from a daemon that lives somewhere in your corporate network. That single fact constrains the architecture more than anything else in this post, and it breaks in two directions.

Direction one: every guest needs a route to the daemon

This is the private-network-attachment problem wearing a different hat. Your sandboxes are isolated by design — on PandaStack each one gets its own network namespace, its own veth pair and its own /30 out of a 16,384-slot pool per host, with NAT to the outside — and isolation is the opposite of "can reach a machine inside your office". Getting a routed path from a per-job guest to a licence daemon on a private network is a real piece of work, and it is the piece most likely to be underestimated when someone sketches this architecture on a whiteboard.

Being straight about our own status here: PandaStack does not ship a first-class private-network attachment today. Sandboxes egress through the host with NAT, so a licence server has to be reachable from the host's egress path — via a tunnel terminated on the host, a licence proxy you expose deliberately, or a self-hosted deployment sitting inside your own network in the first place. If your licence daemon is behind a corporate VPN and nothing else, that is the gap to close before any of the rest of this matters.

There is also a wrinkle specific to how snapshot-restore networking works that is worth understanding before it surprises you. Because the guest's network identity is frozen at bake time, every guest restored from the same template presents the same MAC address and the same guest-side IP, with the host's NAT translating each to a distinct /30. That is what makes the pre-allocated network pool fast, and it has two consequences for licensing. Node-locked licences keyed to a hostid derived from the MAC will see a stable value across restores, which may be exactly what you want or may be a licence-compliance question you should ask your vendor rather than me. And floating licences counted per host will see a large number of checkouts arriving from behind one NAT address with identical guest hostnames — how your vendor's daemon counts that is vendor-specific and you should verify it against their current documentation before you build a farm on an assumption about it.

Direction two: a crashed VM burns a seat until timeout

The failure that will actually cost you is duller. A job dies — OOM, a hung simulation, a host that went away — and the licence it checked out is not released, because the daemon's model of a client is a TCP session plus a heartbeat, and it will not reclaim the seat until the timeout expires. With a shared farm this is annoying. With a fan-out of hundreds of short jobs, a 2% crash rate against a 30-minute reclaim timeout can silently consume your entire licence pool while your queue sits idle waiting for seats that belong to processes that no longer exist.

The fix is not clever. It is making release explicit at every exit path, including the ones you did not plan, and then having a reaper for the cases where even that fails.

#!/bin/sh
# run-job.sh -- runs inside the guest. The point of this script is the trap.
# A licence seat that is not released on the abnormal exit paths is a seat you
# lose for the whole reclaim timeout, and abnormal is the common case here.
set -eu

JOB_ID="$1"
export LM_LICENSE_FILE="27000@licence.internal"

release() {
  status=$?
  # Best-effort explicit release. lmutil ships with FlexNet Publisher; check
  # your vendor's current docs for the exact incantation and whether they
  # permit lmremove from a client at all.
  lmutil lmremove -c "$LM_LICENSE_FILE" vcs "$(whoami)" "$(hostname)" "" \
    >/dev/null 2>&1 || true
  echo "job $JOB_ID exit=$status, licence release attempted" >&2
  exit $status
}
# EXIT alone is not enough: a SIGKILL from the host skips it entirely, which
# is exactly why the host-side TTL below is not optional.
trap release EXIT INT TERM

cd /work
timeout 3600 make -f Makefile.regress "seed=$JOB_ID"

The trap covers the polite exits. It does not cover SIGKILL, and it does not cover a host disappearing, so the second layer belongs outside the guest. Set a TTL on every sandbox so a wedged job reaps itself, and run a periodic sweep that compares checked-out seats against the set of sandboxes that actually exist. The query is unglamorous and it is the thing that keeps your licence pool honest.

import re
from pandastack import Sandbox

# Licence-pool reconciler. Run it every few minutes. It answers one question:
# is the licence daemon holding seats for jobs that no longer exist?
#
# We tag every sandbox with the job id at create time, so the set of live job
# ids is a metadata read rather than a bookkeeping table we have to maintain.
live = {
    sbx.metadata.get("job_id")
    for sbx in Sandbox.list()
    if sbx.metadata.get("kind") == "eda-regression"
}
live.discard(None)

# Ask the daemon what it thinks is checked out. Any control sandbox with a
# route to the licence server will do; this one is long-lived on purpose.
ctl = Sandbox.get("<your-control-sandbox-id>")
out = ctl.exec("lmutil lmstat -a -c 27000@licence.internal", timeout_seconds=60)

# Lines look roughly like: "  bob job-8123 /dev/tty (v2024.03) (host/27000 401)"
# Parse defensively -- lmstat output varies by vendor and version.
for line in out.stdout.splitlines():
    m = re.search(r"\b(job-[0-9a-f]+)\b", line)
    if m and m.group(1) not in live:
        print(f"ORPHANED SEAT: {m.group(1)} -- no sandbox holds this job")

# Two rules that make this loop rarely find anything:
#   1. Every create sets ttl_seconds, so a wedged guest reaps itself.
#   2. The in-guest trap releases on every exit path it can observe.
# This reconciler exists for the paths neither of those can reach.

Memory is the binding constraint, not CPU

People size EDA farms by core count because that is what the licence is sold by and what the spreadsheet has a column for. Then they run a gate-level simulation of a full SoC and discover that the job needs a large multiple of what a typical cloud core comes with, and that no amount of parallelism helps because the design does not fit.

Simulation memory scales with design size and with how much visibility you asked for. Full waveform dumping on a large design is a memory decision disguised as a debug flag. Static timing and physical verification are worse — they hold a graph of the whole design in RAM by construction. CPU is elastic and memory is not, which is why the practical capacity question for a microVM farm is how many guests of size X fit on a host, and the answer is arithmetic, not scheduling cleverness.

There is a Firecracker-specific constraint on top of this that catches people out. A guest's memory size is baked into its snapshot. You cannot restore a 4 GiB snapshot as a 16 GiB guest; the vCPU count and the RAM are properties of the frozen machine, not of the create request. So "different memory tiers" means different baked templates, not a parameter. Our own catalogue reflects that directly: the base template bakes at 4 GiB, the agent and code-interpreter templates at 2 GiB, the browser template at 4 GiB, the managed Postgres template at 1 GiB, all with 8 vCPU of burst capacity. A serious EDA deployment would want its own templates at its own sizes, baked once and restored per job.

Size the template to the ninetieth percentile job and route the tail somewhere else. A farm sized for its largest job is a farm that is mostly idle RAM.

The open-source flow you can actually try today

Everything above is testable without a single commercial licence, which is the good news and the reason this is worth prototyping before you commit. Verilator compiles SystemVerilog to a C++ model and is fast enough that the platform overhead becomes visible, which makes it an honest benchmark. Icarus Verilog handles the event-driven cases Verilator will not. Yosys does synthesis, OpenROAD takes it through place-and-route, and ngspice covers the analogue side. The flow is not equivalent to a commercial one, but the workload shape — fan-out, pristine environments, memory-bound tails — is identical, and that is what you are trying to measure.

#!/bin/bash
# regress.sh -- one job's worth of work, run inside one guest.
# Deliberately boring: lint, simulate one seed, synthesise, emit JSON.
set -euo pipefail

SEED="${1:?usage: regress.sh <seed>}"
OUT=/work/out/"$SEED"
mkdir -p "$OUT"

# 1. Lint first. Catches most of what a long simulation would catch, in a
#    fraction of a second, and a fast fail is worth a lot at 4,000 jobs.
verilator --lint-only -Wall --timing rtl/*.sv --top-module cpu_top

# 2. Build the C++ model, then run this seed against it.
verilator --cc --exe --build --timing -j "$(nproc)" \
  -CFLAGS "-O2" --trace-fst \
  --top-module cpu_top rtl/*.sv tb/sim_main.cpp \
  -o "$OUT/sim" --Mdir "$OUT/obj_dir"

"$OUT/sim" +seed="$SEED" +timeout=200000 --trace-file "$OUT/dump.fst" \
  | tee "$OUT/sim.log"

# 3. Synthesis, as a coarse check that the RTL is still mappable and as a
#    cheap area-regression signal.
yosys -q -p "read_verilog -sv rtl/*.sv; hierarchy -top cpu_top; synth -top cpu_top; stat -json" \
  > "$OUT/synth.json"

# 4. Machine-readable verdict. The host reads this file; nothing else in the
#    guest is authoritative.
python3 - "$OUT" <<'PY'
import json, pathlib, re, sys
out = pathlib.Path(sys.argv[1])
log = (out / "sim.log").read_text()
cells = json.loads((out / "synth.json").read_text())
verdict = {
    "seed": out.name,
    "pass": "UVM_FATAL" not in log and "ASSERT FAILED" not in log,
    "errors": len(re.findall(r"UVM_ERROR", log)),
    "cells": cells["modules"]["\\cpu_top"]["num_cells"],
}
(out / "verdict.json").write_text(json.dumps(verdict))
print(json.dumps(verdict))
PY

Note what that script does not do: it does not clean up after itself, it does not check whether a previous run left artefacts, and it does not defend against a stale work directory. It does not need to, because the guest it runs in did not exist ten seconds ago and will not exist ten seconds after it finishes. That is most of the value of the model in one sentence.

Fanning out a sweep with fork_tree

The naive version of a regression sweep creates N sandboxes, uploads the design to each, and elaborates it N times. That works and it wastes the most expensive part of the job. The better version does the expensive work once in a parent guest and forks the result, so every child starts from a memory image in which the design is already loaded.

import json
from concurrent.futures import ThreadPoolExecutor
from pandastack import Sandbox

PROJECT = "orion"            # per-project isolation starts here
SEEDS = list(range(1, 33))

# 1. One parent guest does the expensive, shared, idempotent work: fetch the
#    design, build the simulator model, warm the caches. Nothing job-specific
#    happens here, and crucially no licence is checked out yet -- a snapshot
#    taken with a live licence session produces children holding a session the
#    daemon has never heard of.
parent = Sandbox.create(
    template="base",
    ttl_seconds=7200,
    metadata={"kind": "eda-regression", "project": PROJECT, "role": "parent"},
)
parent.filesystem.upload("./rtl", "/work/rtl")
parent.filesystem.upload("./tb", "/work/tb")
parent.exec("cd /work && ./build-model.sh", timeout_seconds=900, check=True)

# 2. Fork the elaborated state. fork_tree snapshots the parent ONCE and boots
#    the children in parallel from that snapshot; each child inherits memory
#    and filesystem but gets a fresh network identity. Same-host forks land in
#    400-750ms, cross-host in 1.2-3.5s. The per-call child count is capped
#    (16 at the time of writing), so a wider sweep is a loop over batches.
results = []
for batch_start in range(0, len(SEEDS), 16):
    batch = SEEDS[batch_start:batch_start + 16]
    children = parent.fork_tree(
        count=len(batch),
        metadata={"kind": "eda-regression", "project": PROJECT},
    )

    def run(pair):
        child, seed = pair
        try:
            # The licence is checked out INSIDE the child, after restore, by
            # regress.sh -- which is also where the release trap lives.
            r = child.exec(f"/work/regress.sh {seed}", timeout_seconds=3600)
            if r.exit_code != 0:
                return {"seed": seed, "pass": False, "stderr": r.stderr[-2000:]}
            raw = child.filesystem.read(f"/work/out/{seed}/verdict.json")
            return json.loads(raw)
        except Exception as err:
            return {"seed": seed, "pass": False, "error": str(err)}
        finally:
            child.kill()          # release the RAM before the next batch

    with ThreadPoolExecutor(max_workers=len(batch)) as pool:
        results.extend(pool.map(run, zip(children, batch)))

parent.kill()

failed = [r for r in results if not r.get("pass")]
print(f"{len(results) - len(failed)}/{len(results)} passed")
for r in failed:
    print("  FAIL seed", r["seed"], r.get("error") or r.get("errors"))

Two details in there are load-bearing. The first is that the licence checkout happens in the child after restore, never in the parent before the snapshot — otherwise every child in the batch inherits the same dead TCP session. The second is that `kill()` runs in a `finally`, because the resource you are actually contending for is host memory, and a batch that leaks guests will fail the next batch rather than itself.

The project tag on every sandbox is what makes per-project isolation operational rather than aspirational. It gives you the query you need for the licence reconciler, for cost attribution per project, and for the awkward moment when someone asks you to prove that no Vega job ran on the same host as an Orion job in the last quarter.

What does not work, stated plainly

The parts of this that I would not want you to discover after committing to an architecture.

• No GPUs. PandaStack sandboxes are CPU-only, and there is no device passthrough. Any GPU-accelerated step in your flow — the emulation-adjacent simulators, some newer place-and-route and lithography tooling, ML-based optimisation passes — is out of scope here and needs different infrastructure. If your flow is GPU-bound end to end, this is the wrong tool and I would rather say so now.
• No first-class private-network attachment. Guests egress through the host with NAT. A licence daemon that is only reachable inside a corporate VPN needs a tunnel terminated on the host, a deliberately exposed proxy, or a self-hosted deployment inside your own network. Solve this before anything else, because nothing else in the flow works without it.
• Memory is baked, not requested. Guest RAM and vCPU count are properties of the snapshot. Different sizes mean different templates, baked separately, and re-baking a template invalidates snapshots taken from the old one.
• Very long jobs fit awkwardly. A twelve-hour full-chip run is not what a fan-out fleet of short guests is for. It will work, but you are paying for isolation machinery whose main benefit — cheap per-job provisioning — has been amortised into irrelevance by the job length.
• Shared scratch is your problem. Regression farms usually assume a shared NFS mount for work areas and result databases. Per-job guests with private filesystems do not have that by default, so plan how results get out — object storage per job, or reading a small verdict file back over the API as above — rather than assuming a mount will be there.
• Commercial tool EULAs are not obviously virtualisation-neutral. Some vendors have opinions about running in VMs, about cloud deployment, and about hostid stability across restores. That is a contract question, not a technical one, and the answer is in your agreement rather than in this post.

The order I would build it in

1. Prove the licence path first. One guest, one checkout, one release, verified from the daemon's side. If a single VM cannot reach the licence server and give the seat back cleanly, nothing downstream is worth building.
2. Build a template per project, not per tool. The isolation boundary you care about is the design and the PDK, so bake the project's environment into an image whose contents you can enumerate for an auditor.
3. Snapshot after tool init and before any network session. That is the whole trick for startup cost, and getting the ordering wrong produces a fleet of children holding sessions that do not exist.
4. Measure the memory distribution of your real jobs before choosing a template size. Not the peak, the distribution. Size for the ninetieth percentile and route the tail to a bigger template or a dedicated machine.
5. Set a TTL on every sandbox, and add the orphaned-seat reconciler on day one rather than after the first time your queue stalls behind licences held by dead processes.
6. Tag every guest with project and job id at create time. It costs nothing and it is the difference between being able to answer a compliance question in a query and answering it in a week.
7. Only then optimise the fan-out. Fork from an elaborated parent instead of elaborating N times, and batch the forks so a wide sweep does not try to hold the whole regression in host memory at once.

The summary is that EDA verification is a workload that microVMs suit unusually well and that microVM platforms are usually not built for. The fan-out, the pristine-environment requirement and the disposability of each job are close to the ideal case for snapshot-restore. The licence server, the memory tail and the absence of GPUs are the friction, and none of them are hidden — they are just rarely mentioned by people selling you the fan-out.

If you want to know whether it works for your flow, do not start with the commercial tools. Put Verilator and Yosys in a template, fan out a thousand seeds, and measure where the time actually goes. The answer will be some mix of tool startup, memory pressure and licence queueing, in proportions specific to you — and that measurement is worth more than any architecture diagram, including this one.