Build vs Buy: Rolling Your Own Firecracker Sandbox

Every few months an engineering lead reaches the same conclusion: 'Firecracker is open source and AWS runs Lambda on it, so we'll just run it ourselves.' It's a reasonable thought and it's half right. Firecracker really is a small, fast, well-audited VMM, and getting one microVM to boot on a KVM host is genuinely a weekend. The trap is believing that weekend is the project. It's the tutorial. The project is everything the tutorial doesn't mention — and this post is an honest teardown of that everything, subsystem by subsystem, with the person-months each one actually costs and the specific sharp edge that will find you at 4am.

I'm the founder of PandaStack, which is one of the 'buy' options, so weight this accordingly. I've kept it honest the only way that works: I quote specific numbers only for PandaStack, I give a real 'build if' verdict rather than pretending nobody should ever build, and the whole framing rests on a fact that undercuts the usual vendor pitch — PandaStack's core is open source under Apache-2.0, so 'buy' here also means 'self-host the exact same code.' It's a build-vs-buy question where the buy option isn't a lock-in trap. That changes the math, and I'll be specific about how.

What Firecracker actually hands you (and what it doesn't)

Firecracker's job description is deliberately narrow: it takes a kernel, a rootfs, and a machine config over an HTTP socket, and it boots a microVM with a minimal virtio device model (net, block, vsock, a serial console) under a KVM boundary. It does that superbly. What it explicitly does not do is decide where the VM's network lives, how its disk is cloned, which host it should run on, how you run a command inside it, when it should be reaped, or who gets billed. Those are not oversights — they're out of scope by design, the same way the Linux kernel isn't a Linux distribution.

So 'we'll run Firecracker' is like saying 'we'll run the JVM.' True, and irrelevant to the size of the app you still have to write. The raw boot path looks trivial, which is exactly what makes it seductive:

# The whole "we'll just run Firecracker" demo. Looks like the finish line.
# It is the starting line.

firecracker --api-sock /tmp/fc.sock &

curl --unix-socket /tmp/fc.sock -X PUT 'http://x/boot-source' \
  -d '{"kernel_image_path":"vmlinux","boot_args":"console=ttyS0 reboot=k"}'

curl --unix-socket /tmp/fc.sock -X PUT 'http://x/drives/rootfs' \
  -d '{"drive_id":"rootfs","path_on_host":"rootfs.ext4","is_root_device":true}'

curl --unix-socket /tmp/fc.sock -X PUT 'http://x/network-interfaces/eth0' \
  -d '{"iface_id":"eth0","host_dev_name":"tap0"}'   # ...but who made tap0? in what netns? with what NAT?

curl --unix-socket /tmp/fc.sock -X PUT 'http://x/actions' \
  -d '{"action_type":"InstanceStart"}'

# It boots! You have ONE vm, on ONE host, with a tap0 you hand-made,
# no way to run code in it, no way to clean it up, and no second host.

Every comment in that snippet is a subsystem. Let's walk them, in the order they'll bite you.

Networking: the one that eats a quarter

That `host_dev_name: tap0` in the boot config assumes a TAP device already exists, in some network context, with a route to the world. Firecracker will not make it. For one VM on your laptop you create a TAP by hand and move on. For a multi-tenant platform you now need: a per-sandbox network namespace (so tenant A can't ARP for tenant B or scan your internal Postgres), a veth pair connecting each namespace to the root, an IP allocator that hands out non-overlapping subnets and — this is the part everyone underestimates — reliably reclaims them, and a NAT/egress layer with per-sandbox policy. We wrote the full anatomy in /blog/firecracker-networking-explained; the summary is that doing it cold is about 100ms of `ip netns add` plus `ip link` plus `iptables` per create, which you cannot afford on the hot path, so you also end up building a pre-allocation pool.

Honest cost: 1–2 engineer-months to get isolated per-sandbox networking working, plus an ongoing tax on the allocator/reclaimer because the failure modes are all in the unhappy path you don't hit until production volume.

Snapshots and a portable snapshot store

Firecracker's snapshot/restore is real and it's the feature that makes sub-second creates possible — but the API gives you two files (a memory image and a VM state file) and a hard problem: everything else. You have to decide when to bake a snapshot, capture the guest in a known-good state, store the artifacts somewhere durable, distribute them to every host that might restore them, and — the quiet killer — keep the metadata that ties a snapshot to its exact kernel, rootfs, and machine config from drifting out of sync with the artifacts themselves.

Multi-host makes it worse. A snapshot baked on host A is useless on host B until B has the memory image and the rootfs locally (memory can be streamed; the rootfs must be local for copy-on-write). So you're now running an artifact-distribution system: object storage, a generation/pointer scheme, checksums, and a sync path that runs when a host joins or a template changes. Miss any of it and you get the drift class of bug, where a host restores a stale snapshot against a mismatched rootfs and the guest comes up subtly wrong — or a template re-bake silently invalidates every snapshot on every host and you don't find out until creates start cold-booting.

Honest cost: 2–3 engineer-months for a snapshot store that survives multi-host, more if you want on-demand memory streaming so new hosts don't download multi-gigabyte images before their first restore.

The scheduler you didn't think you needed

One host needs no scheduler. Two hosts need one, and now you're building a small distributed system: a heartbeat protocol so the control plane knows which hosts are alive and how loaded, a scoring function to pick a host for each create, a lease mechanism so a create doesn't land on a host that just died, and affinity rules (a fork wants the parent's host for local CoW; a durable-volume sandbox is pinned to the host holding the volume). None of this is exotic, and all of it is the kind of code that's easy to write and hard to make correct — stale heartbeats, split-brain leases, the thundering-herd when a host drains. It's a quarter of careful work to make 'pick a host' boring.

Honest cost: 1–2 engineer-months for a working multi-host scheduler with health-based exclusion and leases, and it never fully stops asking for attention.

Copy-on-write rootfs (and why fork is hard)

Booting each VM from a full rootfs copy is slow and wastes disk. The real design uses copy-on-write — XFS reflinks or dm-snapshot — so a new sandbox's disk is an O(metadata) clone that shares data blocks with the template until something writes. Getting that right means picking the CoW mechanism, managing the base images, and handling the teardown so you don't leak snapshot volumes (dm-snapshot has its own orphan class). If you want forking — clone a running sandbox, not just boot a fresh one — you also inherit copy-on-write guest memory (MAP_PRIVATE) on top of the disk CoW, and now fork correctness is a two-dimensional problem. We covered the disk side in /blog/copy-on-write-rootfs.

Honest cost: 1–2 engineer-months for CoW rootfs; add another month or two if fork with CoW memory is a product requirement rather than a nice-to-have.

A guest agent for exec and filesystem

Here's a subsystem the boot demo hides entirely: once the VM is up, how do you run a command in it? Firecracker gives you a serial console and a vsock channel, and nothing higher-level. So you write a guest agent — a small init-adjacent process baked into the rootfs — that listens on vsock and exposes exec, streaming stdout/stderr, file read/write, directory listing, and a readiness signal so the control plane knows the guest is actually up (not just that the kernel booted). Then you write the host side that talks to it, handles the VM that hangs mid-exec, streams a PTY for interactive sessions, and survives a snapshot restore reconnecting the vsock. It's a protocol, and protocols are where the long tail of bugs lives.

Honest cost: 2–3 engineer-months for a guest agent plus host client that handles exec, streaming, filesystem, PTY, and readiness robustly — and it's load-bearing, so every rough edge is user-visible.

Security hardening: jailer, seccomp, egress

Firecracker ships a jailer and a default seccomp filter, which is a real head start — but wiring them correctly is on you: chroot and privilege-drop per VM, cgroup limits so one guest can't starve the host, a seccomp profile you understand well enough to not have silently widened, and the egress controls that stop a compromised or prompt-injected guest from exfiltrating or reaching your metadata endpoint. The isolation primitive is strong; assembling it into a defensible posture — and keeping it defensible as you add features — is ongoing security work, not a one-time setup. See /blog/firecracker-jailer-explained and /blog/seccomp-explained for what's actually involved.

Honest cost: 1–2 engineer-months to a solid baseline, then a permanent line item, because 'is our seccomp profile still tight after that feature we shipped' is a question that never stops needing an answer.

Lifecycle, metering, and observability

The unglamorous trio that decides whether you have a platform or a demo. Lifecycle: TTLs, idle detection, a reaper that kills forgotten sandboxes — because orphaned VMs are silent until the bill, and 'why is this host full of VMs nobody remembers creating' is a rite of passage. Metering: if anyone pays for this, you need per-sandbox usage capture (CPU-seconds, memory, creates, egress) that reconciles into billing without double-counting or losing events. Observability: boot-time histograms, per-stage latency, snapshot hit rates, leak counters — without them you're debugging a distributed VM system by vibes.

Honest cost: 2–4 engineer-months across the three, and metering in particular is deceptively deep the moment real money depends on it being correct.

The tally, and the table

Add it up and the 'weekend project' is comfortably 12–18 engineer-months to reach a platform you'd put untrusted, multi-tenant, paid traffic on — and that's before the ongoing operational tax of running KVM hosts, patching, and being on call. None of it is impossible; all of it is real. Here's the same walk as a subsystem-by-subsystem table, with what each one costs to build and what PandaStack gives you instead (its core being Apache-2.0, 'gives you' also means 'code you can read and self-host').

• Networking — Build cost: 1–2 months for per-sandbox netns/veth/tap + IP allocator, plus a permanent orphan-reclaimer to fight pool exhaustion. What PandaStack gives you: NATID, with 16,384 pre-allocated /30 subnets per agent and per-sandbox egress isolation, so a fully isolated network is a ~9ms slot patch on every create instead of ~100ms of cold setup.
• Snapshots — Build cost: 2–3 months for capture, a portable store, multi-host distribution, and metadata that doesn't drift. What PandaStack gives you: baked snapshot-restore on every create with a generation/pointer store and checksums, plus optional UFFD memory streaming so new hosts page vm.mem on demand from object storage instead of downloading it whole.
• Scheduler — Build cost: 1–2 months for heartbeats, scoring, leases, and affinity across hosts. What PandaStack gives you: a deterministic load-spreading scheduler with health-based exclusion, leases, and fork/volume affinity built in.
• CoW rootfs — Build cost: 1–2 months for reflink/dm-snapshot cloning and leak-free teardown; more for CoW memory forking. What PandaStack gives you: XFS-reflink CoW rootfs plus MAP_PRIVATE CoW memory, exposing fork as a first-class primitive (~400–750ms same-host, 1.2–3.5s cross-host).
• Guest agent — Build cost: 2–3 months for a vsock exec/fs/PTY/readiness protocol on both guest and host sides. What PandaStack gives you: a baked-in guest agent with exec (streaming), filesystem ops, PTY, and readiness — the host client already handles the hang/reconnect cases.
• Security — Build cost: 1–2 months to wire jailer + seccomp + cgroups + egress into a defensible posture, then forever. What PandaStack gives you: jailer + seccomp + per-sandbox netns egress as the default substrate, not a bolt-on you assemble.
• Lifecycle + metering + observability — Build cost: 2–4 months for reapers, correct usage metering, and boot/leak/hit-rate telemetry. What PandaStack gives you: idle reaping and TTLs, usage metering wired to billing, and boot-path/streaming metrics out of the box.

The verdict: build if, buy if

This is a genuine trade, not a foregone conclusion. Building your own Firecracker platform is the right call for a real set of teams — here's an honest split.

Build if…

• The sandbox platform IS your product. If you're building the next E2B, the substrate is your moat and owning every layer is the point — buying it would be buying your own competitor's core.
• You have a platform team that will own this for years. Not one heroic quarter — the ongoing on-call, the reclaimer that needs tending, the seccomp reviews. If 'that's Tuesday' is an honest reaction to the sharp-edges list, you can carry it.
• You have an unusual requirement no product exposes. A bespoke device model, an exotic isolation stack, a scheduling policy tied to hardware you own — if your needs genuinely live outside every product's surface, the build is justified.
• Your scale makes the per-unit economics dominate everything else. At extreme, steady volume the amortized cost of owning the whole stack can beat any metered bill — but you should have modeled this against real numbers, not assumed it.

Buy (or self-host the same code) if…

• Sandboxing is a means, not the mission. Your product is the AI agent, the notebook, the CI system — the microVM is infrastructure you need to be reliable and forget about. Twelve-plus engineer-months on undifferentiated substrate is twelve months not spent on what users pay you for.
• You want the control of self-hosting without building it. This is the crux: because PandaStack's core is Apache-2.0, you can run the exact same agent and control plane on your own KVM hosts. You get VPC isolation, data residency, and code your security team can read — without writing the netns reclaimer or the snapshot store yourself. 'Buy' and 'self-host' stop being opposites.
• You don't want to discover the unhappy paths in production. The leaked namespace, the drifted snapshot, the orphaned dm-snapshot volume, the split-brain lease — these are found the hard way. Buying (or adopting the OSS core) means someone already paid for those lessons.
• You'd rather benchmark than build. If your real question is 'is it fast and isolated enough,' that's a week of testing a create/exec/fork API on your workload — not two quarters of building one to find out.

Why PandaStack is the honest buy option

The usual objection to 'buy' is lock-in: you trade build effort for dependence on a vendor's roadmap and prices. PandaStack is structured to defuse exactly that, because the core is open source under Apache-2.0. 'Buy' here means one of two things you choose between — use the hosted API and operate nothing, or self-host the identical binaries (a control-plane API plus a per-host agent) on your own Linux KVM hosts, with the SDK's base URL configurable so the same code points at either. That's the difference between buying a black box and buying a head start on code you can also run yourself.

The substrate is what those 12–18 months would have built, already built. Every sandbox is a Firecracker microVM with its own guest kernel under KVM. There's no warm pool — every create restores a baked snapshot on demand at ~179ms p50 (~203ms p99); the only slow path is the first-ever spawn of a brand-new template, which cold-boots (~3s) and bakes the snapshot for everyone after. Forking is copy-on-write and first-class (~400–750ms same-host, 1.2–3.5s cross-host). Networking is NATID with its 16,384 pre-allocated /30 subnets per agent. Managed PostgreSQL comes up in 30–90s on the same isolation model. The trade-off worth naming: vCPU and RAM are fixed at bake time, so guest size is a template property — you can't resize at restore.

The buy side of all that platform work collapses to about four lines. This is the same code whether you point it at the hosted API or your own self-hosted control plane:

from pandastack import Sandbox

# All 12–18 months of platform work reduced to a create + exec + teardown.
with Sandbox.create(template="base", ttl_seconds=300) as sbx:
    sbx.filesystem.write("/workspace/build.sh", "#!/bin/sh\nmake test\n")
    result = sbx.exec("sh /workspace/build.sh", timeout_seconds=120)
    print(result.exit_code, result.stdout)
# microVM (netns, tap, rootfs, snapshot) is reclaimed here — no orphan reaper of yours involved.

If you're weighing this decision, the two posts to read next are /blog/self-hosted-code-execution-sandbox for when self-hosting is genuinely the right call (and when it isn't), and /blog/firecracker-networking-explained for a concrete look at one of the subsystems you'd otherwise be building from scratch. The honest bottom line: build if the platform is your product and you'll own it for years; otherwise buy the head start — and with an Apache-2.0 core, buying doesn't mean giving up the option to run it all yourself.