What it takes to build a PaaS on Firecracker

There's a genre of blog post — I've enjoyed most of them — where somebody wires up Firecracker over a weekend, boots a VM in 125 milliseconds, runs a web server inside it, and concludes that building a Heroku is mostly a matter of tidying up. I wrote roughly that post myself once. Then I spent the following two years on the tidying up.

This is the honest inventory of what sits between 'a microVM boots and serves HTTP' and 'a platform other people deploy to'. I run PandaStack, so I've paid for each of these; the point isn't to talk you out of building one, it's to make sure you scope the right thing. Some teams genuinely should build this. Most who start don't finish.

The genuinely easy part

Credit where it's due: Firecracker is a pleasure. A REST API on a Unix socket, a kernel and a rootfs image, a TAP device, and you have a VM. Snapshot and restore are built in. The security model is well documented, the jailer handles the sandboxing of the VMM process itself, and boot times are genuinely sub-second.

If your goal is 'run untrusted code in a VM', you can be done in a week and it will be good. The problems start when the code is a long-lived application belonging to somebody who expects a URL.

The nine things nobody scopes

1. The build pipeline

Users push a repo. Something must decide what it is, what runtime it needs, how to install dependencies, how to build, and how to start. Auto-detection across Node, Python, Go, Ruby, static sites, and the eight ways a monorepo hides its real entrypoint is not hard in the interesting sense — it's hard in the long-tail sense, where every week brings a repo shaped in a way your detector has never seen. You need an override mechanism, a manifest format, and a precedence order between them. Then you need to make the build reproducible, and to stream its logs somewhere users can watch, because a build with no visible output is indistinguishable from a hang.

2. Runtime version management

Users want Node 22, and also Node 18, and also Python 3.12 with a specific patch. You can bake an image per runtime version and drown in a maintenance matrix, or you can install versions on demand with a tool like mise and accept a slower first build. We chose the second — one universal base image, runtimes resolved at deploy time from whatever version file the repo already has. The trap: build commands run in non-login shells, so none of the environment a version manager normally installs into your profile exists, and every command needs it exported explicitly.

3. Routing and TLS

Each app needs a stable URL that survives redeploys even though the VM behind it is replaced every time. That's a router holding app-to-backend mappings, updated atomically at the moment of the flip, plus a wildcard certificate. Then a customer asks for their own domain, and you're implementing domain verification, per-domain certificate issuance and renewal, SNI routing, and — a detail that costs people real money — protection against domain squatting, where one tenant claims a hostname belonging to another and quietly receives their traffic.

4. Health checks that mean something

You cannot flip traffic to a new deploy until the app is ready, so you poll its port. Sounds trivial. Two subtleties bit us. First, probe the address the proxy will actually dial — the guest's network interface — and not localhost inside the guest, or an app bound only to 127.0.0.1 passes the health check and then serves nothing to the outside world. Second, decide what a failed health check means after the flip: restart automatically, and cap the restarts, because unbounded auto-restart turns a broken deploy into a crash loop that runs all weekend and bills for every cycle.

5. State reconciliation

This one is the actual hard problem and it never appears in the weekend post. The control plane believes an app is running on a VM on a host. Reality disagrees: the host rebooted, the VM was killed, the process exited, the deploy died halfway, or the network partitioned during the flip. You need a reconcile loop that continuously compares intent to reality and repairs the difference, and you need it to be idempotent, because it will run concurrently with itself.

Get the ownership model wrong and it does real damage. We shipped a version where every agent's janitor judged every row in a shared table with no ownership column — so during a rolling deploy, one agent cheerfully deleted another agent's live database VM, having concluded it was an orphan. The fix was obvious in hindsight, which is true of every distributed-systems bug after it has eaten something.

6. Image and snapshot distribution

Multi-host means the base image and its baked snapshot must exist on every host, including hosts that joined the fleet ten minutes ago. So you need object storage, versioned generations, a pointer to the current generation, checksums, background sync on boot, and garbage collection of old generations. Then you need the pointer flip to be atomic under concurrent publishers — we ran a mutable pointer with no compare-and-swap and had fleet-wide fan-out races where the pointer named a generation the collector had already deleted, which turned every subsequent pull into a 500.

7. Scheduling and capacity

Which host gets the next app? You need heartbeats, a freshness cutoff so a dead host is excluded rather than scored, a scoring function, leases so two schedulers don't double-place, and affinity rules — a fork wants to land where its parent's memory already is, a volume-bound workload must land where the volume is. And then the part everyone defers: autoscaling the fleet itself. Stateful hosts holding live VMs cannot be scaled by a naive instance-group autoscaler, because scaling in means killing customer workloads. That constraint reshapes the whole architecture, and it is much cheaper to discover in a design doc than in production.

8. Logs, metrics, and metering

Users need build logs, runtime logs, and the ability to tell them apart. You need per-app resource metrics. If you're charging, you need metering that's accurate across restarts, pauses, migrations, and crashes — including the awkward cases where a VM is destroyed before its final usage was recorded, which is a revenue leak, or double-counted across a restart, which is worse because customers notice that one.

9. The guest side

Inside the VM you need an agent for exec, file transfer, and health signalling; a way to inject credentials without baking them into images; log capture; and clock synchronisation on resume, because a snapshot-restored guest wakes with the clock frozen at capture time and will reject valid TLS certificates as not-yet-valid the moment upstream certificates rotate past that date. That's an afternoon to fix and a week to diagnose if you've never seen it.

The tell: what a deploy actually does

Here's the sequence a working platform runs on every push. Each numbered step is a component above, and each has its own failure mode requiring cleanup that leaves no orphans.

push received
  -> verify webhook signature, match branch, dedupe concurrent deploys
  -> mint short-lived clone credential          (never persisted)
  -> schedule a host                            (heartbeat fresh? capacity? lease?)
  -> restore base snapshot -> new VM            (~179ms p50)
  -> clone repo at exact commit
  -> detect framework / read manifest / apply pins
  -> resolve runtime versions, install deps, build   (stream logs)
  -> start process detached, capture stdout/stderr
  -> health-check the guest's network IP, up to 60s
  -> flip router mapping atomically
  -> mark previous deployment superseded, destroy its VM
  -> record usage boundaries for billing

any step fails -> old version still serving, no orphan VM, logs retained
the process dies mid-deploy -> reconcile loop cleans up within one interval

Read the last two lines again, because that's the part that separates a demo from a platform. Every step must be safe to interrupt, and something must notice and clean up when it is.

So should you build it?

Build it if the platform is your product — if you're selling hosting, or per-tenant compute is your core differentiator, or you have hard isolation requirements that no vendor satisfies. Then this list is your roadmap, not a warning, and it's a good business to be in.

Build a deliberately smaller thing if your needs are narrow. A single-host internal deployer for a known set of apps skips scheduling, image distribution, and most of reconciliation. That genuinely is a couple of weeks, and it's a perfectly respectable piece of infrastructure. The mistake is starting there and letting it grow into a multi-tenant platform one urgent request at a time, without ever deciding to build one.

Don't build it if you just want your apps deployed. The nine components above are not incidental complexity you can skip by being smart; they're what a platform is. My rough measure: the VM layer was under 10% of the effort, and the deploy pipeline plus reconciliation was more than half. Firecracker was never the hard part. Firecracker is the part that works.