Running Firecracker Under Nomad

There is a particular kind of team that ends up asking this question, and I have a lot of time for them. They run HashiCorp Nomad. It works. Nobody has been paged about the scheduler in a year. They have three or four workloads — customer-supplied build steps, an AI agent that runs generated shell, a plugin runtime, a per-tenant analytics job — where a container's shared kernel has stopped feeling like an acceptable boundary. And they would very much like to solve that without adopting Kubernetes, hiring a platform team, or writing a scheduler from scratch.

I'm Ajay. I build PandaStack, a Firecracker microVM platform, so read this as opinionated — I sell the thing at the far end of this post's spectrum. I am going to try to earn your trust the only way that works here: by spending most of the post on the list of things Nomad will not do for you, in enough detail that you could go build them yourself, and by being explicit that for a lot of teams building them yourself is the correct answer.

Why Nomad is a genuinely reasonable host for this

Most schedulers assume containers and then bolt on an escape hatch. Nomad is the opposite: its core abstraction is a task driver, and containers are one implementation of it. Out of the box it runs Docker, raw executables, Java jars, and QEMU images, and it treats all of them as equally legitimate. So when you say "I want to schedule a process that happens to be a virtual machine monitor," you are not fighting the design. You are using it as intended.

That matters more than it sounds. On Kubernetes, running a VM means either the CRI runtime layer (Kata, and now you own a runtimeClass and a containerd shim) or a whole second control plane (KubeVirt, and now you own CRDs, virt-launcher pods, and a live-migration story you did not ask for). On Nomad, a Firecracker driver is a Go plugin binary in a directory, and the job file that uses it looks like every other job file your team already reads.

The second reason is temperament. Nomad is one binary that is both server and client depending on a flag. No etcd to operate, no CNI-plus-CSI-plus-ingress-controller matrix, no six-month upgrade treadmill. That is the same instinct that draws people to Firecracker — a 50-ish-thousand-line VMM that does one job and refuses to grow a device model. Teams who like one tend to like the other, and there is something to be said for an infrastructure stack you can hold entirely in your head at 3am.

The practical on-ramp is the community Firecracker task driver, which has existed in various forms for several years. Qualitatively: it is a Go plugin implementing Nomad's driver interface, it launches a Firecracker process per allocation with vCPU, memory, kernel image, and rootfs taken from the job's config stanza, it typically delegates networking to CNI, and it reports the guest back to Nomad as a running task. That is a real, working starting point and it will get you a booting VM inside an hour. It is also a community project rather than a HashiCorp-supported one, so before you build on it, go read its commit history, its open issues, and — critically — which Nomad plugin API version it targets, because Nomad's driver interface is versioned and a driver built against an older one will not load.

What the job file actually looks like

Here is a realistic job. Note the two constraints at the top, which are the first thing people forget: Firecracker needs /dev/kvm, and half of your Nomad fleet probably does not have it. If you do not constrain placement, the scheduler will cheerfully put a microVM on a t3.micro with no nested virtualization and you will get a driver error that reads like a kernel problem.

job "agent-sandbox" {
  datacenters = ["dc1"]
  type        = "service"

  group "vm" {
    count = 3

    # Firecracker needs /dev/kvm. Tag your bare-metal or nested-virt-capable
    # hosts with a node class and constrain to it, or the scheduler will place
    # this on a box that physically cannot run it.
    constraint {
      attribute = "${attr.kernel.name}"
      value     = "linux"
    }
    constraint {
      attribute = "${node.class}"
      value     = "kvm"
    }

    network {
      mode = "host"
      port "guest_ssh" { static = 2222 }
    }

    restart {
      attempts = 2
      interval = "5m"
      delay    = "15s"
      mode     = "fail"   # give up and let the scheduler reschedule
    }

    task "firecracker" {
      # Community plugin, dropped in the client's plugin_dir. Config keys track
      # the driver's own schema -- check the repo, they have changed over time.
      driver = "firecracker-task-driver"

      config {
        Vcpu        = 2
        Mem         = 2048
        KernelImage = "/opt/fc/vmlinux-5.10"
        BootDisk    = "/opt/fc/rootfs/agent-base.ext4"
        BootOptions = "console=ttyS0 reboot=k panic=1 pci=off"
        Network     = "fcnet"                 # a CNI conflist name on the host
        Firecracker = "/usr/bin/firecracker"
      }

      # This is Nomad's accounting, NOT the guest's. Firecracker sizes the guest
      # from Vcpu/Mem above; these numbers only tell the bin-packer how much of
      # the node to spend. Keep them in sync by hand or you will overcommit.
      resources {
        cpu    = 2000
        memory = 2048
      }

      service {
        name = "agent-sandbox"
        port = "guest_ssh"

        # Nomad is probing a TCP port. sshd answering does not mean the guest
        # is healthy -- see the lifecycle section below.
        check {
          type     = "tcp"
          interval = "10s"
          timeout  = "2s"
        }
      }
    }
  }
}

Run "nomad job run agent-sandbox.nomad" and, assuming your driver loaded and your CNI config is sane, you have three hardware-isolated Linux guests placed across your fleet, restarted on failure, and registered in Consul. That is a real result for twenty minutes of work, and it is worth pausing to appreciate before I spend the next two thousand words explaining what is missing.

What Nomad actually gives you

• Placement and bin-packing — a real scheduler that knows node capacity, respects constraints and affinities, and spreads or packs according to your scheduler config. This is the hardest thing on this list to write yourself and Nomad hands it to you for free.
• Restarts and rescheduling — a task that dies gets restarted locally per your restart stanza, and if the node itself goes away, the allocation is rescheduled elsewhere. Node draining works.
• Service registration and health — Consul integration, TCP/HTTP checks, and a name your other services can resolve.
• Resource accounting — CPU shares and memory limits enforced with cgroups on the VMM process, plus the bin-packing arithmetic that keeps you from oversubscribing a node into swap.
• Secrets and templating — Vault integration and the template stanza, which is genuinely excellent and which you will end up using to render per-VM config files onto the host before the guest boots.
• A UI, an API, and job versioning — deploy, roll back, inspect, all of it, and your team already knows how to use it.

That is not nothing. It is, in fact, the entire top half of a microVM platform. The problem is that a microVM platform is mostly bottom half.

What Nomad does not give you, in the order it will hurt

This is the section that is worth your time. Every item below is something that either does not exist in Nomad's model or exists in a shape that does not fit a VM. None of them is impossible. All of them are yours.

Gap one: rootfs supply, and the copy you are about to pay for

Nomad's artifact stanza fetches a file. It will go get your rootfs from S3 or an HTTP endpoint, verify a checksum, and drop it in the allocation directory. What it will not do — because the concept does not exist anywhere in Nomad — is clone it copy-on-write.

This is the difference between a 4 GB read-and-write per VM start and a metadata operation. Firecracker mounts a rootfs read-write by default, so two VMs cannot share one file; each needs its own writable disk. If you naively cp the base image per allocation, a start that should take milliseconds takes as long as your disk takes to move four gigabytes, times however many VMs you just scheduled at once, on a node whose page cache you have now destroyed. Congratulations, your fast microVM platform is IO-bound at launch.

The fix is reflink on XFS or Btrfs, or a dm-snapshot device per VM on filesystems that will not reflink. Either turns the clone into an O(metadata) operation where blocks are shared until written. Neither is something Nomad knows about, so it lives in a prestart task, a driver fork, or a wrapper script. And once you own it, you also own the cleanup: an orphaned dm device after a hard node reboot is a fun morning.

Gap two: networking, and why cold setup is a tax you pay every time

Each microVM wants its own network namespace, a tap device inside it, a veth pair to the host, and NAT rules — created atomically at start and, more importantly, torn down atomically at stop. CNI gets you most of the way; the Firecracker driver typically leans on it, and that is the right call.

The tax is that doing it cold costs real time. ip netns add, ip link add, moving the tap in, plumbing iptables — that is a syscall-heavy sequence measured in tens of milliseconds, which is an absurd thing to pay in front of a boot you have spent months optimising. The fix is to stop doing it per start: pre-build a pool of ready namespaces at agent startup and hand one out at create time, so the fast path is patching a MAC address rather than building a network. PandaStack pre-allocates 16,384 /30 subnets per agent for exactly this reason, one per possible sandbox slot, and allocation from the pool is a lookup rather than a syscall storm.

There is no place in Nomad's model for a warm pool of network namespaces. Allocations are the unit, and an allocation's network is created for it and destroyed with it. You can pre-warm outside Nomad and have the driver claim from your pool, but now the pool is a piece of state Nomad does not know about, which means it is a piece of state that can leak when the node reboots mid-allocation.

Gap three: Nomad restarts a task by starting it again

This is the deepest mismatch, and it is worth being precise about. When a task fails, Nomad's recovery is to run it again from the top. For a stateless HTTP server that is exactly right. For a microVM it means a cold boot: kernel init, systemd or your init, the runtime warming up, your process reaching ready. That is roughly three seconds of territory in our measurements for a first cold boot with no snapshot — and three seconds is fine for a long-lived service and completely unacceptable as a per-request or per-task latency.

The thing that makes microVMs feel fast is snapshot-restore: boot the guest once at bake time, snapshot memory and disk, and start every subsequent VM by restoring that snapshot rather than booting. Our restore path is about 49ms for the load step and 179ms p50, 203ms p99 end to end for a full create including networking, disk clone, and readiness probe. That is a different regime — the difference between "schedule a machine" and "call a function."

Nomad has no notion of it. There is no snapshot verb, no artifact type for a memory image, no lifecycle hook that says "restore instead of start." If you want restore semantics, you own the snapshot store, the bake pipeline that produces snapshots per template, the distribution of multi-gigabyte memory files to every node that might need one, and — the part everyone underestimates — invalidation, because rebaking a template silently orphans every snapshot taken from the old one and the failure mode is a guest that restores into a world that no longer matches its assumptions.

Gap four: Nomad talks to processes, and your process is not the workload

nomad alloc exec runs a command in your task. Your task is the Firecracker process. Exec'ing into it gets you a shell next to the VMM on the host — which is precisely the machine you were trying to keep the workload away from. The workload is behind a hypervisor boundary, and reaching it needs a channel you build: vsock with a small guest agent on the other side, or sshd in the guest with a key injected at create time.

So the whole exec surface is yours. Run a command and stream stdout. Read and write files. Get a PTY for an interactive terminal. Know when the guest is actually ready to accept work rather than merely powered on. We ship a guest init that speaks vsock and, in parallel, an SSH bridge with ed25519 keys injected per sandbox, because vsock is faster and SSH is more forgiving when vsock does something surprising during a concurrent restore. Two paths, because in production one of them will occasionally not be there.

Gap five: a healthy VMM with a wedged guest looks fine

Nomad's signals and health checks target a process. A SIGTERM to Firecracker does not gracefully shut down the guest — it kills the hypervisor out from under a running Linux, which is the moral equivalent of yanking the power cable. If the guest was mid-write to a durable volume, you now have a filesystem to think about. Graceful shutdown means asking the guest to shut itself down, over your channel, and then waiting, and then killing the VMM — a sequence you write and a timeout you tune.

The health story is worse in a subtler way. The TCP check in the job above proves that something is listening. Firecracker is a well-behaved process; it will sit there at 0% CPU, its check passing, for as long as you let it, while the guest's init has panicked, or the guest is out of memory and the OOM killer is grinding, or the application deadlocked twenty minutes ago. The VMM's health and the guest's health are simply different quantities, and Nomad can only see the one that does not matter. Real health means an in-guest probe reported back over your channel, which means the guest agent again, which means you are building the thing anyway.

The glue nobody hands you

To make gaps one and two concrete, here is roughly the work that has to happen between "Nomad decided to place this allocation" and "Firecracker can be exec'd." A driver may do some of this for you; the community one leans on CNI for the network half. The disk half is generally yours.

#!/usr/bin/env bash
# Prestart glue: what the scheduler does NOT do for you.
# Runs on the client node before firecracker is exec'd.
set -euo pipefail

ALLOC="${NOMAD_ALLOC_ID:?}"
ID="${ALLOC:0:8}"
BASE=/opt/fc/rootfs/agent-base.ext4
RUN=/var/lib/fc/$ALLOC

mkdir -p "$RUN"

# 1. COPY-ON-WRITE ROOTFS.
#    Nomad's artifact stanza fetches files; it has no concept of a clone.
#    On XFS/Btrfs this is O(metadata) and shares blocks until written.
#    A plain "cp" here is a full 4 GB read+write per VM start, per node,
#    and it will quietly become the slowest thing in your platform.
cp --reflink=always "$BASE" "$RUN/rootfs.ext4"

# 2. PER-VM NETWORK.
#    Cold path: five syscall-heavy commands, and it is not cheap.
#    In production you pre-build a pool of these at agent start and
#    allocate from it, so the hot path is patching a MAC, not building
#    a namespace. Nomad has nowhere to keep that pool.
ip netns add "ns-$ID"
ip link add "vh-$ID" type veth peer name "vg-$ID"
ip link set "vg-$ID" netns "ns-$ID"
ip netns exec "ns-$ID" ip tuntap add tap0 mode tap
ip netns exec "ns-$ID" ip addr add 10.200.0.1/30 dev tap0
ip netns exec "ns-$ID" ip link set tap0 up
iptables -t nat -A POSTROUTING -s 10.200.0.0/30 -j MASQUERADE

# 3. TEARDOWN IS THE PART THAT ROTS.
#    Every resource above outlives a SIGKILLed VMM. If the node reboots
#    between step 2 and the driver's cleanup, you leak a netns, a veth,
#    a tap, an iptables rule and a reflinked file -- and nothing in Nomad
#    knows they exist, so nothing in Nomad will ever reap them. You need
#    a reconciler that walks the host and deletes orphans. Write it early;
#    you will not enjoy writing it during an incident.
trap 'ip netns del "ns-$ID" 2>/dev/null || true; rm -rf "$RUN"' EXIT

Nomad schedules the allocation. Everything between the allocation and a usable guest — the clone, the namespace, the snapshot, the channel, the reconciler that cleans up after all four — is a platform, and you are now writing one.

The stateful problem: volumes pin, schedulers reschedule

The gap list above is tractable. This one is a genuine architectural tension, and it is where I have seen Nomad-plus-Firecracker designs actually come apart.

A microVM with a durable volume is pinned to the host holding that volume. A scheduler's entire value proposition is that it can put work anywhere. Those two facts are in direct conflict, and the conflict does not resolve — it just moves to wherever you handle it.

Nomad has answers. Host volumes let you declare a path on specific clients and constrain jobs to nodes that have it, which is the simple and honest version: your allocation now goes exactly one place. CSI volumes get you network-attached storage that can follow a workload, which is genuinely better, at the cost of running a CSI plugin and accepting network-attached IO for something that was fast partly because its disk was local. Neither makes the pin go away; they choose where you feel it.

The failure this produces is specific and worth naming. Node goes down. Nomad, doing its job, reschedules the allocation. With a host volume, it cannot place it and the workload sits pending — correct behaviour, alarming dashboard. With CSI, it places it somewhere and now you are one detach-timeout away from two allocations believing they own the same volume, which for a database is the bad kind of interesting. We hit our own version of this: our managed Postgres VMs are pinned to their host and exempt from the idle reaper, and getting them off a dead host means a deliberate rebuild from an object-store archive, not a reschedule. That was not a design we chose so much as one the physics chose for us.

The practical advice: if your microVM workloads are ephemeral — build steps, agent runs, per-request sandboxes — Nomad's model fits well and you should stop worrying. If they are stateful, decide early whether the state lives inside the VM (accept pinning, use host volumes, constrain hard) or outside it (put Postgres and object storage elsewhere and keep the guest disposable). The design that hurts is the one where state is inside the VM and you kept believing the scheduler could move it.

The four honest options

Here is the spectrum, weighed on operational weight, cold-start story, and what you must build yourself. I am not neutral — I am the last row — so I have tried to write the other three the way their advocates would.

• systemd units, one per VM — Operational weight: lowest; a unit file, a template, and systemctl. No scheduler, no cluster state, no plugin API to track. Cold-start: whatever your boot takes, every time, unless you build snapshot-restore yourself. You build: placement (i.e. you decide by hand which host), rootfs cloning, networking, snapshots, guest comms, and cleanup. Honestly the right answer for a fixed set of long-lived VMs on a handful of known hosts, and it is embarrassing how far it goes.
• Nomad plus a Firecracker task driver — Operational weight: low if you already run Nomad, moderate if you do not; one binary, plus a community plugin whose maintenance and API compatibility you must verify against its repo. Cold-start: a cold boot per start, because restart means start again. You build: CoW rootfs cloning, the network pool, the entire snapshot layer, the guest channel for exec and real health, and an orphan reconciler. You get for free: placement, bin-packing, restarts, service registration, Vault templating, and a UI your team already uses.
• Kubernetes plus Kata Containers — Operational weight: highest; a control plane, a CRI runtime class, a containerd shim, CNI, CSI, and the upgrade cadence that comes with all of it. Cold-start: slower than a container, and pod startup includes image pull plus VM boot. You build: less than the other rows — Kata deliberately makes a VM look like a pod, so admission control, network policy, storage, and the operator ecosystem mostly just work. Verify Kata's current Firecracker support and device model constraints against its docs; the supported hypervisor matrix moves. Correct if you are already deep in Kubernetes and want isolation as a runtime class rather than a new platform.
• A purpose-built microVM platform — Operational weight: near zero for the hosted product; moderate if you self-host it on your own KVM boxes. Cold-start: snapshot-restore on every create, which for us is 179ms p50 and 203ms p99, with a first cold boot around 3s before a snapshot exists and a same-host fork in the 400–750ms range. You build: nothing on this list, which is exactly the trade — you also give up the ability to change how any of it works, and you are taking a dependency on someone else's roadmap. This is PandaStack's category, so weight it accordingly.

The contrast, stated plainly

For calibration, the equivalent of the job file and the prestart script above, on a platform where the gap list is somebody else's problem. The snapshot restore, the reflinked disk, the pre-allocated netns, and the guest channel are all in there — you just do not get to see them, which is the point and also the cost.

from pandastack import Sandbox

# Snapshot-restore on every create: ~179ms p50, ~203ms p99. No warm pool
# of idle VMs, so an idle tenant costs nothing.
sbx = Sandbox.create(template="base", ttl_seconds=300)

sbx.filesystem.write("/app/main.py", "print('hello from a microVM')\n")
r = sbx.exec("python /app/main.py")
print(r.exit_code, r.stdout)

# Fork the running machine: CoW memory + reflinked disk. Same host is
# 400-750ms; cross-host is 1.2-3.5s because the memory image has to move.
branch = sbx.fork()
branch.exec("rm -rf /  # someone's model generated this, and that is fine")

sbx.kill()

The guest here is Ubuntu 24.04 on kernel 5.10 under Firecracker v1.16, which is the same stack you would be running under Nomad. The difference is not the hypervisor. It is the two thousand words above.

So should you do it?

Yes, in one specific case, and it is a common one: you already run Nomad, you have a small number of workloads that need a hardware boundary, those workloads are long-lived rather than per-request, and their state either lives outside the VM or you are content to pin them. In that world you write a prestart script for the reflink, lean on CNI for networking, accept cold boots because a three-second start is invisible for a service that runs for days, and add a real health check that goes through the guest instead of past it. That is a weekend of work and a genuinely good outcome. Do not let anyone — including me — talk you into a platform for three workloads.

It stops being the right answer when start latency lands on a user's critical path. The moment you want a fresh machine per request, per agent step, or per CI job, the snapshot layer becomes mandatory, and the snapshot layer is not a weekend. It is a bake pipeline, a distribution story for multi-gigabyte memory files, an invalidation scheme, and — if you want restores to be fast on a node that does not have the snapshot locally yet — demand-paging memory over the network, which is a userfaultfd handler and a chunk cache and a genuinely hard debugging surface when it fails. We wrote all of that. I would not describe the experience as fun.

The clarifying question is not "Nomad or not." It is whether starting a microVM is an operational event or a request-time operation. Nomad is a good answer for the first and structurally the wrong shape for the second, and no amount of driver work changes that — because the mismatch is not in the driver, it is in the idea that restarting a task means starting it again.