Firecracker vs nsjail: which for running untrusted code?

nsjail and Firecracker both answer "how do I run code I don't trust," but they answer it at completely different layers. nsjail is Google's lightweight process-isolation tool: it takes a single Linux process and wraps it in namespaces, a seccomp-bpf syscall filter, cgroups, resource limits, and a chroot or pivot_root. Firecracker is a Virtual Machine Monitor that boots a whole guest kernel behind KVM. The one-line difference: with nsjail the untrusted code still calls your host kernel's syscalls; with Firecracker it calls its own throwaway kernel's syscalls. That single fact drives almost every trade-off below. (I built PandaStack, which runs on Firecracker, so I'll be upfront about where nsjail is genuinely the better tool.)

What nsjail actually does

nsjail is a shared-kernel sandbox. It doesn't emulate hardware or boot an OS — it launches your target binary and, before handing over control, stacks up every process-isolation primitive the Linux kernel offers. Roughly: clone(2) with new namespaces (PID, mount, network, user, UTS, IPC) so the process sees its own tiny world; a pivot_root or chroot into a minimal filesystem; rlimits and cgroups to cap CPU time, memory, file size, and process count; and a seccomp-bpf program that whitelists (or blacklists) the exact syscalls the process is allowed to make. The result is a hardened process with essentially zero boot cost — there's no VM to start, just a fork and some setup.

Where nsjail shines is precisely where its constraints are a feature: you have one process, you know what it should do, and you can enumerate the syscalls it legitimately needs. A judge running `./submission < input.txt` doesn't need to open sockets, fork a thousand children, or mount filesystems — so you forbid all of that and the sandbox is tight. Fast, simple, no hypervisor, no guest OS to maintain.

The shared-kernel problem

Here's the catch, and it's the same catch every namespace-based sandbox has: the untrusted process is still making syscalls into your host kernel. Namespaces limit what it can name; seccomp limits which syscalls it can call. Neither changes the fact that it's poking the same kernel that runs everything else on the box. Every isolation boundary nsjail builds is one kernel bug away from being irrelevant.

Two failure modes matter. First, seccomp filters are hard to write correctly. seccomp is you writing a list of every way the code is NOT allowed to hurt you, and hoping you didn't forget one. Forget to block a single syscall — or allow one with an argument you didn't fully reason about — and you've left a door open. Argument-level filtering is coarse (BPF can't dereference pointers), so "allow ioctl but only these commands" is trickier than it sounds. Second, kernel local privilege escalation CVEs defeat namespace sandboxes structurally. A steady stream of these (Dirty Pipe, Dirty COW, io_uring bugs, various netfilter and eBPF issues over the years) let an unprivileged process become root or escape its namespace — and once someone is exploiting a kernel bug, your carefully-written seccomp filter was allowing exactly the syscall they used.

This isn't a knock on nsjail specifically — it's the ceiling of the whole shared-kernel model. If the attacker gets to send syscalls to the kernel you also depend on, your isolation is only as strong as that kernel's entire syscall surface, which is enormous and continuously finding new bugs.

What Firecracker changes

Firecracker moves the boundary down to hardware. The untrusted code runs inside a microVM with its own guest kernel; that guest talks to the outside world only through a handful of emulated virtio devices behind KVM. When the code makes a syscall, it hits the guest kernel — a throwaway kernel that belongs to that one VM. To reach your host, an attacker has to escape the hypervisor itself, which is a far smaller and far more heavily audited surface than the full Linux syscall interface. A guest kernel LPE gets them root in a VM that gets deleted in a moment; it does not get them your host.

The historical reason people reached for nsjail instead of a VM was cost: VMs booted in seconds and ate hundreds of MB. Firecracker collapses that. It boots in a few milliseconds cold, and platforms restore a baked snapshot per request instead of cold-booting — on PandaStack a create is ~179ms at p50 (~203ms p99), and the snapshot restore itself is around 49ms. Cold boot when there's no snapshot yet is about 3 seconds, a one-time cost. So you get VM-grade isolation at roughly namespace-sandbox latency, which is the trade that used to not exist.

Side by side

• Isolation — nsjail: namespaces + seccomp-bpf + cgroups on the shared host kernel. Firecracker: hardware-virtualized microVM with its own guest kernel behind KVM.
• Boot cost — nsjail: near-zero (a fork plus setup, no VM). Firecracker: a few ms cold, ~49ms to restore a snapshot; ~3s only on the very first cold boot.
• Escape surface — nsjail: the entire host-kernel syscall interface; one LPE CVE or a filter gap ends the isolation. Firecracker: the hypervisor boundary; a guest kernel bug stays inside the throwaway VM.
• Full environment — nsjail: one constrained process, minimal chroot. Firecracker: a whole guest OS — install packages, run a network stack, spawn processes freely.
• Best fit — nsjail: a known single binary whose syscalls you control (online judges, CTF graders, audited tools). Firecracker: arbitrary, multi-tenant, or AI-generated code that needs a real environment.

What each looks like in practice

An nsjail invocation for an online-judge-style run: put the submission in its own namespaces, chroot into a minimal filesystem, cap resources hard, and whitelist a tiny set of syscalls. (Flags vary by version — check the docs before copying this.)

# Run one untrusted binary under nsjail: fresh namespaces, chroot,
# hard rlimits, and a seccomp whitelist of only the syscalls it needs.
nsjail \
  --mode o \
  --chroot /srv/judge/rootfs \
  --user 99999 --group 99999 \
  --disable_clone_newnet \
  --time_limit 5 \
  --rlimit_as 256 \
  --rlimit_cpu 5 \
  --rlimit_fsize 8 \
  --rlimit_nofile 32 \
  --seccomp_string 'ALLOW { read, write, exit, exit_group, brk, mmap, \
    munmap, arch_prctl, fstat, close } DEFAULT KILL' \
  -- /submission < /input.txt

# Every syscall the submission is allowed is on that list.
# Forget one it needs -> it dies. Forget one it can abuse -> you're exposed.

The Firecracker equivalent doesn't ask you to enumerate syscalls at all — you hand the code a whole VM and let it do what it wants inside a disposable kernel. With the PandaStack SDK that's a create, a filesystem write, and an exec with a timeout:

from pandastack import Sandbox

# The untrusted submission can call any syscall it likes -- it's calling
# its OWN guest kernel, not yours. No seccomp whitelist to get right.
submission = """
import sys
total = sum(int(x) for x in sys.stdin.read().split())
print(total)
"""

with Sandbox.create(template="code-interpreter", ttl_seconds=300) as sbx:
    sbx.filesystem.write("/workspace/submission.py", submission)
    sbx.filesystem.write("/workspace/input.txt", "3 4 5 6\n")
    result = sbx.exec(
        "python3 /workspace/submission.py < /workspace/input.txt",
        timeout_seconds=5,
    )
    print("exit:", result.exit_code)
    print("stdout:", result.stdout.strip())
    if result.stderr:
        print("stderr:", result.stderr)
# VM is destroyed on block exit -- nothing survives the run.

Note what disappears in the second version: there's no syscall list to maintain, no chroot to assemble, no rlimit tuning to get exactly right. The blast radius is the VM, and the VM is thrown away. You trade a bit of per-instance weight (a guest kernel and device model, a few MB) for not having to be perfect about a filter.

When nsjail is genuinely enough

Don't reach for a VM out of reflex. nsjail is the right call when the workload is constrained and you actually control the syscalls: a single audited binary, a compiled contest submission, a known interpreter running with a narrow whitelist. Online judges are the canonical case — thousands of short runs per minute where boot cost matters and the syscall set is small and stable. If you can write a tight seccomp filter and mean it, and you're comfortable that a kernel LPE would be your worst day, nsjail is lighter, simpler, and faster to start. There's no VM to operate.

When you want a VM instead

Reach for Firecracker (or a platform built on it) when the code is arbitrary or the tenancy is real. AI-generated code that installs its own dependencies and needs a network can't be captured by a fixed syscall list. Multi-tenant SaaS running many customers' code on shared hosts needs a boundary that a single kernel CVE can't unravel across tenants. Anything that wants a full environment — a real filesystem, background processes, a package manager — is a VM-shaped problem, not a jailed-process one. This is the line PandaStack is built on: every sandbox is its own Firecracker microVM created in ~179ms via snapshot-restore, so you get hardware isolation without the old boot tax, and you can fork a configured VM copy-on-write (same-host forks land in 400–750ms) to start each run from a known-good state.

The honest summary: nsjail and Firecracker aren't really rivals so much as tools for two different threat models. If you have one known binary and control its syscalls, nsjail is a sharp, cheap jail and a VM is overkill. If you're running arbitrary, untrusted, or multi-tenant code that needs a real environment, the shared kernel is the thing that gets you, and a throwaway guest kernel is the boundary you actually want.