Slurm and HPC Batch Scheduling vs an On-Demand MicroVM Fleet

If you have run a Slurm cluster you already know the two facts that determine everything about it: the machines were bought, and the machines are finite. Nearly every feature Slurm is famous for follows from those two facts. It is not a job runner that happens to have a queue; it is an arbitration system that happens to run jobs.

An ephemeral microVM fleet answers a superficially identical question — I have a thousand independent pieces of work, please run them — from the opposite premise. Most comparisons in this space are marketing written by one side. The honest position is that the two are good at different workload shapes, and the boundary is sharper than a vendor would like.

What a batch scheduler is actually solving

Read Slurm's feature list as a list of policies rather than capabilities and it snaps into focus. Almost none of it is about executing your program. All of it is about deciding whose program executes first on hardware that cannot run everything at once.

• Fair-share. A decayed usage tree that lowers your priority because your group has already had more than its allocation this fortnight. A political instrument encoded as arithmetic, and it exists because the alternative is the loudest professor getting the cluster.
• Backfill. Small jobs are slotted into gaps ahead of a large job waiting for nodes, provided they finish before those nodes are ready. The single most valuable thing Slurm does for utilisation, and it only works because every job declares a walltime.
• Partitions and QOS. Static carve-ups of the machine — a short debug partition, a bigmem partition, a GPU partition — plus caps on how much any one account can hold at once.
• Reservations. Hold nodes for Thursday's demo, a deadline, or a maintenance window. A booking system for a scarce resource.
• Gang scheduling and preemption. Time-slice competing jobs on the same nodes, or evict a low-priority job so a high-priority one can start. Both are ways of sharing something you cannot duplicate.

Notice what none of these do: make another machine appear. Every one is a policy for deciding who is disappointed, and how gracefully. That is not a criticism. When the hardware is a capital purchase that must serve a department for five years, arbitration is the product, and Slurm is genuinely very good at it.

The inversion: when capacity is elastic, the queue becomes a failure mode

On a fleet, a create call returns a running machine or it returns an error. There is no third state in which you are forty-seventh in line with an estimated start of 03:40. On PandaStack a create is a restore of a baked Firecracker snapshot rather than a boot — no warm pool of idle VMs, a p50 of 179ms and a p99 around 203ms — so for the common case queue wait has nowhere to live.

Elastic is not infinite, and pretending otherwise would be dishonest. When the fleet genuinely has no room the API returns a 503 with "no compute capacity available". The queue has not been abolished, it has been relocated into your client as a retry loop. Our own app-hosting path does exactly this: a deploy or a wake refused for capacity is parked in a waiting_capacity state with exponential backoff rather than failed outright. That is a queue. It just has no fair-share, no backfill, and no promise about when your turn comes.

Mapping Slurm concepts onto a fleet API

Most of the vocabulary translates. Some of it translates into nothing, and one item translates into something that looks the same and behaves differently, which is the dangerous kind.

• sbatch job.sh → POST /v1/sandboxes, then exec. Submission is no longer decoupled from execution: no spool directory holds your script until a node frees up, so durability of the intent becomes your problem.
• sbatch --array=1-1000 → a thousand independent creates from your own loop, or fork_tree(count) off a parent that has already done the expensive setup. The array is client-side; there is no server object called an array.
• --time=04:00:00 → ttl_seconds, and this is the mistranslation. See the next section before relying on it.
• --mem and --cpus-per-task → mostly not a knob. Guest RAM is baked into the snapshot, so memory is a template property, not a request parameter: base is 4 GiB, code-interpreter 2 GiB, and every first-party template bakes 8 vCPU as burst capacity arbitrated by cgroup weight. "Choose a template" replaces "request resources".
• module load python/3.12 gcc/13.2 → baked into the image. Nothing to load, because nothing else is installed to conflict with it.
• Partitions and QOS limits → per-account tier quotas. On our card that is 5 concurrent sandboxes and 60 creates an hour on Free, 50 and 600 on Pro, 500 and 6,000 on Team. Coarser than a QOS, and enforced against your account rather than a partition.
• sacct and sreport → per-sandbox usage metering, billed on active CPU-seconds and resident GiB-seconds.
• scancel → kill(), called in a finally block every time, plus a TTL for the day you forget.
• $SLURM_TMPDIR and node-local scratch → the guest's own rootfs, discarded when the sandbox dies. Same semantics, stronger guarantee: a different filesystem in a different kernel.
• salloc for an interactive session → create a persistent sandbox and attach an exec or a PTY to it.
• srun --mpi=pmix → nothing. There is no equivalent, and no amount of squinting produces one.

--time and ttl_seconds are not the same thing

Slurm's --time is a hard walltime kill measured from job start, and it is also a scheduling input: declaring it is what makes backfill possible. PandaStack's ttl_seconds is an idle timeout. The agent's reaper wakes on a 30-second tick, compares now minus the sandbox's last activity against its TTL, and deletes it if the idle window has been exceeded. The default is five minutes. Three consequences follow.

• A long silent job can be reaped out from under you. A forty-minute computation kicked off with a single exec produces no further API traffic for forty minutes. Set ttl_seconds comfortably above your worst-case duration rather than equal to it.
• A wedged job that keeps talking will never be reaped, because any activity resets the clock. Slurm's --time would have killed it at four hours without being asked. For a genuine ceiling, set timeout_seconds on the exec and kill in a finally block.
• Nothing backfills. Because no job declares a duration, the placement scheduler has no model of the future at all — it scores hosts on free CPU and free memory and spreads load. That is right for short jobs on elastic capacity and strictly worse than Slurm's when the hardware is fixed and full.

What you actually gain

• Zero queue wait in the normal case. The gap between submitting work and it running collapses from minutes-or-days to roughly two hundred milliseconds, which for an interactive research loop changes what you are willing to try at all.
• A pristine environment per job, enforced by a machine boundary. The module system exists because a shared node has one filesystem and many incompatible software stacks; a fresh guest per job makes that problem structurally absent. No LD_LIBRARY_PATH archaeology, no run that stopped reproducing because the site default gcc moved.
• Cost attribution that is real money rather than accounting fiction. sacct tells you a job was allocated 32 cores for six hours, not that it used 1.4 of them, because on a machine you already own the distinction has no financial consequence. Metering here is active CPU-seconds and resident GiB-seconds against one rate card — $0.054 per vCPU-hour, $0.0162 per GiB-hour — so a job given 8 vCPU that burns one pays for one.
• Warm start via copy-on-write. Load the reference genome or the trained model once in a parent, then fork per trial: same-host forks land in 400 to 750 milliseconds, and the child inherits the parent's memory rather than rebuilding it. There is no cluster equivalent.
• Idle costs nothing. A cluster quiet at 4am cost exactly as much as one that was saturated. The largest economic difference, and entirely structural.

What you genuinely lose

This is the section that decides the answer for most people, so it gets detail rather than a hand-wave.

• No MPI-grade interconnect. Each sandbox gets its own network namespace and a /30 out of a pre-allocated pool, with NAT'd egress — an excellent substrate for jobs that ignore each other, a terrible one for an all-reduce. No RDMA, no InfiniBand, no tuned fabric.
• No gang scheduling. Slurm guarantees all 512 ranks start together or none do. A fleet has no such primitive: 512 creates give 512 independent outcomes, across hosts, possibly with a few 503s among them. For a coupled job that is not degraded but a non-starter — rank 300 arriving four seconds late means 511 ranks sat idle.
• No shared parallel filesystem. No Lustre, no GPFS, no cluster-wide /scratch a thousand ranks write into concurrently. Durable volumes here are host-pinned sparse ext4 images, and attaching one pins the sandbox to that host — a local disk, not a fabric-attached one. The idiom is object storage in, object storage out: right for per-sample work, wrong for anything that memory-maps a shared multi-terabyte dataset.
• No GPU. Sandboxes are CPU-only with no passthrough. If the work is model training, GPU-accelerated molecular dynamics, or anything ending in a CUDA kernel, the conversation ends here rather than after a workaround.
• No reservations. Nothing holds nodes for Thursday's demo. The fleet's answer to "will the capacity be there" is "probably", backed by an autoscaler and a rate card rather than a booking you can point at.
• No fair-share. Two groups competing for the same fleet compete by whose retry lands first. Slurm's decayed share tree has no analogue here, and per-account concurrency caps are a much blunter instrument.
• No topology awareness, and no reason for it. Placement scores free CPU and free memory and spreads load; nothing models where your work sits relative to anything else.

Where the line actually falls

The test that decides it is the ratio of communication to computation. Ask what your ranks do while they run.

If they talk to each other during the run, you want a real cluster and you should stop reading. A CFD solver, a climate model, a lattice QCD run, an FEM simulation whose every timestep ends in a halo exchange — these are one machine spread across a fabric, not N independent jobs, and the fabric is the point. Nothing in this post makes that work, and nothing in it is trying to.

If they never talk, an ephemeral fleet fits better than a queue, and the usual reason such work runs on Slurm is that Slurm is what the institution has. Parameter sweeps. Monte Carlo trials. Per-sample bioinformatics, where independence is guaranteed by the biology. Batch inference over a directory of files. Regression suites. Frame rendering. Each is a for-loop that ended up on a cluster because that is where the cores were, and each pays Slurm's arbitration overhead for coordination it does not need.

The middle is narrower than people expect. Map-then-reduce work is fine provided the reduce is small enough for the client or one sandbox. What is genuinely awkward is a large shared read set with modest per-task compute, because you pay to move that data to every guest instead of having it already on a parallel filesystem. Forking from a parent that already holds it in memory is the first mitigation to try.

A parameter sweep, translated

The Slurm version, which anyone reading this could write from memory.

#!/bin/bash
#SBATCH --job-name=sweep
#SBATCH --array=1-1000%50
#SBATCH --time=00:30:00
#SBATCH --mem=4G
#SBATCH --cpus-per-task=1
#SBATCH --partition=compute
#SBATCH --output=logs/sweep-%A_%a.out

module load python/3.12 gcc/13.2 hdf5/1.14

srun python run.py --cell "${SLURM_ARRAY_TASK_ID}"

The fleet version. Roughly the same length, with the throttle, the retries and the accounting moved from the scheduler into your process — work you now own, not work that vanished.

from concurrent.futures import ThreadPoolExecutor
from pandastack import Sandbox

CELLS = list(range(1, 1001))

def run_cell(cell: int):
    # The template is the module environment: python, gcc and hdf5 are
    # already in the image, so there is nothing to load at runtime.
    sbx = Sandbox.create(
        template="code-interpreter",
        ttl_seconds=3600,              # generously ABOVE the worst-case job
        metadata={"sweep": "run-17", "cell": str(cell)},
    )
    try:
        res = sbx.exec(
            f"python /opt/sweep/run.py --cell {cell}",
            timeout_seconds=1800,
        )
        if res.exit_code != 0:
            return cell, None, res.stderr
        return cell, sbx.filesystem.read("/work/out.json").decode(), None
    finally:
        sbx.kill()                     # scancel, except you always call it

# %50 in the array directive becomes a thread pool. The throttle is yours
# now, not the scheduler's.
with ThreadPoolExecutor(max_workers=50) as pool:
    for cell, out, err in pool.map(run_cell, CELLS):
        if err:
            print(f"cell {cell} failed: {err[:200]}")

When the per-task setup is the expensive part — loading a reference dataset, warming a model — the interesting move has no Slurm equivalent. Pay the setup once in a parent and fork children out of its warm memory.

parent = Sandbox.create(template="code-interpreter", ttl_seconds=7200)
parent.exec("python /opt/sweep/preload.py", timeout_seconds=900)  # read the 8 GB reference once

children = parent.fork_tree(64, metadata={"sweep": "run-17"})
for i, child in enumerate(children):
    child.exec(f"python /opt/sweep/trial.py --seed {i}", timeout_seconds=600)

The hybrid most groups end up with

Almost nobody who has a cluster should replace it. The move that pays is narrower: keep Slurm for the coupled runs it was designed for, and move the high-throughput tail off it. A partition clogged with 4,000 single-core sweep tasks is the standard complaint of the person trying to start a 512-rank job, and those two workloads have no reason to share a queue. Draining the parallel tail onto elastic capacity makes the cluster better at what only it can do.

The one piece of architecture to get right on the fleet side is what Slurm gave you for free: durable intent. sbatch wrote your job into a spool directory that survived your session, a scheduler restart and a node dying. A create call over HTTP survives none of that. Put the work list in a table, mark rows claimed, run the sandbox, record the outcome, and sweep back rows whose claim expired. It is fifty lines of Postgres and it is not optional — without it, a fleet is a very fast way to lose a sweep to a laptop lid.

Slurm is a queue in front of a machine you already own. A fleet is a machine that appears when you ask. Neither is a better answer to the other's question.