Isolating Genomics Pipelines in Per-Job microVMs

You run a genomics platform. Researchers upload a Nextflow, Snakemake, or WDL pipeline, point it at a dataset, and hit go. Somewhere in that pipeline is `bwa mem`, `samtools sort`, a GATK step, a custom R script a postdoc wrote at 2am, and a `nextflow.config` that pulls three container images you've never seen. The data it runs against is patient sequence data — PHI, under a BAA, the kind of thing that ends up in an incident report if it goes anywhere it shouldn't. This is a great product. It's also the moment your platform stops running only code you wrote.

I'm Ajay — I build PandaStack, an open-source Firecracker microVM platform, so I spend my days thinking about exactly where the isolation boundary sits when you run other people's code against other people's data. Genomics pipelines are a near-perfect storm of the properties that make a shared execution environment dangerous: the code is effectively untrusted, the jobs are heavy and long-running, reproducibility is not optional, and the data is sensitive. This post is about running each pipeline job in its own microVM — and using snapshot and fork so every run starts from an identical, disposable, reference-genome-loaded state.

Why a genomics pipeline is the hard case

Most "run untrusted code" posts are about a short snippet. A pipeline is worse on every axis. It's untrusted in a way that's easy to underestimate: even a well-meaning lab's Nextflow workflow pulls arbitrary container images, runs arbitrary tools, and executes custom scripts you never reviewed and couldn't review at the rate they're submitted. A process that can shell out to `bwa` can just as easily open a socket, read an environment variable, or walk the filesystem looking for the last job's data. Nobody has to be malicious. A copy-pasted `process.container` line or a helpful `curl | bash` in a setup step is enough.

And it's heavy in ways that don't stay politely inside one job. Alignment and variant calling are memory-hungry and disk-hungry — a genome index, a 400GB BAM file, a sort that spills tens of gigabytes of temp, a `while True` in someone's cleanup step that pins a core for six hours. On a shared cluster, that pressure doesn't respect job boundaries: page cache thrashes, the host OOM killer starts making decisions on everyone's behalf, and another lab's perfectly reasonable RNA-seq run — which happened to land on the same node — dies for a job it didn't submit. A 400GB BAM file and a `while True` in someone's cleanup script are both just Tuesday.

The shared cluster fuses every lab's fate together

The traditional design is a shared HPC-style cluster or a pool of warm containers pulling jobs off a queue. It's efficient and it's familiar, and it fuses tenants together in two ways that both eventually find you.

The first is the noisy neighbor, already described: one lab's runaway alignment starves or OOM-kills another lab's run because they share a kernel, a page cache, and an OOM killer. The second is worse because it's silent. A shared worker has, at some point, held Lab A's PHI on its local scratch disk and in its page cache, and Lab B's PHI right after. It has your platform's credentials in its environment — the object-store keys for the sequence bucket, the warehouse DSN, the API tokens. Tenant-supplied pipeline code running on that worker can read all of it: a temp BAM the last job left in `/tmp`, a dataframe still resident in memory, the env var holding your master key. A shared kernel makes it worse still — a container escape or kernel privilege-escalation bug turns "Lab A's pipeline" into "root on the node that runs everybody's PHI." You didn't leak the data. The architecture did.

One microVM per job

The fix is to stop sharing the machine. Run each pipeline job — or each process step, if you want to go all the way — in its own Firecracker microVM. Each VM boots its own guest kernel and is confined by KVM hardware virtualization; the only path out is a tiny set of emulated virtio devices behind a jailer, not the full Linux syscall surface a container leaves exposed. This is the same isolation model AWS Lambda and Fargate use to run untrusted code from thousands of customers on shared fleets. Lab A's runaway `bwa` job, or its attempt to read the filesystem for the last job's BAM, is contained to one VM. Its memory, its scratch disk, its network are its own. When it misbehaves, you kill one machine and no other lab notices.

The reflexive objection is startup cost — nobody wants a three-second VM boot in front of a job. For a pipeline that runs for an hour, a few seconds of boot is rounding error, but PandaStack sidesteps it anyway with snapshot-restore: every create restores a baked snapshot of an already-booted machine rather than cold-booting. The restore step is around 49ms; an end-to-end create is p50 179ms and p99 about 203ms. A true cold boot, only on the very first spawn of a template, is around 3s — after that you pay restore prices. So per-job hardware isolation costs you effectively nothing at the front of a job that's going to run for an hour anyway.

# Inside the guest microVM: a Nextflow run for ONE job, isolated from every
# other lab. The tools (bwa, samtools, GATK) and the reference genome are
# already present in the baked template / forked snapshot.

# Local-executor Nextflow, no shared scheduler, no cross-tenant queue.
export NXF_HOME=/workspace/.nextflow
export NXF_TEMP=/workspace/tmp        # scratch stays inside THIS vm's disk

nextflow run /workspace/pipeline/main.nf \
  -profile standard \
  --reads    /workspace/data/sample_R{1,2}.fastq.gz \
  --genome   /refs/GRCh38/genome.fa \
  --outdir   /workspace/results \
  -work-dir  /workspace/work \
  -with-report /workspace/results/report.html \
  -with-trace  /workspace/results/trace.txt   # reproducibility receipts

# A single step, run directly, looks exactly like a normal bioinformatics box:
bwa mem -t 4 /refs/GRCh38/genome.fa \
  /workspace/data/sample_R1.fastq.gz /workspace/data/sample_R2.fastq.gz \
  | samtools sort -@ 4 -o /workspace/results/sample.sorted.bam -
samtools index /workspace/results/sample.sorted.bam

# If this job spills 400GB of temp or spins forever, it hits THIS vm's disk
# and CPU ceiling and dies alone. No other lab's run is on this machine.

Reproducibility: fork a reference-genome-loaded VM

Reproducibility is where microVMs quietly earn their keep in genomics, because snapshot and fork give you something a shared cluster can't: a bit-identical, disposable starting state. Building the reference environment for a human WGS pipeline is not cheap — you download and index GRCh38, warm the tool caches, stage the known-sites VCFs, install the exact tool versions. Doing that at the front of every job is wasteful and, worse, non-deterministic: the index you build today isn't guaranteed byte-for-byte identical to the one you built last month.

So do it once. Boot a VM, load the reference genome and index, pin every tool version, and snapshot it. Now every job forks that snapshot. A same-host copy-on-write fork lands in 400–750ms (cross-host 1.2–3.5s) and shares memory and disk copy-on-write, so each run gets a private, writable copy of a known-good, reference-loaded machine without re-doing the setup and without any chance of drift between runs. The reference genome, the indexes, the tool binaries — every fork sees the exact same bytes, because they are the exact same bytes, shared read-only until a step writes. Run the pipeline, collect the results and the Nextflow trace, destroy the fork. The next job forks the identical parent again.

Here's the shape in the Python SDK — provision an isolated job VM, inject only this run's inputs, run the pipeline with a hard timeout, read the results back, and destroy the machine:

from pandastack import Sandbox

def run_pipeline_job(job_id: str, tenant_id: str,
                     pipeline_nf: str, sample_fastq: bytes) -> dict:
    """Run ONE lab's pipeline job in its own microVM, then destroy it.
    The VM sees only this job's inputs and the shared read-only reference
    genome -- never another tenant's PHI, never our platform secrets."""
    with Sandbox.create(
        # A template (or a fork of a reference-genome-loaded snapshot) that
        # already has bwa/samtools/GATK and /refs/GRCh38 baked in.
        template="base",
        ttl_seconds=14400,                       # 4h backstop: a leaked VM reaps itself
        metadata={"tenant_id": tenant_id,        # per-tenant audit + billing
                  "job_id": job_id},
    ) as sbx:
        # Inject ONLY this job's inputs. No object-store keys, no warehouse
        # DSN, no platform env -- the pipeline runs against local files only.
        sbx.filesystem.write("/workspace/pipeline/main.nf", pipeline_nf)
        sbx.filesystem.write("/workspace/data/sample_R1.fastq.gz", sample_fastq)

        # The researcher's pipeline. We never reviewed it and never trust it.
        # timeout_seconds is the circuit breaker for the runaway alignment /
        # infinite-loop class of failure. It kills THIS vm, no one else's.
        result = sbx.exec(
            "cd /workspace && nextflow run pipeline/main.nf "
            "--reads 'data/sample_R{1,2}.fastq.gz' "
            "--genome /refs/GRCh38/genome.fa --outdir results",
            timeout_seconds=14000,
        )
        if result.exit_code != 0:
            # Hand logs back to the lab so they can fix their pipeline.
            raise RuntimeError(result.stderr)

        vcf = sbx.filesystem.read("/workspace/results/variants.vcf.gz")
        trace = sbx.filesystem.read("/workspace/results/trace.txt")
        return {"job": job_id, "tenant": tenant_id, "vcf": vcf, "trace": trace}
    # VM (and every byte the pipeline touched, including scratch BAMs) is gone.

The load-bearing details are the same three as always: `metadata` tags the run so your audit log and billing attribute it to exactly one lab; `timeout_seconds` is the hard circuit breaker against the alignment that runs forever; and the only data in the guest is this job's inputs plus the shared read-only reference. The VM has no route to your sequence bucket and no platform credentials in its environment, so even if the pipeline is actively hostile, the worst it can do is compute a wrong answer about its own inputs.

Give each VM only its own inputs — and none of your secrets

Isolating the machine is necessary but not sufficient. A perfectly isolated VM that you hand a superuser object-store key is a well-contained way to let a pipeline read every lab's PHI. The VM boundary contains the code; it does not sanitize the powers you grant it. So the second rule is: the guest gets exactly the data this run is allowed to see, and nothing else.

• Inject this job's inputs, not a bucket credential. Stage that job's FASTQ/BAM into the guest filesystem yourself (`filesystem.write`, or a per-job scoped download), and let the pipeline run against the local copy. The VM has no network route to your sequence store at all, so a nosy pipeline step has nothing to be nosy about.
• Keep platform secrets out of the guest. Your object-store master keys, warehouse DSN, and internal API tokens must never be exported into the sandbox environment. If the running pipeline can read an env var, treat that env var as leaked. Pass only run-scoped, least-privilege credentials — never the platform's own.
• Make the reference read-only and shared. The reference genome and indexes are the same for everyone and safe to share; ship them baked into the template or the forked snapshot as read-only data. Copy-on-write means every job sees the same bytes without a per-job copy.
• Attribute everything. Tag the sandbox with tenant and job id so a runaway run, a suspicious egress attempt, or a billing line traces back to exactly one lab and one job — which is also most of your BAA audit story.

Resource limits: cap the job to one VM you can kill

The whole reason a runaway alignment is survivable here is that it's fenced inside one machine with a finite ceiling. A microVM has a fixed memory and CPU budget baked into its template — a BAM sort that tries to hold 200GB in memory doesn't eat the host, it hits the VM's ceiling and the guest OOM-kills the offending process, which is exactly the blast radius you want. The tenant's job dies; the host and every other lab's VM keep humming. Disk works the same way: the scratch spill is bounded by the guest's own disk, so a pipeline that fills its scratch fills only its own.

Layer a timeout on top for the failure that isn't about memory — the step that would technically finish sometime next quarter, the retry loop that never converges. The `timeout_seconds` on `exec` cancels the run; the `ttl_seconds` on the sandbox is the backstop that reaps the whole VM even if your orchestration process crashes mid-run and forgets about it. Two independent cutoffs, because with real pipelines the expensive run is the common case, not the exception.

• Reproducibility — Shared cluster: the reference index and tool versions depend on whatever the setup script produced this time; drift between runs is possible and hard to notice. microVM per job: every run forks the same snapshot, so it starts from a byte-identical, reference-loaded state and diverges only where its inputs differ.
• Noisy neighbor — Shared cluster: a 400GB alignment or a pinned core degrades or OOM-kills a neighbor's job on the same node. microVM per job: the pressure hits this VM's baked memory/CPU ceiling and OOM-kills only this guest's process; neighbors are untouched.
• PHI isolation — Shared cluster: a worker holds multiple labs' data on scratch and in page cache, plus platform credentials in its environment, all reachable by tenant code. microVM per job: the guest has its own kernel, disk, and memory and only its own inputs — another lab's PHI is physically not present.
• Blast radius of a runaway — Shared cluster: an infinite loop or a fork bomb is a host problem you mitigate with cgroups and hope. microVM per job: it's capped to one disposable VM you `kill()`; teardown takes the scratch BAMs and half-written temp with it.
• Cleanup — Shared cluster: reset the worker and hope no temp BAM, open FD, or stale credential survived to the next tenant. microVM per job: destroy the VM; memory, filesystem, and scratch vanish, so the leak surface between one job and the next is zero.

When the pipeline needs a real database

Some pipelines write results, sample metadata, or a variant catalog to a real SQL database — a per-project Postgres a downstream step queries, a store a reporting layer reads. Handing the sandbox a role on a shared warehouse re-introduces exactly the cross-tenant leak you just designed out: now a hallucinated or hostile query is one missing filter away from another lab's cohort. The stronger answer is a database that physically contains only one tenant's data. On PandaStack a managed database is a dedicated Firecracker microVM plus its own durable volume — not a schema carved out of a shared instance — created in 30–90s (the create blocks until Postgres is genuinely accepting connections). A per-tenant results database is then a hardware-enforced boundary: a query that forgets its filter returns that tenant's rows and nobody else's, because there's no one else in the database. Provision it once per tenant, give the sandbox a read-only, run-scoped role with a statement timeout, and even a hostile query can't mutate anything, can't run forever, and can't see across the boundary.

Teardown is the feature, not an afterthought

The reason this architecture stays safe and compliant under load is that destruction is total and cheap. When a job finishes — or blows its timeout, or the lab churns — you kill the VM, and its memory, its scratch disk, its staged FASTQ/BAM, and any half-written temp go with it. There's no worker to reset, no scratch directory to sweep, no lingering object-store handle to reap, no chance the next job inherits a stale BAM. Fresh-per-job VMs tear down automatically via `ttl_seconds` and the `with` block; a long-lived reference-parent VM you keep warm and fork from, and destroy only when you re-bake the reference. Either way, the PHI leak surface between one job and the next is zero, because there is no next job on the same machine.

Putting it together

Letting researchers run their own pipelines is a great feature and a genuine liability, and the liability is entirely about where that code runs and against whose data. Don't run it on a shared cluster, where one lab's runaway alignment OOM-kills a neighbor and one lab's pipeline can read another's PHI and your object-store keys. Run each job in its own Firecracker microVM: give the guest only that job's inputs plus a shared read-only reference and none of your platform credentials, cap it with a timeout and the VM's own memory and disk ceilings, fork a reference-genome-loaded snapshot so every run starts from a byte-identical reproducible state, reach for a per-tenant managed database VM when a step needs live SQL, and let snapshot-restore keep the front of every job fast enough that isolation isn't a tax. The runaway `bwa` job still happens. It just takes down one disposable VM instead of another lab's afternoon — and it never gets anywhere near their patients' data.