How to Run a Job Queue Without a Worker Fleet

The standard background job setup is Redis plus a pool of worker processes that run permanently. It works, and for most teams it's more machinery than the workload justifies. If your jobs arrive in bursts — a few hundred at 2am, nothing for six hours — you're paying for idle workers around the clock and operating a Redis instance whose main job is to be up.

There's a version of this that has no always-on component. The queue is a table in the Postgres database you already have. The workers are isolated VMs that exist only while a job runs. Between bursts, nothing is running and nothing is billing.

This post is the actual pattern, including the parts that are easy to get wrong — the locking, the retries, and the honest performance trade you're making.

Step 1: the queue is a table

Postgres has had the primitive that makes this work since 9.5: `SELECT ... FOR UPDATE SKIP LOCKED`. It lets multiple consumers pull distinct rows from the same table without blocking each other or handing the same row to two workers. That is essentially all a job queue is.

CREATE TABLE jobs (
  id           bigserial PRIMARY KEY,
  kind         text        NOT NULL,
  payload      jsonb       NOT NULL,
  status       text        NOT NULL DEFAULT 'pending',
  attempts     int         NOT NULL DEFAULT 0,
  max_attempts int         NOT NULL DEFAULT 3,
  run_after    timestamptz NOT NULL DEFAULT now(),
  locked_at    timestamptz,
  last_error   text,
  created_at   timestamptz NOT NULL DEFAULT now()
);

-- The index the claim query needs. Partial, because only pending rows
-- are ever scanned and the table will be mostly completed rows.
CREATE INDEX jobs_claim_idx ON jobs (run_after)
  WHERE status = 'pending';

Claiming a batch of jobs is one statement. It marks rows as running and returns them atomically, so a crash between claim and execution leaves rows visibly stuck rather than silently lost.

UPDATE jobs SET status = 'running', locked_at = now(), attempts = attempts + 1
WHERE id IN (
  SELECT id FROM jobs
  WHERE status = 'pending' AND run_after <= now()
  ORDER BY run_after
  FOR UPDATE SKIP LOCKED
  LIMIT 20
)
RETURNING id, kind, payload, attempts, max_attempts;

Step 2: workers that don't exist until there's work

Now the part that replaces the worker fleet. Instead of processes waiting for jobs, a scheduled drainer wakes up on an interval, claims whatever is pending, and runs each job in a fresh isolated VM.

The reason to use a VM per job rather than running jobs in one process is isolation with teeth. Background jobs are where the messy work lives — PDF rendering, running customer-supplied code, image processing, scraping. In a shared worker, one job that leaks memory, spins a core, or segfaults takes its neighbours with it. In a per-job microVM, a job that destroys its environment destroys only its own.

from pandastack import Client, Sandbox

ps = Client()

def run_job(job):
    """One job, one VM. Nothing survives it."""
    sbx = Sandbox.create(template="code-interpreter", ttl_seconds=900)
    try:
        sbx.filesystem.write("/workspace/payload.json", json.dumps(job["payload"]))
        r = sbx.exec(
            f"python3 /workspace/handlers/{job['kind']}.py /workspace/payload.json",
            timeout_seconds=600,
        )
        if r.exit_code != 0:
            raise RuntimeError(r.stderr[-2000:])
        return r.stdout
    finally:
        # Always. A leaked sandbox is a leaked bill.
        sbx.kill()

Then the drainer itself, deployed as a function on a short cron schedule:

fn = ps.functions.deploy(
    name="job-drainer",
    runtime="python",
    path="./jobs/drainer",
    entrypoint="handler.py",
    env={"DATABASE_URL": DATABASE_URL},
)

ps.schedules.create(
    name="job-drainer-every-minute",
    function_id=fn["id"],
    cron="* * * * *",
)

Step 3: retries, backoff, and the poison pill

The failure handling is where homegrown queues usually go wrong, and it's not complicated as long as you decide up front what "failed" means.

Three outcomes per job. Success marks it done. Failure with attempts remaining puts it back to pending with a `run_after` in the future — exponential backoff, so a failing dependency doesn't get hammered. Failure with attempts exhausted marks it dead, and dead jobs stay in the table so someone can look at them.

def finish(conn, job, error=None):
    if error is None:
        conn.execute("UPDATE jobs SET status='done', last_error=NULL WHERE id=%s",
                     (job["id"],))
    elif job["attempts"] < job["max_attempts"]:
        # 2^attempts minutes: 2, 4, 8 ... plus jitter so a batch of
        # jobs failing on the same dependency doesn't retry in lockstep.
        delay = (2 ** job["attempts"]) * 60 + random.randint(0, 30)
        conn.execute(
            "UPDATE jobs SET status='pending', run_after=now() + (%s || ' seconds')::interval,"
            " last_error=%s WHERE id=%s",
            (delay, str(error)[:2000], job["id"]),
        )
    else:
        conn.execute("UPDATE jobs SET status='dead', last_error=%s WHERE id=%s",
                     (str(error)[:2000], job["id"]))

One more query you need, and people forget it until it bites: jobs stuck in `running`. If a drainer dies mid-job, its rows sit claimed forever. A reaper that returns long-running claims to pending fixes it.

-- Anything claimed more than 30 minutes ago is presumed dead.
-- Make the interval comfortably longer than your longest real job.
UPDATE jobs
SET status = 'pending', locked_at = NULL
WHERE status = 'running' AND locked_at < now() - interval '30 minutes';

The trade you're making

I'd rather state this plainly than let you discover it later.

A per-job VM is not free latency-wise. Creating a sandbox from a snapshot takes on the order of a couple hundred milliseconds, and a function invocation carries roughly a second of platform overhead for a fresh environment. If your jobs take minutes, that's noise. If you're processing thousands of jobs that each take 50ms, per-job isolation is the wrong shape entirely — batch them and run many per VM.

The cron granularity matters too. A one-minute schedule means up to a minute of queue latency before a job is even claimed. For most background work — sending an email, generating a report, syncing a third party — that's fine, and it's the reason this is a background job pattern and not a request-path one. If you need sub-second pickup, you want a long-running consumer with a real broker, and you should use one.

• Good fit: jobs measured in seconds or minutes, arriving in bursts, where isolation matters and a minute of pickup latency doesn't.
• Bad fit: high-frequency tiny jobs, or anything a user is actively waiting on.
• Middle ground: claim a batch and process several jobs in one VM. You keep the blast radius bounded per batch instead of per job, and amortize startup across the batch.

Why this is worth it

Three reasons, in the order they tend to matter.

Cost, most obviously. Idle worker fleets are pure waste for bursty workloads, and the waste is continuous. If nothing is running between bursts, that line goes to zero rather than to "small".

Fewer moving parts. No Redis to operate, no broker to keep up, no separate durability story to reason about. The queue is in the database you already back up, which means your jobs inherit your existing point-in-time recovery for free. That's a genuinely underrated property — most people's Redis-backed queues would lose in-flight work in a restart and they've never checked.

And isolation. Once each job runs in its own VM, whole categories of problem stop being your problem. A memory leak doesn't accumulate across jobs. A job that runs untrusted or customer-supplied code has a hardware boundary around it rather than a `try/except`. A hung job disappears when its sandbox TTL expires instead of occupying a worker slot indefinitely.

The takeaway

If you already have Postgres, you already have a durable job queue — `SKIP LOCKED` plus a retry column is most of what a broker does for the volumes most applications actually see. Pair it with per-job VMs and a cron drainer, and the always-on part of your background job infrastructure disappears entirely.

Reach for a real broker when you outgrow it: sustained high throughput, sub-second pickup, or fan-out patterns a table doesn't model well. Until then, this is less to run and less to go wrong.