How to Run a BullMQ Worker Alongside Your Node App

Count the jobs your Node app actually enqueues in a day — not the number in the scaling deck, the number in your dashboard this morning. For most apps it is a few thousand: a welcome email, a webhook retried because someone's endpoint 502'd, a nightly report, a handful of thumbnails. At a couple hundred milliseconds each that is minutes of CPU spread over twenty-four hours, and most platforms will happily sell you an entire second always-on service to spend them.

I'm Ajay, I build PandaStack. A Firecracker microVM is a real Linux machine with its own kernel and a root shell, not a function slot, so a second long-running process costs nothing but RAM you already paid for. Here is how to run a BullMQ worker next to your Express or Next.js server — and where to stop.

First, the Redis question — answered plainly

BullMQ is not Redis-flavoured, it is Redis-shaped. Its queues are Redis lists and streams, its delayed and retry sets are sorted sets keyed by timestamp, its job locks are keys with TTLs, and the whole thing leans on Lua scripts to move a job between those structures atomically. There is no Postgres adapter. If what you want is a database-backed queue, you want a different library — pg-boss and Graphile Worker both put the queue in Postgres. Choosing BullMQ means choosing to run Redis.

Option one, and my recommendation whenever losing an enqueued job would cost you money or a customer's trust: point at a managed Redis over TLS. Put the rediss:// URL in REDIS_URL and let somebody else own failover, replication and the backup you are not currently testing.

Option two: run Redis inside the same sandbox. You have root on a full Ubuntu 24.04 userspace, so this is an apt install and a config file, not a platform negotiation. The tradeoff, without spin: a co-located Redis with appendonly yes on a durable volume is fine for jobs you can afford to retry, and wrong for jobs you cannot afford to lose.

# Co-located Redis, configured like it holds something you care about.
apt-get install -y --no-install-recommends redis-server

# /etc/redis/redis.conf
appendonly yes
appendfsync everysec          # <= 1s of enqueues at risk on a hard kill
dir /mnt/data/redis           # a DURABLE volume, not the ephemeral rootfs
maxmemory 512mb
maxmemory-policy noeviction   # NEVER allkeys-lru. See the paragraph below.
save ""                       # AOF is the durability story; skip RDB churn

redis-server /etc/redis/redis.conf --daemonize yes

That noeviction line is the one that quietly ruins afternoons. A Redis set to allkeys-lru will, under memory pressure, delete whichever keys it likes — and sometimes the keys it likes are the sorted set holding your delayed jobs. Nothing errors. Nothing logs. The jobs are simply gone, and the first person to notice is a customer asking where their invoice went. Set noeviction so a full Redis fails an enqueue loudly instead of eating a queue quietly.

What you accept with the co-located option: no replica, so a lost machine is a lost queue; backups are yours to write and yours to test; everysec fsync can drop the last second of enqueues on a hard kill. Fine for a thumbnail the user can regenerate. Bad for a payment, where the enqueue is the only durable record the work was requested — for those, write a row to Postgres first and make the job a pointer to it.

The connection option that costs everyone an hour

Before the process layout, the gotcha. BullMQ workers spend most of their life blocked inside Redis — a blocking pop or an XREAD BLOCK, waiting for work that has not arrived. ioredis sensibly assumes a command that does not return is a command in trouble, and after maxRetriesPerRequest attempts it throws. Applied to a blocking pop, that default turns normal idleness into a crash loop. BullMQ knows, and refuses to construct a Worker unless you set the option to null yourself.

// lib/queue.ts -- one connection factory, shared by web and worker.
import IORedis from "ioredis";
import { Queue } from "bullmq";

// maxRetriesPerRequest MUST be null for anything that hosts a Worker.
// BullMQ throws at construction time if it is not, and the error message
// is the single most-searched BullMQ string on the internet.
export const connection = new IORedis(process.env.REDIS_URL!, {
  maxRetriesPerRequest: null,
  enableReadyCheck: false,
});

export const emails = new Queue("emails", {
  connection,
  defaultJobOptions: {
    attempts: 5,
    backoff: { type: "exponential", delay: 2_000 },
    // Completed jobs are Redis keys. Unbounded retention is how a
    // 512mb maxmemory turns into a failing enqueue three weeks later.
    removeOnComplete: { age: 3_600, count: 1_000 },
    removeOnFail: { age: 86_400 },
  },
});

// worker.ts -- the process the web server does NOT run.
import { Worker } from "bullmq";
import { connection, emails } from "./lib/queue";

export const worker = new Worker(
  "emails",
  async (job) => {
    // Idempotent by construction: the key comes from the job payload,
    // not from Date.now(). See the shutdown section for why.
    await sendWelcome(job.data.userId, job.data.eventId);
  },
  {
    connection,
    concurrency: Number(process.env.WORKER_CONCURRENCY ?? 5),
    lockDuration: 30_000, // must exceed your slowest handler. See below.
  },
);

Note removeOnComplete. Every finished job is a Redis hash that lingers until something deletes it, so a queue with no retention policy is a slow-motion memory leak that ends at maxmemory with an enqueue throwing at three in the morning.

One start command, two processes

The shape that works is not symmetric, and that asymmetry is the whole trick: the worker runs in the background under a restart loop, the web server runs in the foreground via exec so it is the process the platform actually signals and probes.

#!/bin/sh
# bin/start -- worker in the background, web in the foreground.
set -e

# EXPORT before anything is backgrounded. Start commands run as a
# non-login 'sh -c'; nothing sources a profile, and a variable that is
# assigned but not exported never reaches a child process.
export NODE_ENV=production
export MISE_DATA_DIR=/opt/mise MISE_CONFIG_DIR=/opt/mise
export PATH=/opt/mise/shims:$PATH

( backoff=1
  while true; do
    node dist/worker.js 2>&1 | sed -u 's/^/[worker] /'
    echo "[worker] exited (rc=$?), restarting in ${backoff}s" >&2
    sleep "$backoff"
    backoff=$(( backoff < 30 ? backoff * 2 : 30 ))
  done ) &
WORKER=$!

# Forward the deploy's SIGTERM to the worker so it gets a drain window,
# then WAIT for it -- the shell must outlive the signal to do that.
trap 'kill -TERM "$WORKER" 2>/dev/null; wait "$WORKER"' TERM INT

exec node dist/server.js

Three details in there are load-bearing. The exports come first because a child inherits the environment as it existed at fork time — set PORT without exporting it and your server binds its default while the platform probes something else, which produces clean logs and a failing health check. The exec on the last line replaces the shell with the server, so SIGTERM lands on Node rather than on a shell that never forwards it. And the wait in the trap is what buys the worker its drain window: without it the shell exits the instant it has sent the signal, the platform sees the parent gone, and the SIGKILL arrives while your handler is halfway through a Stripe call.

If you would rather not maintain shell, concurrently -k -n web,worker "node dist/server.js" "node dist/worker.js" gets you the same two processes with prefixed logs, and it does forward signals. What it does not give you is the backoff restart loop, so a worker that crash-loops on a bad deploy takes the whole app with it. Pick deliberately.

Both processes write to one stream. On PandaStack that is /var/log/pandastack-app.log, served by the runtime-logs endpoint — hence the sed prefix, because within a week you will want to grep worker lines out of interleaved request logs.

Concurrency versus the pool versus the baked RAM

This is the mistake everyone makes exactly once. BullMQ's concurrency is how many jobs one Worker has in flight at a time, and it looks cheap to raise because it is async, not threaded. What is not cheap is what those jobs do: nearly every one checks out a Postgres connection. Set concurrency to 50 against a pool of 10 and forty handlers queue for a connection until the acquire timeout fires and throws an error that mentions BullMQ nowhere.

The worse version: the web process shares that database. A burst of jobs drains the pool, request handlers cannot get a connection either, and your API starts returning 500s because someone queued four thousand thumbnails. That is the actual outage co-location introduces, and it is preventable with napkin arithmetic.

// db.ts -- separate pools, sized from what each process actually does.
import { Pool } from "pg";

const isWorker = process.env.ROLE === "worker";

export const pool = new Pool({
  connectionString: process.env.DATABASE_URL,
  // Worker: concurrency + a little headroom for health checks and cron.
  // Web:    a fixed budget that a job burst can never eat into.
  max: isWorker ? Number(process.env.WORKER_CONCURRENCY ?? 5) + 2 : 10,
  connectionTimeoutMillis: 5_000,
  idleTimeoutMillis: 30_000,
});

// The arithmetic for ONE microVM:
//   web:    1 node process, pool max 10            = 10 connections
//   worker: 1 node process, concurrency 5 (+2)     =  7 connections
//   -------------------------------------------------------------
//   total                                          = 17
//
// Postgres default max_connections is 100. Seventeen is comfortable.
// Redo it with concurrency 50 and three instances: (10 + 52) x 3 = 186.
// You run out of connections long before you run out of CPU, and the
// stack trace will blame pg, not the concurrency number that caused it.

The third term is memory, and on Firecracker it is not negotiable at runtime. A template's RAM is baked into its snapshot — base is 4 GiB — so guest memory is a property of the template, not a per-app slider you nudge when the worker starts allocating. Measure RSS for both processes under a realistic burst first. A handler that loads a 200 MB CSV at concurrency 10 is a 2 GB spike, and the OOM killer does not care that the web server was the innocent party.

Start at concurrency 5. On a shared machine it is not a throughput dial, it is a claim on CPU, connections and heap that your request handlers are also using. If five genuinely cannot keep up, that is not a tuning problem — it is the signal the worker has outgrown co-location.

Graceful shutdown, locks, and why handlers must be idempotent

BullMQ has no transaction around your job. It has a lock: when a worker picks up a job it takes a lock key with a TTL of lockDuration and renews it while the handler runs. If renewals stop — the process died, the event loop was blocked by a synchronous CPU burn, Redis went away — the lock expires, the job is marked stalled, and another worker runs it again from the top.

That is the entire delivery model, and it is at-least-once. A deploy that kills a worker mid-job re-runs that job. A handler that charges a card must be safe to run twice, or an ordinary Tuesday release double-charges somebody.

// shutdown.ts -- give in-flight jobs a window, then stop cleanly.
import { worker } from "./worker";
import { server } from "./server";
import { connection } from "./lib/queue";

let closing = false;

async function shutdown(signal: string) {
  if (closing) return; // a second SIGTERM must not race the first
  closing = true;
  console.log(`[shutdown] ${signal}: draining`);

  // Stop accepting new HTTP work first; the LB has already stopped
  // sending it, this just closes keep-alives.
  server.close();

  // worker.close() stops FETCHING new jobs and waits for the in-flight
  // ones. Pass true to force-close immediately (jobs go back stalled).
  const drain = worker.close();

  // Hard cap below the platform's SIGTERM->SIGKILL window, or the
  // kernel decides your drain window for you.
  const budget = Number(process.env.DRAIN_SECONDS ?? 25) * 1_000;
  const timedOut = Symbol("timeout");
  const result = await Promise.race([
    drain.then(() => "drained"),
    new Promise((r) => setTimeout(() => r(timedOut), budget)),
  ]);

  if (result === timedOut) {
    console.warn("[shutdown] drain budget exceeded; jobs will stall+retry");
    await worker.close(true);
  }
  await connection.quit();
  process.exit(0);
}

for (const sig of ["SIGTERM", "SIGINT"]) {
  process.on(sig, () => void shutdown(sig));
}

Two numbers have to agree here and usually do not. The drain budget must be shorter than the platform's SIGTERM-to-SIGKILL window, or the kernel truncates you whatever the code intends. And lockDuration must exceed your slowest handler, or a perfectly healthy job has its lock expire, gets declared stalled, and starts a second copy of itself running concurrently with the first — with a log that gives no hint why.

• Derive an idempotency key from a stable identifier in the payload — an order id, a webhook delivery id — never from Date.now() or a UUID generated inside the handler.
• Make the side effect conditional on that key. An INSERT ... ON CONFLICT DO NOTHING against a jobs_done table is frequently the entire fix.
• Split a handler that does several irreversible things into several handlers that each do one, or a retry that failed while emailing the receipt will charge the card again.
• Raise lockDuration for genuinely slow work, or call job.updateProgress() periodically, which also renews the lock and gives you a progress bar for free.

Telling a live worker from a wedged one

Here is the failure mode co-location introduces that nothing in your monitoring is looking for. The worker dies — unhandled rejection, OOM kill, a Redis hiccup that outlives the reconnect. The web server keeps serving. Your health check hits the web port, gets a 200, everyone is satisfied. Jobs pile up for six hours and the first alert is a support ticket.

An HTTP check on the web port tells you the web port is up. Process liveness is barely better: a worker whose event loop is blocked by a runaway regex is very much running and doing nothing at all. The signal you want is queue depth over time plus a heartbeat only a functioning worker can produce.

Concretely: expose an endpoint that calls queue.getJobCounts("waiting", "active", "delayed", "failed") and checks the age of the oldest waiting job against a threshold. Separately, enqueue a trivial heartbeat job on a repeatable schedule whose handler only writes a timestamp; the endpoint returns 503 when that timestamp is older than a couple of intervals. Depth alone false-alarms during a legitimate burst; a heartbeat alone stays green while a poison job blocks a queue nobody watches. Together they catch both. Point an external uptime monitor at it, and alert on oldest-waiting-job age rather than raw queue length — length is a function of your traffic, age is a function of whether the worker is working.

Where to stop

Co-location has a real window, narrower than enthusiasm suggests. Decide the exit criteria now, while nothing is on fire, rather than at 2am with a queue-depth graph going vertical.

• CPU-heavy jobs starve the web process. Sharp image resizing, PDF rendering, a big CSV parse — anything that pegs a core takes it from your request handlers. Run your latency benchmark with the queue full and look at p99, not the median.
• The two workloads want different capacity curves. Web capacity tracks users, worker capacity tracks queue depth, and one process manager welds them into a single scaling decision that suits neither.
• A memory-hungry job is OOM-killing the server. Guest RAM is fixed by the template snapshot, so the fix is a different template or a different process, not a runtime knob.
• The queue has acquired its own on-call. Once someone gets paged specifically for jobs, shared blast radius is an argument you will lose every time.

And the one specific to modern hosting, said plainly: scale-to-zero is the wrong feature for a queue worker. A web app hibernates because an inbound request wakes it. A worker has no inbound request — it sits blocked on Redis. Either that blocking counts as activity and the machine never sleeps, so the feature buys you nothing, or the process is not running and your jobs just sit there until somebody notices. There is no clever third answer. Do not put an idle-hibernating app in front of a queue that must drain on schedule.

So buy the warm machine deliberately for latency-sensitive queues, or restructure tolerant work as a scheduled run that starts, drains what is waiting, and exits. That shape is cheap when a fresh machine restores from a snapshot with a p50 of about 179 milliseconds. The same reasoning covers fan-out — a nightly batch where each tenant is independent and one bad tenant should not wedge a shared worker.

from pandastack import Sandbox

# One tenant, one microVM, one batch. No shared worker to wedge, no
# noisy-neighbour CPU, and the VM is gone when the work is done.
sbx = Sandbox.create(template="base", ttl_seconds=300)
r = sbx.exec("node process-batch.js --tenant acme")
sbx.destroy()

Splitting the worker out is then a fifteen-minute change, not a migration: same repo, same build, same REDIS_URL and DATABASE_URL. Deploy a second app whose start command is node dist/worker.js, delete the background subshell, and redo the connection arithmetic.

The short version

Solve Redis first and honestly. Set maxRetriesPerRequest to null. Export the environment before you background anything, exec the web server so signals land where the platform looks, and wait on the worker in the trap. Size pools from concurrency, per process. Close the worker inside a budget shorter than the kill window, keep lockDuration above your slowest handler, and make every handler idempotent. Health-check queue age and a heartbeat, not the web port. Keep scale-to-zero away from the queue.

A background worker earns its own deployment when it has its own failure mode, its own scaling curve, or its own pager. Until then it is a second process on a machine you are already renting.