Running background workers next to your web app

There's a moment in every application's life where someone needs to send an email after a signup, or resize an uploaded image, or generate a report that takes forty seconds. The first implementation does it inline in the request handler. It works, until the day it doesn't, and then it fails in one of a small number of predictable ways.

This post is about where that work should live once you take it out of the request, and what each option costs you.

Why inline stops working

• Timeouts. Load balancers, proxies and browsers all give up somewhere between 30 and 120 seconds. Your forty-second report becomes a 504 on a slow day, and the user retries, and now it's running twice.
• Deploys interrupt it. A deploy that replaces the running version kills whatever it was doing mid-flight. If the work isn't idempotent and isn't retried, it's simply lost, and nothing reports that it was lost.
• One user's heavy job degrades everyone. A request handler blocked on image processing is a worker not serving anyone else. Enough of them and the whole app looks down.
• No retries. A transient failure calling a third-party API becomes a user-visible error rather than something the system quietly handles on the second attempt.

The fix is always the same shape: the request handler writes down what needs doing and returns immediately, and something else does it. The interesting question is what that something else is.

Option 1: same process, background task

The lowest-effort version. FastAPI has `BackgroundTasks`, Node has whatever you get from not awaiting a promise, Rails has `perform_now` misused as `perform_later`.

@app.post("/signup")
async def signup(user: User, background: BackgroundTasks):
    await db.create_user(user)
    background.add_task(send_welcome_email, user.email)  # returns immediately
    return {"ok": True}

This is fine for work that is genuinely optional and genuinely fast — under a second, no correctness impact if it's dropped. The moment it matters, it's wrong, because the task lives only in the memory of a process you are about to restart on your next deploy. There is no queue, no retry, and no record that the work existed.

Option 2: separate process, same machine

The web server and a worker run side by side, coordinating through a durable queue — Postgres, Redis, whatever you already run. This is where most applications should stop, and a lot of teams skip past it because it feels insufficiently sophisticated.

// worker.js — a separate entrypoint, same codebase, same deploy
import { Worker } from "bullmq";

const worker = new Worker("emails", async (job) => {
  await sendEmail(job.data);
}, { connection: { url: process.env.REDIS_URL } });

// Deploys will kill this. Finish in-flight work first.
process.on("SIGTERM", async () => {
  await worker.close();   // stop taking new jobs, drain the current one
  process.exit(0);
});

The advantages are real: one codebase, one deploy, shared configuration, no network hop to your own code, and the worker's failures are visible in the same logs you already read. The disadvantage is that web and worker scale together and compete for the same CPU and memory, which is only a problem once one of them is genuinely large.

Whichever process manager you use, that SIGTERM handler is not optional. A worker that ignores it loses every in-flight job on every deploy, and because the jobs disappear rather than erroring, the loss is silent.

Option 3: worker as its own deployment

Same repository, same build, different start command. On PandaStack that's two apps pointed at the same git repo — the web one runs your server, the worker one runs the consumer, and both get the same environment and the same database credentials.

# Web app
pandastack apps create --name api --git-url https://github.com/acme/app \
  --start-cmd 'node dist/server.js'

# Worker: same repo, same build, different entrypoint
pandastack apps create --name api-worker --git-url https://github.com/acme/app \
  --start-cmd 'node dist/worker.js'

pandastack apps env set api-worker DATABASE_URL "$DATABASE_URL"
pandastack apps env set api-worker REDIS_URL "$REDIS_URL"

You get independent scaling and independent failure — a worker stuck in a crash loop doesn't take your website down, and a traffic spike on the web tier doesn't starve the queue. The cost is coordination: two things to deploy, and a window during a deploy where the two versions differ. Job payloads become an interface between versions, with all the compatibility rules that implies.

The scale-to-zero problem

This one bites specifically on platforms that sleep idle applications, and it's worth understanding before you deploy a worker to one.

Web apps sleep well because they wake on an inbound HTTP request — the request is the wake signal, and the platform can hold it while the app starts. A queue consumer has no inbound request. It wakes itself up by polling, which means either it never idles (polling is activity) or it idles and then nobody is listening when a job arrives.

• For latency-tolerant work, run the worker on a schedule instead of continuously. A cron trigger every five minutes that drains the queue and exits costs almost nothing and sleeps properly between runs.
• For latency-sensitive work, make the enqueue side push rather than the worker poll. An HTTP endpoint on the worker app turns job delivery into a request, which is exactly the signal a sleeping app can wake on.
• Keep genuinely continuous consumers awake deliberately, and price that in. A worker that must respond in milliseconds is a thing you are choosing to run all the time.

The push variant is usually the right answer and is underused. It also composes better with retries, because the enqueuer gets an HTTP status code rather than dropping a message into a queue and hoping.

Scheduled work is a different problem

Cron and queues get conflated because both are 'background', but they fail differently. A queue's problem is throughput and delivery. A schedule's problem is that it must run exactly once at the right time, on exactly one machine.

Running cron inside your web app is the classic mistake: scale to three instances and your nightly billing job runs three times. If your framework's scheduler doesn't do leader election, use a platform-level scheduler that guarantees a single execution, or take an advisory lock as the very first thing the job does.

-- Cheap single-execution guard for any scheduled job
SELECT pg_try_advisory_lock(hashtext('nightly-billing')) AS got_it;
-- if false, another instance is already running it: exit quietly

Picking one

Start at option two. A separate worker process on the same deploy, backed by a durable queue, handles a genuinely large amount of traffic and costs you almost no operational complexity. Move to a separate deployment when the web tier and worker tier want different amounts of hardware, or when worker crashes have started affecting request latency.

And whichever you choose, the two things that determine whether it's reliable are unglamorous: handle SIGTERM so deploys drain instead of dropping, and make every job idempotent so a retry is safe. Get those right and the rest is a scaling question rather than a correctness one.