How to Run a Sidekiq Worker Alongside Your Rails App

Count the jobs your Rails app actually enqueues. For most apps the honest number is a few thousand a day: welcome emails, a webhook retry, a nightly report, a few thumbnails. At 200ms a job that is thirteen minutes of CPU, and on most platforms you pay for it with a whole second always-on service.

I'm Ajay, I build PandaStack. On a real Linux microVM you have a machine, not a slot, so a second process is free. Here is how to run Sidekiq next to Puma, and where to stop.

First, the Redis question — answered plainly

Sidekiq does not merely prefer Redis. Its queues are Redis lists, its retry and scheduled sets are sorted sets, its heartbeats are hashes with TTLs. There is no database adapter. A database-backed queue is Solid Queue: a different tool.

Option one, and my recommendation if losing an enqueued job would cost you money or trust: a managed Redis over TLS. Put the rediss:// URL in REDIS_URL and let someone else own replication and backups.

Option two: run Redis inside the same sandbox. You have root on a full Ubuntu userspace, so it is an apt install. The tradeoff, without spin: a co-located Redis with appendonly yes on a durable volume is fine for jobs you can afford to retry, and wrong for jobs you cannot afford to lose.

# Co-located Redis, configured to actually persist.
apt-get install -y --no-install-recommends redis-server

# /etc/redis/redis.conf
#   appendonly yes
#   appendfsync everysec        # <= 1s of writes at risk on a hard kill
#   dir /mnt/data/redis         # a DURABLE volume, not the ephemeral rootfs
#   maxmemory 512mb
#   maxmemory-policy noeviction # NEVER allkeys-lru: it will evict your queue

redis-server /etc/redis/redis.conf --daemonize yes

That noeviction line quietly ruins people. An LRU policy on a Redis holding a queue means Redis deletes jobs under memory pressure, silently. Set noeviction and let enqueue fail loudly.

What you accept: no replica, so a lost machine is a lost queue; backups are yours; everysec fsync can drop the last second of enqueues. Fine for a thumbnail the user can re-trigger. Not fine for a payment, where the enqueue is the only record the work was requested — write that to Postgres and make the job a pointer to the row.

Configure Sidekiq for a machine it has to share

The default sidekiq.yml assumes a machine that belongs to Sidekiq. Yours does not. Turn concurrency down and weight the queues, so low-value work cannot starve password resets.

# config/sidekiq.yml
:concurrency: <%= ENV.fetch("SIDEKIQ_CONCURRENCY", 5) %>
:timeout: 25          # seconds to finish in-flight work on SIGTERM

# Weighted queues: 'critical' is polled 5x as often as 'low'.
:queues:
  - [critical, 5]
  - [default, 2]
  - [mailers, 2]
  - [low, 1]

production:
  # Keep this deliberately small. Every thread here is a thread NOT
  # available to Puma, and a DB connection you must account for below.
  :concurrency: <%= ENV.fetch("SIDEKIQ_CONCURRENCY", 5) %>

Five threads, not twenty-five. On a shared machine concurrency is not a throughput dial, it is a claim on the CPU and memory Puma is using. If five cannot keep up, the worker has outgrown co-location.

Start both processes from one command

The shape that works: Sidekiq supervised in the background under a restart loop, Puma in the foreground via exec, so it is the process the platform signals.

#!/bin/sh
# bin/start -- worker in the background, web in the foreground.
set -e

# EXPORT before anything is backgrounded. Build and start commands run as
# a non-login 'sh -c', nothing sources a profile, and a variable that is
# only assigned (not exported) never reaches a child process.
export RAILS_ENV=production
export RAILS_LOG_TO_STDOUT=1
export MISE_DATA_DIR=/opt/mise MISE_CONFIG_DIR=/opt/mise
export PATH=/opt/mise/shims:$PATH

( backoff=1
  while true; do
    bundle exec sidekiq -C config/sidekiq.yml 2>&1 | sed -u 's/^/[sidekiq] /'
    echo "[sidekiq] exited (rc=$?), restarting in ${backoff}s" >&2
    sleep "$backoff"
    backoff=$(( backoff < 30 ? backoff * 2 : 30 ))
  done ) &
WORKER=$!

# Pass the deploy's SIGTERM through to Sidekiq so it gets to drain.
trap 'kill -TERM "$WORKER" 2>/dev/null' TERM INT

exec bundle exec puma -C config/puma.rb

The export block must come before the subshell and before any setsid, because a child inherits the environment as it existed at fork time. Assign PORT without exporting it and the child sees nothing: Puma binds to its default while the platform probes another port. Clean logs, failed health check. The exec matters for the same reason — without it Puma is a grandchild of the shell and never sees SIGTERM.

Both processes write to one stream. On PandaStack that is /var/log/pandastack-app.log, served by the runtime-logs endpoint. Hence the sed prefix: you will want to grep worker lines out of interleaved request logs.

Your health check does not know the worker exists

Here is the failure mode co-location introduces. Sidekiq dies — an exception in middleware, an OOM kill, a Redis storm. Puma keeps serving, /up returns 200, the platform is satisfied. Jobs pile up for six hours and a customer notices first.

An HTTP check on the web port tells you the web port is up. Check the worker directly: every live Sidekiq process heartbeats to Redis, and a stale beat is the signal you want.

# app/controllers/health_controller.rb
require "sidekiq/api"

class HealthController < ApplicationController
  STALE_AFTER = 60 # seconds

  def workers
    live = Sidekiq::ProcessSet.new.count do |p|
      Time.now.to_i - p["beat"].to_i < STALE_AFTER
    end
    latency = Sidekiq::Queue.all.map(&:latency).max.to_f

    problems = []
    problems << "no live sidekiq process" if live.zero?
    problems << "oldest job waiting #{latency.round}s" if latency > 300

    if problems.empty?
      render json: { ok: true, processes: live, latency: latency.round }
    else
      # 503 so an external uptime monitor pages someone.
      render json: { ok: false, problems: problems }, status: :service_unavailable
    end
  end
end

Point an uptime monitor at /health/workers, not only /up. Queue latency, the age of the oldest waiting job, is the number worth alerting on: it catches a worker that is alive but wedged, which a process count never will. The alternative is a heartbeat job touching a row every minute.

Concurrency versus the connection pool: do the arithmetic

This is the mistake everyone makes once. Concurrency is a thread count, every thread that touches ActiveRecord checks out a connection, and Rails' default pool is 5. Concurrency 25 against a pool of 5 leaves twenty threads queuing until checkout_timeout raises ActiveRecord::ConnectionTimeoutError — an error that reads like a database problem and is a config problem.

The rule is one line: pool must be at least the thread count in that process. Puma and Sidekiq have different thread counts, so they need different pools from the same database.yml.

# config/database.yml
default: &default
  adapter: postgresql
  encoding: unicode
  # Puma's number. Threads are per-process, so the pool is per-process too.
  pool: <%= ENV.fetch("RAILS_MAX_THREADS", 5) %>
  checkout_timeout: 5

production:
  <<: *default
  url: <%= ENV["DATABASE_URL"] %>

# config/initializers/sidekiq.rb -- resize the pool for the WORKER only:
#
#   Sidekiq.configure_server do |config|
#     config.redis = { url: ENV.fetch("REDIS_URL") }
#     db = ActiveRecord::Base.connection_db_config.configuration_hash
#     ActiveRecord::Base.establish_connection(
#       db.merge(pool: Sidekiq.default_configuration[:concurrency] + 2)
#     )
#   end
#
# The arithmetic for ONE machine:
#   web:    2 puma workers x 5 threads      = 10 connections
#   worker: 1 sidekiq x (5 concurrency + 2) =  7 connections   <- +2 for the
#   ------------------------------------------------------        heartbeat
#   total                                   = 17                  and poller
#
# Postgres default max_connections is 100. Seventeen is comfortable.
# Redo it with concurrency 25 and three app instances: (10 + 27) x 3 = 111.
# You run out of connections before you run out of CPU, and the exception
# will name ActiveRecord, not Sidekiq.

Memory follows the same logic: each Puma worker is a full copy of the app, and Sidekiq threads allocate in one shared heap. Measure RSS under load before picking a memory tier — on Firecracker, guest RAM is fixed by the template snapshot.

Graceful shutdown, and why jobs must be idempotent

On SIGTERM, Sidekiq stops fetching and gives in-flight jobs the -t timeout — 25 seconds by default, the :timeout key above. Anything still running then goes back on the queue and runs again from the top on the next boot.

That is the entire safety model. Sidekiq is at-least-once: a deploy that kills the worker mid-job re-runs it. A job that charges a card must be safe to run twice, or a routine release double-charges somebody.

• Make the timeout longer than your slowest job, or the job shorter. If the SIGTERM-to-SIGKILL window is 30 seconds, a minute-long job dies every deploy whatever the config says.
• Derive an idempotency key from an order or event id and make the side effect conditional on it. An insert with ON CONFLICT DO NOTHING is often the whole fix.
• Split a job that does several irreversible things into jobs that each do one, or a retry that failed on the receipt charges the card again.

When to split the worker out anyway

Co-location has a narrow window. Leave it when any of these becomes true, and decide now rather than mid-incident.

• CPU-heavy jobs starve the web process. Image processing, PDF generation, CSV imports: anything that pegs a core steals it from Puma. Run your latency benchmark with the queue full and watch p99.
• The two workloads want different hardware: web capacity tracks users, worker capacity tracks queue depth, and co-location welds them together.
• A memory-hungry job is OOM-killing your web workers: the job that allocates 2 GiB takes request handlers with it.
• The queue has its own on-call. Shared processes make blast-radius arguments impossible to win.

And the one specific to modern hosting: scale-to-zero and a Sidekiq worker pull in opposite directions. A web app hibernates because a request wakes it, on PandaStack in about 1.2 seconds. A worker has no inbound request. It blocks on Redis, which is either activity that keeps the machine awake, or it is not running and nobody drains the queue.

There is no clever resolution. Buy the warm machine deliberately, or move tolerant work to a scheduled job that starts, drains and exits — snapshot-restore create is a p50 of 179ms, so a machine per drain is cheap. Do not put an idle-hibernating app in front of a queue that must drain on schedule.

Splitting out is then a fifteen-minute change: same repo, same build, same REDIS_URL. Deploy a second app running bundle exec sidekiq, drop the background block.

The short version

Solve Redis first and honestly. Keep concurrency small and weight the queues. Export the environment before you background anything, and exec Puma so signals land where the platform is looking. Size the pool from thread counts, per process. Health-check the worker, not the web port. Make jobs idempotent.

A background worker earns its own deployment when it has its own failure mode, its own scaling curve, or its own pager. Until then it is a second process on a machine you are already paying for.