Webhook Replay and Relay: Ingest First, Process Later

The incident always reads the same way. A customer says their subscription never activated. You look in your database and there is nothing — no row, no error, no log line. You look in the provider's dashboard and there it is: the delivery, six attempts, all 500s, marked failed, spread over eleven hours that happened while you were asleep. The payload is still visible in their UI, which is the only reason you know what was in it. Your side of the wire has no memory of the event at all.

What went wrong is not the 500. Handlers throw; that is normal. What went wrong is that the only copy of that event lived in the provider's retry queue, on the provider's schedule, subject to the provider's patience. You outsourced the durability of your own business events to a company whose retry policy you did not choose and cannot change.

I'm Ajay; I build PandaStack, an open-source Firecracker microVM platform. We send webhooks (deploy events, quota warnings) and we consume a pile of them — GitHub push events drive our auto-deploys, Stripe events drive billing. This post is about the receiving layer I wish I had built on day one instead of month four: a relay. Verify, persist, acknowledge, then process on your own terms, with a replay button for when you get it wrong.

Why inline processing fails, precisely

The inline handler is the obvious first implementation and it is the right one for about a week. Request arrives, you verify the signature, you parse the JSON, you update the subscription, you send the welcome email, you return 200. One function, easy to read, easy to test.

It fails in four ways, and they compound.

1. Slowness becomes duplication. Your handler now sends an email, and the mail provider has a bad minute. Your handler takes twelve seconds. The sender's timeout is ten. From your side the work completed; from theirs it failed, so it retries — and you send a second email. Every second you spend inline is a second of duplicate risk.
2. Throwing becomes data loss on a timer. A bug in one branch of your handler returns a 500. Most providers retry with exponential backoff over a period of hours and then stop permanently. If you do not deploy a fix inside that window — which you cannot do if you are asleep, and cannot even start if you do not know — the event is gone.
3. You have no record to inspect. When it does go wrong, the thing you want most is the exact bytes that arrived. If you never stored them, your only copy is in a vendor dashboard with its own retention policy, and you are reconstructing an incident from a screenshot.
4. You cannot replay. Even when the provider offers a redeliver button, it is per-delivery, human-driven, and rate-limited. You cannot say "reprocess every subscription event from Tuesday between 09:00 and 11:00 against the fixed code" — which is exactly the sentence you will want to say.

None of these are exotic. They are the default outcome of putting business logic in the same call stack as an HTTP response you did not initiate.

The shape of the fix: a relay in two halves

Split the endpoint into two programs that share a table. The first is the receiver, and its entire job description is to be fast and boring: verify the signature over the raw bytes, write those bytes to durable storage, return 200. It contains no business logic whatsoever. It does not know what a subscription is. It cannot fail because of a bug in your billing code, because it does not import your billing code.

The second is the processor: a worker that reads unprocessed rows and does the actual work, with a retry policy you wrote, backed off however you like, running for as long as you like, and — critically — re-runnable on demand against events that are hours or weeks old.

That is the whole idea. It is not clever. The value is entirely in the seam: once the raw event is durable and acknowledged, every failure downstream of it is a bug you can fix and re-run, rather than an event you lost. You have converted "we dropped it" into "we have not processed it yet," which is a completely different class of problem.

Part 1: verify against the raw body, before anything touches it

This is the step people get wrong most often, and the failure mode is maddening because it is environment-dependent — works in tests, works in staging, 401s in production behind a different proxy.

The provider computed an HMAC over the exact byte sequence it put on the wire. Your framework, trying to help, parses that JSON into an object before your handler is called. That object does not remember whether the provider wrote a space after the colon, what order the keys were in, or how it escaped a non-ASCII character in a customer's name. Re-serialise it and you get different bytes, a different digest, and a signature failure with no useful error message.

• Express: apply the raw-body option on that route specifically, before any JSON middleware. Mounting the JSON parser globally and then trying to recover the raw body afterwards is a well-trodden path to an afternoon you will not get back.
• FastAPI/Starlette: read `await request.body()` first and parse it yourself. Do not type the handler parameter as a Pydantic model, because that parses before you see it.
• Go: read the `io.Reader` once into a buffer, verify, then unmarshal from the buffer. Wrap it in `http.MaxBytesReader` so a hostile body cannot exhaust memory — push payloads from GitHub get large, and an unbounded read on an unauthenticated route is its own bug.
• Anywhere: check what your proxy does. A gateway that decompresses, re-compresses, or normalises the body will break verification in one environment and not another, and you will blame the crypto for a week.

Three more details decide whether verification is real or decorative: compare digests in constant time (a plain string compare is a timing oracle), enforce a timestamp tolerance where the scheme provides a timestamp, and handle more than one active secret so you can rotate. Here is a receiver that does all of it, for the two signing schemes you are most likely to meet.

import base64, hashlib, hmac, os, time, uuid
from fastapi import APIRouter, Request, Response, HTTPException

router = APIRouter()
TOLERANCE = 300  # seconds of clock skew + transit we will accept

# Two active secrets, always. During a rotation both are set; afterwards
# PROVIDER_WEBHOOK_SECRET_OLD is empty and the loop below is a no-op.
def secrets_for(provider: str) -> list[bytes]:
    keys = [
        os.environ.get(f"{provider}_WEBHOOK_SECRET", ""),
        os.environ.get(f"{provider}_WEBHOOK_SECRET_OLD", ""),
    ]
    return [k.encode() for k in keys if k]


def verify_stripe_style(raw: bytes, header: str, secrets: list[bytes]) -> None:
    """Stripe-style: 'Stripe-Signature: t=<unix>,v1=<hex>' over '<t>.<raw>'."""
    parts: dict[str, list[str]] = {}
    for chunk in header.split(","):
        k, _, v = chunk.strip().partition("=")
        parts.setdefault(k, []).append(v)

    ts = parts.get("t", [""])[0]
    sigs = parts.get("v1", [])          # a header may carry several v1 values
    if not ts or not sigs:
        raise HTTPException(400, "malformed signature header")

    try:
        skew = abs(time.time() - int(ts))
    except ValueError:
        raise HTTPException(400, "malformed timestamp")
    if skew > TOLERANCE:
        raise HTTPException(400, "timestamp outside tolerance")

    signed = ts.encode() + b"." + raw   # sign the RAW bytes, never a re-dump
    for secret in secrets:
        expected = hmac.new(secret, signed, hashlib.sha256).hexdigest()
        for sig in sigs:
            if hmac.compare_digest(expected, sig):   # constant time
                return
    raise HTTPException(401, "bad signature")


def verify_github_style(raw: bytes, header: str, secrets: list[bytes]) -> None:
    """GitHub-style: 'X-Hub-Signature-256: sha256=<hex>' over the raw body.
    No timestamp in the scheme, so the delivery id is your replay guard."""
    if not header.startswith("sha256="):
        raise HTTPException(400, "malformed signature header")
    got = header[len("sha256="):]
    for secret in secrets:
        expected = hmac.new(secret, raw, hashlib.sha256).hexdigest()
        if hmac.compare_digest(expected, got):
            return
    raise HTTPException(401, "bad signature")


def verify_shopify_style(raw: bytes, header: str, secrets: list[bytes]) -> None:
    """Shopify-style: 'X-Shopify-Hmac-Sha256' is BASE64, not hex. Getting the
    encoding wrong is the single most common copy-paste bug in this file."""
    for secret in secrets:
        digest = hmac.new(secret, raw, hashlib.sha256).digest()
        if hmac.compare_digest(base64.b64encode(digest).decode(), header):
            return
    raise HTTPException(401, "bad signature")


@router.post("/webhooks/{provider}")
async def receive(provider: str, request: Request) -> Response:
    raw = await request.body()          # FIRST. Nothing parses before this.
    h = request.headers
    secrets = secrets_for(provider.upper())
    if not secrets:
        raise HTTPException(503, "webhook secret not configured")

    if provider == "stripe":
        verify_stripe_style(raw, h.get("stripe-signature", ""), secrets)
        event_id = None                 # extracted after verification, below
    elif provider == "github":
        verify_github_style(raw, h.get("x-hub-signature-256", ""), secrets)
        event_id = h.get("x-github-delivery")
    elif provider == "shopify":
        verify_shopify_style(raw, h.get("x-shopify-hmac-sha256", ""), secrets)
        event_id = h.get("x-shopify-webhook-id")
    else:
        raise HTTPException(404, "unknown provider")

    # Only NOW is it safe to parse. The bytes are proven; the JSON is derived.
    import json
    payload = json.loads(raw)
    event_id = event_id or payload.get("id") or str(uuid.uuid4())
    event_type = payload.get("type") or h.get("x-github-event") or "unknown"

    ingest(provider, event_id, event_type, raw, dict(h))  # one INSERT
    return Response(status_code=200)     # ack. no business logic ran.

Part 2: the event log, and the constraint that does the work

The table is the product. Everything else in this post is a consequence of having it. Design it so that the receiver's insert is a single statement with no reads, and so that duplicate delivery is a constraint violation rather than an `if`.

CREATE TABLE webhook_events (
  id              bigserial PRIMARY KEY,
  provider        text        NOT NULL,          -- 'stripe' | 'github' | ...
  -- The provider's own delivery/event id. Stable across THEIR retries, which
  -- is the entire basis of deduplication. Never hash the body instead: two
  -- genuinely distinct events can carry byte-identical payloads.
  event_id        text        NOT NULL,
  event_type      text        NOT NULL,
  -- Raw bytes exactly as received. bytea, not jsonb: you need these to
  -- re-verify a signature during an incident, and jsonb normalises.
  raw_body        bytea       NOT NULL,
  headers         jsonb       NOT NULL DEFAULT '{}'::jsonb,
  -- Provider-asserted event time, if any. Used for ordering decisions,
  -- NEVER for ordering guarantees.
  occurred_at     timestamptz,
  received_at     timestamptz NOT NULL DEFAULT now(),

  status          text        NOT NULL DEFAULT 'pending',
                              -- pending | processing | done | failed | skipped
  attempts        int         NOT NULL DEFAULT 0,
  next_attempt_at timestamptz NOT NULL DEFAULT now(),
  last_error      text,
  processed_at    timestamptz,

  -- THE line that makes at-least-once delivery survivable.
  CONSTRAINT webhook_events_dedupe UNIQUE (provider, event_id)
);

-- Claim query: the worker's hot path. Partial index keeps it tiny even when
-- the table holds years of done rows.
CREATE INDEX webhook_events_due
  ON webhook_events (next_attempt_at)
  WHERE status IN ('pending', 'failed');

-- Human path: "show me everything for this customer on Tuesday".
CREATE INDEX webhook_events_lookup
  ON webhook_events (provider, event_type, received_at DESC);

-- Fan-out: one inbound event, N independent consumers, each with its own
-- success/failure state. Without this, a broken analytics consumer blocks
-- the billing consumer, and "replay" becomes all-or-nothing.
CREATE TABLE webhook_deliveries (
  event_id_fk     bigint      NOT NULL REFERENCES webhook_events(id) ON DELETE CASCADE,
  consumer        text        NOT NULL,          -- 'billing' | 'crm' | 'analytics'
  status          text        NOT NULL DEFAULT 'pending',
  attempts        int         NOT NULL DEFAULT 0,
  next_attempt_at timestamptz NOT NULL DEFAULT now(),
  last_error      text,
  processed_at    timestamptz,
  PRIMARY KEY (event_id_fk, consumer)
);

The receiver's insert is then `INSERT ... ON CONFLICT (provider, event_id) DO NOTHING`. A duplicate delivery becomes a zero-row write and a 200, which is exactly right: the provider asked "did you get this," and the honest answer is yes. Returning an error for a duplicate means "retry me," which is the opposite of what you meant, and some providers will eventually disable an endpoint that keeps failing.

Two design notes I would argue for and you may disagree with. First, store `raw_body` as bytes rather than parsed JSON. During an incident the question is often "is our verification wrong, or did they send something odd," and you can only answer that by re-running the HMAC over the original bytes. A jsonb column has already thrown that away. Second, keep the event log and your application state in the same database if you possibly can, so that marking an event processed and applying its effect happen in one transaction. Splitting them across two stores reintroduces the exactly-once problem you just solved, in a place where it is much harder to see.

Part 3: the processor, which is also the replay button

Here is the part that surprises people: there is no separate replay system. Replay is the normal processing path pointed at rows you choose. If reprocessing an event is safe — and it is, because your handlers are idempotent, because they have to be anyway — then "replay" is one UPDATE that sets some rows back to pending.

import json, time, psycopg
from datetime import datetime, timedelta, timezone

# Your own retry policy, not the provider's. Tune it to your dependencies.
BACKOFF = [10, 60, 300, 1800, 7200, 28800]   # seconds; ~10h across 6 attempts
MAX_ATTEMPTS = len(BACKOFF)

CLAIM = """
UPDATE webhook_events SET status = 'processing', attempts = attempts + 1
WHERE id = (
  SELECT id FROM webhook_events
  WHERE status IN ('pending', 'failed') AND next_attempt_at <= now()
  ORDER BY next_attempt_at
  FOR UPDATE SKIP LOCKED          -- many workers, no double-claim, no queue
  LIMIT 1
)
RETURNING id, provider, event_id, event_type, raw_body, occurred_at, attempts;
"""


def run_once(conn) -> bool:
    with conn.cursor() as cur:
        cur.execute(CLAIM)
        row = cur.fetchone()
        if row is None:
            return False
        eid, provider, event_id, event_type, raw, occurred_at, attempts = row

        try:
            payload = json.loads(bytes(raw))
            # Handlers run INSIDE the transaction that marks the row done, so
            # "applied the effect" and "recorded that we applied it" commit
            # together or not at all. This is the whole exactly-once trick.
            handle(cur, provider, event_type, event_id, payload, occurred_at)
            cur.execute(
                "UPDATE webhook_events SET status='done', processed_at=now(),"
                " last_error=NULL WHERE id=%s", (eid,))
            conn.commit()
        except SkipEvent as e:
            conn.rollback()
            with conn.cursor() as c2:   # stale/superseded: not a failure
                c2.execute("UPDATE webhook_events SET status='skipped',"
                           " last_error=%s WHERE id=%s", (str(e), eid))
            conn.commit()
        except Exception as e:
            conn.rollback()
            delay = BACKOFF[min(attempts - 1, MAX_ATTEMPTS - 1)]
            terminal = attempts >= MAX_ATTEMPTS
            with conn.cursor() as c2:
                c2.execute(
                    "UPDATE webhook_events SET status=%s, last_error=%s,"
                    " next_attempt_at=%s WHERE id=%s",
                    ("failed", repr(e)[:2000],
                     datetime.now(timezone.utc) + timedelta(seconds=delay), eid))
            conn.commit()
            if terminal:
                alert(f"webhook {provider}/{event_id} exhausted retries: {e!r}")
        return True


# --- Replay is not a separate system. It is this SELECT, re-armed. ---------
REPLAY = """
UPDATE webhook_events
   SET status = 'pending', attempts = 0, next_attempt_at = now(), last_error = NULL
 WHERE provider = %(provider)s
   AND (%(types)s::text[] IS NULL OR event_type = ANY(%(types)s))
   AND received_at BETWEEN %(start)s AND %(end)s
   AND status IN ('failed', 'skipped', 'done')
RETURNING id;
"""
# "Reprocess every Stripe subscription event from Tuesday 09:00-11:00 against
# the fixed code" is one call. That sentence is the reason this table exists.

if __name__ == "__main__":
    with psycopg.connect(DSN) as conn:
        while True:
            if not run_once(conn):
                time.sleep(1.0)

`FOR UPDATE SKIP LOCKED` is doing a lot here: it turns an ordinary Postgres table into a work queue that several workers can drain concurrently without double-processing, without a broker, and without a second piece of infrastructure to operate. For webhook volumes — which are almost always lower than people assume — this is entirely sufficient, and it keeps the claim and the effect in one transactional world.

Note the `skipped` status. It exists because "this event is stale and should not be applied" is a legitimate outcome, not a failure, and conflating the two means your alerting cries wolf every time a late retry arrives for a subscription that has since been cancelled.

Ordering: you do not get it, so stop designing for it

Nothing in the webhook contract promises order. An `updated` event can land before the `created` event for the same object. A cancellation can beat the renewal it cancels. A retry from forty minutes ago can arrive after three newer events for the same entity have already been applied. Add a relay with parallel workers and you have introduced your own reordering on top of theirs.

Handlers written as a state machine driven by arrival order corrupt data quietly and confidently, which is the worst combination. Three defences, in order of how much I like them:

1. Treat the event as a hint, not a delta. On `subscription.updated`, do not apply the payload's changes — call the provider's API, fetch the object's current state, and reconcile to that. The event tells you what to look at; the API tells you what is true. This is immune to ordering entirely, and it also fixes the case where you missed an event completely.
2. Version-guard the write. Where events carry a sequence number or an event timestamp, store the last applied one on the row and make the update conditional: `WHERE last_event_at < %s`. An older event that arrives late loses the compare and is marked skipped rather than applied.
3. Serialise per entity, not globally. If you genuinely must apply deltas, take a per-object advisory lock so two events for the same subscription cannot interleave, while events for different subscriptions still run in parallel. This preserves throughput and only constrains what has to be constrained.

The reconcile-from-API approach is unfashionable because it costs an extra request per event, and it is the one I would pick nine times out of ten. Every reordering bug I have debugged would have been prevented by it, and the cost is a few milliseconds against a failure mode that produces wrong money.

Clock skew, tolerance windows, and the two meanings of replay

The timestamp in a signature exists to stop an attacker who captured one valid delivery from replaying it forever. So you enforce a tolerance — five minutes is the usual default. Then two things go wrong in opposite directions.

Too tight and you reject legitimate traffic. A provider's retry after a long backoff arrives with the original timestamp and fails your window, so a recoverable retry becomes permanent loss. Well-behaved senders re-sign on retry with a fresh timestamp, but you should not assume it. Too loose and the guard stops guarding. Somewhere around five minutes is the honest compromise, and if you are rejecting real deliveries at that width, the problem is not the width.

It is usually your clock. Drift makes valid deliveries fail with "invalid signature," which sends the whole team into the crypto when the bug is NTP. Before you touch the HMAC code, print the skew: log `abs(now - t)` on every rejection and you will find out in one delivery whether you have a signature bug or a time bug. We hit a microVM-flavoured version of this directly — a guest restored from a snapshot wakes believing it is whatever time the snapshot was taken, so every signature it checked was "too old" and every outbound TLS handshake failed on certificate validity. We now force a clock sync on restore. Anywhere your handler runs snapshotted, suspended, or restored, check the guest clock first.

The word "replay" is doing double duty in this post and it is worth separating the meanings, because they pull against each other. Adversarial replay is someone re-sending a captured signed request to make you process it twice; your defences are the timestamp window and the dedupe constraint. Operational replay is you deliberately reprocessing a stored event because your code was wrong; that happens entirely inside your system, on rows you already verified, and never re-enters the signature path.

Secret rotation without a maintenance window

The reason people never rotate webhook secrets is that the naive procedure has a gap: you change the secret in the provider's dashboard, and every delivery in flight — plus every retry of an older delivery still signed with the old key — fails until you deploy the new value. So it never happens, and the secret from 2023 is still in an env file that four ex-employees had access to.

Accept two secrets and the gap disappears. The verifier in the code above loops over a list; that is the entire mechanism. The procedure:

1. Deploy the two-secret verifier first, with the new slot empty. Nothing changes behaviourally; you are just making the next step non-breaking.
2. Generate the new secret at the provider. Where the provider supports overlapping secrets, add rather than replace, and note that some signature headers can carry several signatures at once — that field is plural precisely for this.
3. Set the new value as the primary and move the current one to the OLD slot. Deploy. You now accept both.
4. Wait longer than the provider's full retry horizon — hours, not minutes — so any delivery signed with the old key has finished retrying.
5. Remove the old secret at the provider, then clear the OLD slot and deploy. Rotation complete, zero rejected deliveries.

Use a distinct secret per environment. A staging endpoint sharing production's secret means a captured production payload replays cleanly into staging, and a stale staging fixture can be pointed at production. One secret per environment, per provider, and an alert on any signature rejection so a botched rotation is loud rather than silent.

Fan-out: one event, several consumers, independent failure

The second thing that always happens: a Stripe invoice event needs to update billing, sync the CRM, post to Slack, and feed an analytics table. The tempting implementation is one handler doing four things in sequence, and it is wrong for the same reason inline processing was wrong — the slowest and flakiest consumer sets the reliability of all four. If the CRM API is down, your billing update retries too, and the sixth attempt re-posts to Slack.

That is what the `webhook_deliveries` table in the schema above is for. On successful ingest, insert one row per interested consumer; each carries its own status, attempt count, and backoff. Then:

• A failing CRM sync retries alone. Billing committed an hour ago and is not touched.
• Replay gets a scope. "Re-run only the analytics consumer for last week" is a WHERE clause, not a migration script.
• Adding a consumer later is backfill, not archaeology. Insert delivery rows for the historical events you care about and let the worker drain them — the raw bodies are still there.
• Per-consumer idempotency stays honest. Each consumer's handler must tolerate being re-run, which is a smaller and much more testable requirement than "the whole pipeline is idempotent".

The cost is one more table and a bit of insert fan-out. I would not build it on day one for a single consumer. I would build it the moment there is a second, because retrofitting it after an incident means reasoning about which half of a compound handler already ran.

Local development, and why staging wants a relay rather than a tunnel

The tunnel problem is well known: providers cannot reach localhost, so you run ngrok or Cloudflare Tunnel or your provider's CLI relay, paste the public hostname into a dashboard, and iterate. For the first hour of an integration this is the right tool and I use it constantly. Where a provider ships an outbound-dialling CLI — Stripe's `listen` is the archetype — prefer it, because nothing inbound is opened on your machine and you still exercise real signature verification against a session-scoped secret.

Where tunnels break down is staging, and it is not a subtle failure. The hostname dies when the laptop sleeps, so the provider records failures against a URL nobody owns. Two developers cannot share one registration. CI cannot use it, because there is no laptop in CI. And a tunnel points a public hostname at a machine holding cloud credentials, a production database password in a dotfile, and an SSH agent — answered by a dev server with debug mode on and possibly signature checks disabled because you were testing.

A persistent relay fixes all of that with the same design you already built. Run the receiver — just the receiver, the boring one with no business logic — on real infrastructure at a stable URL. Register that URL with every provider, once, and stop touching provider dashboards. It verifies, stores, acknowledges. Then developers pull unprocessed events down to their machines and run the processor locally against real captured bodies, or you run a second processor pointed at the staging database with `consumer = 'dev'` delivery rows. Nobody's laptop is on the internet, the registration outlives every branch, and CI drives the same processor the same way.

# The staging relay: a stable URL you register once, per provider.
#   https://relay.staging.example.com/webhooks/stripe
#   https://relay.staging.example.com/webhooks/github
#
# Developers never register anything. They pull real captured bodies down
# and drive their local processor with them.

# 1. What arrived, and what happened to it?
psql "$STAGING_DSN" -c "
  SELECT id, provider, event_type, status, attempts, left(last_error, 60)
    FROM webhook_events
   WHERE received_at > now() - interval '2 hours'
   ORDER BY received_at DESC LIMIT 20;"

# 2. Pull the exact bytes of one event as a local fixture.
psql "$STAGING_DSN" -At -c \
  "SELECT encode(raw_body, 'escape') FROM webhook_events WHERE id = 4812;" \
  > fixtures/invoice_paid_4812.json

# 3. Re-arm one event for the worker. This is the replay button, in anger.
psql "$STAGING_DSN" -c "
  UPDATE webhook_events
     SET status='pending', attempts=0, next_attempt_at=now(), last_error=NULL
   WHERE id = 4812;"

# 4. Re-arm a whole window after shipping a fix — the sentence you will
#    actually want to say at 2am.
psql "$STAGING_DSN" -c "
  UPDATE webhook_events
     SET status='pending', attempts=0, next_attempt_at=now(), last_error=NULL
   WHERE provider='stripe'
     AND event_type LIKE 'customer.subscription.%'
     AND received_at >= '2026-09-02T09:00Z'
     AND received_at <  '2026-09-02T11:00Z'
     AND status IN ('failed','skipped');"

The reason to run staging's relay on infrastructure that creates and destroys cheaply is that you will want one per branch eventually — a processor build under test, driven by the same captured fixtures, with its own database. On PandaStack that is a sandbox per branch: creates are snapshot-restore at a p50 of 179ms, so a per-branch environment is ordinary rather than a budget line, and compute is $0.054 per vCPU-hour and $0.0162 per GiB-hour for the minutes it exists. Whether you use ours or somebody else's, the property that matters is that an environment is cheap enough to be disposable, because a shared staging relay that everybody's half-finished processor drains at once is its own kind of incident.

When this is overkill, and what I would skip

I would rather you skipped this than cargo-culted it. If you receive one webhook type from one provider and the handler updates one boolean, the inline version is fine, and adding a table and a worker to it is a way of feeling productive rather than being productive. The threshold I would use: does a lost event cost money, access, or a support ticket? If not, ship the handler and move on.

Even when you do build it, build it in order. The receiver and the events table with the unique constraint are the whole win — that is one afternoon, and it converts data loss into a backlog. The per-consumer fan-out table waits for the second consumer. A separate broker waits until Postgres with `SKIP LOCKED` actually stops keeping up, which for webhook volumes is later than you think and possibly never.

And be honest about what the relay does not fix. It does not give you ordering. It does not make a bad handler correct — an event you reprocess against still-broken logic just fails again, more visibly. It adds a database write to a path that had none, so a database outage is now a webhook outage, and your receiver should return a 5xx in that case so the provider's retry does the right thing. What it buys you is the ability to be wrong and recover, which is the only property that has ever mattered to me at 2am.

Inline processing means the provider's retry policy is your durability guarantee. A relay means the only thing you can lose is the ability to write one row.