How to Receive Webhooks for Deploys and Quota Events
The polling loop starts innocently. You trigger a deploy from CI, then poll the deployment endpoint every few seconds to see whether it finished, because the alternative is a job that exits before you know the answer. It works. Then you add a Slack notification, so now two things poll. Then someone wants an alert when the workspace approaches its quota, and that polls too, and your platform's rate limiter starts to have opinions.
Webhooks replace all of that with one HTTP endpoint you own. This guide walks the whole path — registering an endpoint, verifying signatures without introducing a timing bug, handling retries idempotently, and debugging the delivery that did not arrive — using PandaStack's webhook API as the concrete example. The patterns transfer to any platform that signs its deliveries.
Step 1: register an endpoint
# What can I subscribe to?
curl -s https://api.pandastack.ai/v1/webhooks/events \
-H "Authorization: Bearer $PANDASTACK_API_KEY"
curl -s -X POST https://api.pandastack.ai/v1/webhooks/endpoints \
-H "Authorization: Bearer $PANDASTACK_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://hooks.example.com/pandastack",
"events": ["deployment.succeeded", "deployment.failed", "quota.warning"],
"description": "prod deploy notifications"
}'The response includes a secret. Store it now — it is returned on create and on a single-endpoint GET, and it is the only thing that lets you distinguish a real delivery from someone POSTing JSON at a URL they guessed. Treat it exactly like a database password: environment variable or secret manager, never the repo.
Subscribe narrowly. Every event you take is code you have to handle, and 'we subscribed to everything and filtered in the handler' is how you end up with a handler nobody understands. The current catalogue covers deployment lifecycle, database failover, quota warnings and exhaustion, plus a test event you can fire on demand.
Step 2: know what arrives
POST /pandastack HTTP/1.1
X-PandaStack-Event: deployment.succeeded
X-PandaStack-Delivery: wd_9f2a1c4e8b03
X-PandaStack-Signature: t=1774483200,v1=6b1c...af
{
"id": "wd_9f2a1c4e8b03",
"type": "deployment.succeeded",
"created_at": "2026-08-26T09:20:00Z",
"workspace": "acme",
"data": { "app_id": "app_...", "deployment_id": "dep_...", "git_commit": "..." }
}A stable envelope — id, type, created_at, workspace, data — with the event-specific payload nested under data. Write your handler to dispatch on type and to ignore fields it does not recognise, because new fields inside data are additive changes that should never break you.
Step 3: verify the signature (the part people get wrong)
The signature header carries a timestamp and an HMAC: t is the Unix time the delivery was signed, and v1 is hex(HMAC-SHA256(secret, t + '.' + raw_body)). Three details decide whether your verification is real or decorative.
- Sign the raw bytes, not a re-serialised object. If your framework parses JSON before you see it, re-encoding will reorder keys or change spacing and every signature will fail. Capture the raw body.
- Compare in constant time. A plain string comparison leaks how many leading bytes matched, which is enough to forge a signature given enough attempts. Every language has a constant-time compare — use it.
- Reject old timestamps. Without a freshness window, a delivery captured once can be replayed at any point in the future and will still verify. Five minutes is a reasonable window.
import hashlib, hmac, os, time
from flask import Flask, request, abort
app = Flask(__name__)
SECRET = os.environ["PANDASTACK_WEBHOOK_SECRET"].encode()
TOLERANCE = 300 # seconds
def verify(raw: bytes, header: str) -> None:
parts = dict(p.split("=", 1) for p in header.split(",") if "=" in p)
ts, sig = parts.get("t", ""), parts.get("v1", "")
if not ts or not sig:
abort(400, "malformed signature header")
if abs(time.time() - int(ts)) > TOLERANCE:
abort(400, "timestamp outside tolerance") # replay guard
expected = hmac.new(SECRET, ts.encode() + b"." + raw, hashlib.sha256).hexdigest()
if not hmac.compare_digest(expected, sig): # constant time
abort(401, "bad signature")
@app.post("/pandastack")
def hook():
verify(request.get_data(), request.headers.get("X-PandaStack-Signature", ""))
event = request.get_json()
if not seen_before(event["id"]): # idempotency, see below
handle(event["type"], event["data"])
return "", 200 # ack fastStep 4: assume every event arrives more than once
Retries are what make webhooks reliable, and they are also why duplicates are guaranteed rather than possible. If your handler takes eleven seconds and the sender's timeout is ten, the delivery succeeded from your side and failed from theirs — and it comes back. Any at-least-once system has this property; the only question is whether your handler survives it.
The fix is small: the delivery id is stable across retries, so record it and skip anything you have already processed. A unique index on the id in your database is the whole implementation, and it is far more robust than an in-memory set that resets on every deploy.
Then acknowledge fast and do the work afterwards. Return 200 as soon as you have verified and recorded the event; push the Slack message, the database write, and anything else onto a queue. A handler that does real work inline is a handler that eventually exceeds the timeout and generates the duplicates you were just defending against.
Step 5: debugging the delivery that did not arrive
# Fire a synthetic event at your endpoint.
curl -s -X POST \
https://api.pandastack.ai/v1/webhooks/endpoints/$ID/test \
-H "Authorization: Bearer $PANDASTACK_API_KEY"
# What happened to recent deliveries?
curl -s https://api.pandastack.ai/v1/webhooks/endpoints/$ID/deliveries \
-H "Authorization: Bearer $PANDASTACK_API_KEY"
# -> status, attempts, last_code, last_error, next_try_atThe delivery log answers the question you actually have, which is whether the problem is on the sending side or yours. A last_code of 401 means your verification is rejecting real deliveries — usually the raw-body issue. A last_code of 000 with a connection error means we could not reach you at all. And a status of dead means the attempts are exhausted and the event is gone; fix the endpoint, then re-trigger the source action rather than waiting for a retry that will not come.
Developing against webhooks locally
You cannot register localhost, and you should not want to. Use a tunnel — ngrok, Cloudflare Tunnel, or your platform's equivalent — to expose your local handler on a public HTTPS URL, register that, and use the test endpoint to drive it. Keep a couple of captured payloads as fixtures so your handler tests do not need the network at all; the signature verification is easy to unit-test once you can construct a valid header yourself.
Frequently asked questions
Why does my signature verification fail on real deliveries but pass in tests?
Almost always because your framework parsed the JSON before you got to it and you signed a re-serialised version. HMAC is over exact bytes, and re-encoding changes key order, whitespace, and unicode escaping. Capture the raw body — request.get_data() in Flask, the raw buffer in Express before any JSON middleware, the untouched io.Reader in Go — and sign that. Your tests pass because you constructed the body yourself, so nothing re-serialised it.
Do I need to handle duplicate webhook deliveries?
Yes. Any at-least-once delivery system produces duplicates, most commonly when your handler is slower than the sender's timeout: you processed the event, the sender saw a timeout, and it retries. The delivery id is stable across retries, so store it with a unique constraint and skip anything you have seen. That single index is more reliable than any amount of careful handler logic, and it survives your service restarting.
How fast does my webhook handler need to be?
Faster than the sender's timeout, which is ten seconds for our deliveries. In practice you want to be well inside that: verify the signature, record the delivery id, enqueue the work, and return 200 — that path should be milliseconds. Anything that talks to a third party, writes a report, or triggers a build belongs in a background job. A handler that does real work inline is the most common cause of duplicate processing.
What happens if my endpoint is down for an hour?
You have a window, and then you do not. Our schedule is six attempts spread over roughly eleven hours — a minute, five minutes, thirty minutes, two hours, eight hours — so a short outage recovers on its own. Beyond that the delivery is marked dead and will not be retried, which is why the delivery log matters: check it after any incident, and re-trigger the underlying action for anything that expired. Do not build a system that assumes webhooks are guaranteed forever; for anything financial or irreversible, reconcile against the API periodically as well.
Keep reading
49ms p50 cold start. Fork, snapshot, and scale to zero.