How to retry platform API calls safely when there is no idempotency key

A client called our create endpoint. The request timed out at thirty seconds. The client retried, twice, as it had been told to. What actually happened on our side was that all three requests succeeded — the response was just slow getting back — so the account ended up with three databases and a bill for three databases, and the client's code held one identifier and knew about one.

Nobody had written a bug. The retry policy was the default one from the HTTP library. This post is about the small amount of thinking that turns that default into something safe, and about what to do on an API that does not hand you an idempotency key — including ours, which today does not.

A timeout is not a failure

This is the whole idea and it is worth being slow about. When you get a 500 with a body, you know the server processed your request and decided it failed. When you get a connection timeout, you know nothing. The request may have never arrived, or arrived and been rejected, or arrived and completed perfectly while the response was lost on the way back.

Retrying a definite failure is safe. Retrying an unknown is a coin flip whose downside is a duplicate resource. Most HTTP client libraries treat both as "error, retry", and that conflation is where the duplicate databases come from.

Classify before you retry

Every response falls into one of three buckets, and only one of them is a simple retry.

• Definitely did not happen — 400, 401, 403, 404, 422. Do not retry. The request was understood and refused, and sending it again produces the same refusal. Retrying a 401 in a loop is how you get rate limited on top of being unauthorised.
• Definitely did not happen, yet — 429, 503, and a connection refused. Retry with backoff. The server told you it is not ready, which is information.
• Unknown — a timeout, a connection reset mid-flight, a 502 or 504 from an intermediary. For a read, retry freely. For anything that creates or charges, do not blind-retry: reconcile first.

That third bucket is where the work is. Everything else is a lookup table.

Backoff, with jitter, and honour Retry-After

Exponential backoff without jitter synchronises your clients. If fifty callers hit a rate limit at the same moment and all back off by exactly two seconds, they all return at the same moment, and you have rebuilt the thundering herd with a two-second delay in front of it.

async function backoff(attempt: number, res?: Response) {
  // the server's own instruction wins over any local policy
  const ra = res?.headers.get("retry-after");
  if (ra) {
    const secs = Number(ra);
    await sleep((Number.isFinite(secs) ? secs : 60) * 1000);
    return;
  }
  const capped = Math.min(2 ** attempt * 250, 20_000);
  await sleep(Math.random() * capped); // full jitter
}

Full jitter — a random value between zero and the capped ceiling, not the ceiling plus a wobble — is the variant that spreads a herd best, and it is barely more code. Cap the ceiling. And cap the total number of attempts, because an unbounded retry loop against a service that is genuinely down is a denial-of-service tool you are pointing at your vendor and, through the rate limiter, at yourself.

Reconcile-then-retry: idempotency you build yourself

Without an idempotency key, the safe pattern for a create is: tag the resource with something you generate, and on an unknown outcome, look before you leap.

Most create endpoints accept a label, a name or a metadata object. That field is your idempotency key if you use it as one.

import { randomUUID } from "node:crypto";

async function createSandboxOnce(template: string) {
  const opId = randomUUID();          // one id for this logical operation
  for (let attempt = 0; attempt < 5; attempt++) {
    try {
      const res = await fetch(`${API}/v1/sandboxes`, {
        method: "POST",
        headers: auth,
        body: JSON.stringify({ template, metadata: { op_id: opId } }),
        signal: AbortSignal.timeout(30_000),
      });
      if (res.ok) return await res.json();
      if (res.status < 500 && res.status !== 429) throw new Error(await res.text());
      await backoff(attempt, res);
    } catch (err) {
      // unknown outcome: the request may have succeeded. look before retrying.
      const existing = await findByOpId(opId);
      if (existing) return existing;
      await backoff(attempt);
    }
  }
  throw new Error("exhausted retries");
}

async function findByOpId(opId: string) {
  const r = await fetch(`${API}/v1/sandboxes`, { headers: auth });
  const { items } = await r.json();
  return items.find((s: any) => s.metadata?.op_id === opId) ?? null;
}

The important detail is that `opId` is generated once, outside the loop. Generating it inside makes every attempt a distinct operation and you are back to creating duplicates, but now with more code.

Polling a 202 without hammering

Several of our database operations — wake, failover, clone — return 202 immediately and do the work in the background. That design exists precisely because holding a connection open for a two-minute restore is how you get a timeout and then a retry and then a second restore of the same database.

The client side of a 202 has three requirements that people routinely get two out of three on: a terminal error check, a hard timeout, and backing off as you wait.

async function waitUntilRunning(dbId: string, budgetMs = 10 * 60_000) {
  const deadline = Date.now() + budgetMs;
  let delay = 1000;
  while (Date.now() < deadline) {
    const db = await getDatabase(dbId);
    if (db.status === "running") return db;
    if (db.status === "error") throw new Error(`provisioning failed: ${db.error}`);
    await sleep(delay);
    delay = Math.min(delay * 1.5, 15_000);   // ease off as it drags on
  }
  throw new Error(`timed out after ${budgetMs}ms; database ${dbId} may still be provisioning`);
}

Read that last error message carefully, because it is doing something the obvious version does not. A poll timeout does not mean the operation failed — it means you stopped watching. If your code responds to a poll timeout by deleting and recreating, you will occasionally destroy a database that was thirty seconds from being ready. Say what you know, which is that you gave up, and leave the resource identified so a human or a reconciler can pick it up.

The reconciler you will need eventually

Any at-least-once system leaks. Not often, but the failures accumulate and they are invisible because a leaked resource is by definition one your application does not have a reference to. The cheapest fix is a scheduled job that compares what exists against what you believe should exist.

# anything tagged by our client, older than a day, not in our database
curl -sS -H "Authorization: Bearer $PANDASTACK_API_KEY" \
  "$API/v1/sandboxes" \
  | jq -r '.items[] | select(.metadata.op_id != null) | "\(.id) \(.metadata.op_id)"' \
  | while read -r id op; do
      known=$(psql -tAc "select 1 from operations where op_id = '$op'")
      [ -z "$known" ] && echo "orphan: $id (op $op)"
    done

Have it report before it deletes. The first run of a reconciler always finds things you did not expect, and roughly half of them are your understanding of the system being wrong rather than actual orphans.

The short version

1. Never retry a 4xx that is not a 429. It will fail the same way.
2. Retry reads freely. They are idempotent by construction.
3. For creates, generate one operation id per logical operation, put it in the metadata, and reconcile before retrying an unknown outcome.
4. Full jitter on the backoff, a cap on the ceiling, a cap on the attempts, and Retry-After wins over everything.
5. Treat 202 as the start of a poll with a budget, a terminal-error check and increasing intervals — never as done.
6. A poll timeout means you stopped watching, not that the operation failed. Do not clean up on it.
7. Run a reconciler on a schedule. It reports first, deletes later.

None of this is sophisticated. It is about forty lines of client code, and it is the difference between an integration that quietly accumulates duplicate resources for a year and one that does not.