Designing the Idempotency-Key HTTP Header Contract

This runbook turns the header half of API contract design for idempotent endpoints into a specification you can paste into an API reference and an implementation you can paste into a validator. Everything here is a decision that must be identical on both sides of the wire, which is why it belongs in a written contract rather than in whichever handler happened to be implemented first.

Prerequisites. You should already understand why a POST needs a key at all — see HTTP method semantics & safety — and have chosen a key format from idempotency key generation strategies. You should also have an atomic registration mechanism available, whether that is Redis SET NX or a unique constraint.

---

Step 1 — Fix the syntax and length bounds

Idempotency-Key = 16*255( ALPHA / DIGIT / "-" / "_" )

16 characters is the floor because anything shorter cannot carry enough entropy to be collision-resistant when a client generates keys naively. 255 is the ceiling because it fits comfortably inside every proxy’s per-header limit and inside a varchar(255) column without truncation surprises. The character class is the URL-unreserved set minus . and ~, which keeps the value safe to embed in a Redis key, a log field, a metrics label and a URL path without escaping.

import re

IDEMPOTENCY_KEY_RE = re.compile(r"\A[A-Za-z0-9_-]{16,255}\Z")

def validate_key(raw: str | None) -> str:
    if raw is None:
        raise ApiError(400, "idempotency_key_required")
    if len(raw) < 16:
        raise ApiError(400, "idempotency_key_too_short")
    if len(raw) > 255:
        raise ApiError(400, "idempotency_key_too_long")
    if not IDEMPOTENCY_KEY_RE.match(raw):
        raise ApiError(400, "idempotency_key_invalid_characters")
    return raw

Distinct error codes for each failure are not pedantry — they are the difference between a client author fixing the bug in ten seconds and opening a support ticket.

---

Step 2 — Define the scoping tuple

The header value is never the storage key. The storage key is a tuple, and every element of it prevents a specific incident.

// scoped returns the lookup key. Every component is load-bearing:
//   credential — stops tenant A's "order-1" shadowing tenant B's "order-1"
//   route      — stops one client request id colliding across create and cancel
//   version    — lets you invalidate the whole namespace during a migration
func scoped(credentialID, route, key string) string {
    return fmt.Sprintf("idem:v1:%s:%s:%s", credentialID, route, key)
}

-- The relational equivalent: a composite unique constraint, not a single column.
CREATE TABLE idempotency_keys (
    credential_id  uuid        NOT NULL,
    route          text        NOT NULL,
    key            text        NOT NULL,
    fingerprint    bytea       NOT NULL,       -- SHA-256 of the canonical body
    state          text        NOT NULL,       -- pending | completed
    status_code    smallint,
    response_body  jsonb,
    expires_at     timestamptz NOT NULL,
    created_at     timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (credential_id, route, key)
);
CREATE INDEX idempotency_keys_expiry ON idempotency_keys (expires_at);

Include the v1: prefix from day one. Migrating a key namespace without it means either a downtime window or a dual-read path you will maintain forever.

---

Step 3 — Order the validation checks

Order matters because each check is cheaper than the next and each produces a different error. Running them out of order means a malformed key from an unauthenticated caller consumes a store round trip.

1. Authenticate. Without a credential there is no scope, so a key cannot even be looked up.
2. Check presence. On a required-key route, a missing header is 400 idempotency_key_required. Do not silently proceed.
3. Check syntax. Length and character class, per step 1. Still no store access.
4. Read the body and fingerprint it. Bounded at 1 MB; a larger body on an idempotent route is 413.
5. Register atomically. One conditional write. This is the only step that can decide a race.
6. Compare fingerprints on a hit. A mismatch is 422 idempotency_key_payload_mismatch and executes nothing.
7. Branch on state. pending returns 409 idempotency_key_in_use with Retry-After: 1; completed replays the stored response.

// Express, in the order above. Note that nothing touches the store until step 5.
async function guard(req, res, next) {
  const raw = req.get('Idempotency-Key');
  if (!raw) return res.status(400).json({ code: 'idempotency_key_required' });
  if (!/^[A-Za-z0-9_-]{16,255}$/.test(raw)) {
    return res.status(400).json({ code: 'idempotency_key_invalid' });
  }
  const fingerprint = sha256(canonicalJson(req.body));
  const hit = await store.registerOrGet(
    scoped(req.credential.id, req.route.path, raw), fingerprint, 60_000);

  if (!hit) return next();                                   // fresh key — execute
  if (hit.fingerprint !== fingerprint) {
    return res.status(422).json({ code: 'idempotency_key_payload_mismatch' });
  }
  if (hit.state === 'pending') {
    return res.set('Retry-After', '1').status(409)
              .json({ code: 'idempotency_key_in_use' });
  }
  return res.set('Idempotent-Replay', 'true').status(hit.status).json(hit.body);
}

---

Step 4 — Emit the response headers

Three response headers make the contract legible to a client without forcing it to parse error bodies.

HTTP/1.1 201 Created
Idempotent-Replay: false
Idempotency-Key: 01HXYZ3ABCDEFGHJKMNPQRSTVW
Idempotency-Key-Expires: 2026-08-02T09:14:33Z

Idempotent-Replay is the important one. Echoing the status code alone is ambiguous — a client cannot tell whether it created the charge or discovered one it had already created — and changing the status code to signal a replay breaks every SDK that switches on 201. A boolean header carries the information without touching the status line.

Idempotency-Key-Expires gives an absolute instant rather than a relative duration, so a client that queues a retry for later can decide whether the key will still be live when the retry fires.

---

Step 5 — Publish the error-code table

---

Verification and testing

Confirm the syntax gate rejects before touching the store. Point the service at a deliberately unreachable Redis and send a 4-character key; the response must be 400, not 503.

curl -s -o /dev/null -w '%{http_code}\n' -XPOST localhost:8080/v1/charges \
  -H 'Idempotency-Key: abc' -H 'Content-Type: application/json' \
  -d '{"amount":100,"currency":"gbp"}'
# expect: 400

Confirm scoping actually isolates tenants. Send the same key with two different credentials and assert two distinct resources are created.

KEY=01HXYZ3ABCDEFGHJKMNPQRSTVW
for CRED in tenant_a_secret tenant_b_secret; do
  curl -s -XPOST localhost:8080/v1/charges -H "Authorization: Bearer $CRED" \
    -H "Idempotency-Key: $KEY" -H 'Content-Type: application/json' \
    -d '{"amount":4500,"currency":"gbp"}' | jq -r '.id'
done
# expect: two different charge ids

Confirm the replay header flips. The first call must report Idempotent-Replay: false, the second true, with identical bodies.

curl -si -XPOST localhost:8080/v1/charges -H "Idempotency-Key: $KEY" \
  -H 'Content-Type: application/json' -d '{"amount":4500,"currency":"gbp"}' \
  | grep -i 'idempotent-replay\|HTTP/'

Confirm the stored key row looks right.

SELECT state, status_code, encode(fingerprint, 'hex'), expires_at
  FROM idempotency_keys
 WHERE credential_id = :cred AND route = '/v1/charges' AND key = :key;
-- expect: completed | 201 | <64 hex chars> | now() + 24 hours

---

Failure scenarios and debugging

---

SRE / observability checklist

1. idempotency_key_validation_failed_total — Counter labelled by code (too_short, invalid_characters, required). Alert when one credential exceeds 10 in 5 minutes; it identifies a broken client release, not a network problem.
2. idempotency_key_present_ratio — Gauge of requests carrying the header divided by total requests on required-key routes. Page below 0.98, which almost always means an edge hop is stripping it.
3. idempotency_registration_duration_ms — Histogram of step 5 only, excluding handler time. Alert if p99 exceeds 15 ms; the dedup store has become the endpoint’s bottleneck.
4. idempotency_conflict_total — Counter of 409 idempotency_key_in_use. A sustained rate above 1% of requests means clients are retrying faster than handlers complete.
5. idempotency_key structured log field — Present on every request log line for the route, so an incident can reconstruct the window even after the store has expired the keys.
6. idempotency_namespace_version label — On the registration counter, so a namespace migration is observable rather than inferred.

---

Related

• API Contract Design for Idempotent Endpoints — the parent page covering the full request lifecycle state machine and the three contract variants.
• Request Fingerprinting to Detect Payload Mismatch — how to canonicalise a body so the fingerprint in step 3 is stable across clients.
• Returning 409 vs 200 for Replayed Requests — the status-code decision behind the error table above.
• UUIDv4 vs UUIDv7 vs ULID for Idempotency Keys — picking the value format that fits the syntax envelope in step 1.