Verifying Webhook Signatures and Replay Windows

A webhook receiver has two independent obligations that are routinely conflated: prove the request came from the sender, and ensure a genuine event is applied once. Signature verification does the first. It does nothing whatsoever about the second, because the sender’s own retries are authentic. This runbook covers both and sits under webhook delivery guarantees.

Prerequisites. You should have a deduplication store available — Redis or a processed-events table — and be able to read the raw request body in your framework before any JSON middleware consumes it.

Step 1 — Capture the raw body

This is where most integrations break, and it always looks like a signature bug.

// Express: keep the raw bytes alongside the parsed body.
app.use('/webhooks', express.json({
  verify: (req, _res, buf) => { req.rawBody = buf; },   // Buffer, not a string
}));
# FastAPI / Starlette: read the body first and cache it so the route still parses.
@app.post("/webhooks/payments")
async def receive(request: Request):
    raw = await request.body()
    request._body = raw
    verify_signature(raw, request.headers)
// net/http: read once, then hand the bytes back to the decoder.
raw, err := io.ReadAll(io.LimitReader(r.Body, 1<<20))
if err != nil { http.Error(w, "unreadable body", 400); return }
r.Body = io.NopCloser(bytes.NewReader(raw))

Never verify against JSON.stringify(req.body). {"a":1,"b":2} and {"b":2,"a":1} are the same object and different bytes, and the HMAC only sees bytes.

Step 2 — Recompute the HMAC

The signed payload is usually the timestamp and the raw body joined by a separator, not the body alone — including the timestamp is what makes the signature cover it and stops an attacker replaying an old body with a fresh timestamp.

POST /webhooks/payments HTTP/1.1
Webhook-Timestamp: 1785941673
Webhook-Signature: v1=5f2a...c91d,v1=7b3e...11aa
Content-Type: application/json

{"id":"evt_9f8e7d","type":"charge.succeeded","data":{"amount":4500}}

import hashlib, hmac, time

TOLERANCE_S = 300      # 5 minutes

def verify(raw: bytes, headers, secrets: list[bytes]) -> None:
    ts = headers.get("Webhook-Timestamp")
    sig_header = headers.get("Webhook-Signature")
    if not ts or not sig_header:
        raise Unauthorized("missing_signature_headers")

    signed_payload = ts.encode() + b"." + raw          # timestamp is INSIDE the MAC
    presented = {s.split("=", 1)[1] for s in sig_header.split(",") if s.startswith("v1=")}

    # Every currently valid secret is tried, so rotation never drops a delivery.
    for secret in secrets:
        expected = hmac.new(secret, signed_payload, hashlib.sha256).hexdigest()
        if any(hmac.compare_digest(expected, p) for p in presented):
            break
    else:
        raise Unauthorized("signature_mismatch")

    if abs(time.time() - int(ts)) > TOLERANCE_S:
        raise Unauthorized("timestamp_outside_tolerance")
Webhook signature construction and comparison The timestamp header and the raw request body are concatenated with a dot separator to form the signed payload. That payload is passed through HMAC-SHA256 once per active secret, producing a set of expected signatures. Each expected signature is compared in constant time against the set of signatures presented in the Webhook-Signature header. A match on any pair accepts the request; no match rejects it with 401. Webhook-Timestamp 1785941673 raw body bytes unparsed, verbatim signed payload ts + "." + body HMAC-SHA256 secret A (current) HMAC-SHA256 secret B (rotating out) constant-time compare Any expected digest matching any presented digest → accept Trying every active secret is what makes rotation lossless: no delivery is dropped mid-rotation. A valid signature proves authenticity — NOT that the event is new. The sender's own retry is authentic. Deduplication by event id is a separate, mandatory step.

Step 3 — Separate the two windows

The single most common design error is using one window for both purposes.

Window Purpose Typical value Consequence of getting it wrong
Replay tolerance (timestamp) Bound how long a captured request is useful to an attacker 300 seconds Too long: an intercepted request stays replayable for hours. Too short: legitimate deliveries fail on clock skew.
Deduplication window (event id) Absorb the sender’s own retries 24–72 hours Too short: a sender’s retry after a long outage is applied twice. Too long: unnecessary storage, no correctness harm.

The sender’s retry schedule sets the deduplication floor. Most webhook providers retry with escalating backoff for 24 to 72 hours, so a dedup window shorter than the sender’s give-up time leaves a gap in which a genuine, correctly signed retry is applied a second time.

def receive(raw: bytes, headers, conn) -> tuple[int, str]:
    verify(raw, headers, active_secrets())        # step 2: authenticity, 300 s tolerance
    event = json.loads(raw)

    try:
        with conn.transaction():
            conn.execute(
                "INSERT INTO processed_webhooks (event_id, received_at) VALUES (%s, now())",
                (event["id"],),                   # dedup, 72 h retention
            )
            apply_effect(event, conn)
    except UniqueViolation:
        return 200, "duplicate"                   # 200, not an error — the sender did nothing wrong
    return 200, "processed"

Returning 200 on a duplicate matters. A 409 or a 500 tells the sender to retry, which turns one redundant delivery into an escalating series of them.

Step 4 — Handle secret rotation

Lossless webhook secret rotation A timeline in three phases. Before rotation only secret A is accepted. During the overlap window, which must exceed the sender's maximum retry span of seventy-two hours, both secret A and secret B are accepted. After the overlap only secret B is accepted. A note explains that removing secret A before the overlap elapses rejects in-flight retries that were signed with it. accept: A steady state accept: A or B overlap ≥ sender's max retry span 72 hours minimum accept: B A revoked B published A removed Shortening the overlap rejects retries of events first sent before the rotation. 
def active_secrets() -> list[bytes]:
    """Every secret currently inside its validity window, newest first.
    Verification tries all of them, so rotation never drops a delivery."""
    return [s.value for s in WebhookSecret.objects
            .filter(valid_from__lte=now(), valid_until__gte=now())
            .order_by("-valid_from")]

Verification and testing

Confirm verification uses raw bytes. Send a body with non-canonical key order and assert it still verifies.

BODY='{"type":"charge.succeeded","id":"evt_test_1","data":{"amount":4500}}'
TS=$(date +%s)
SIG=$(printf '%s.%s' "$TS" "$BODY" | openssl dgst -sha256 -hmac "$WEBHOOK_SECRET" -hex \
      | awk '{print $2}')
curl -s -o /dev/null -w '%{http_code}\n' -XPOST localhost:8080/webhooks/payments \
  -H "Webhook-Timestamp: $TS" -H "Webhook-Signature: v1=$SIG" \
  -H 'Content-Type: application/json' -d "$BODY"
# expect: 200

Confirm the replay tolerance is enforced.

OLD=$(( $(date +%s) - 600 ))
SIG=$(printf '%s.%s' "$OLD" "$BODY" | openssl dgst -sha256 -hmac "$WEBHOOK_SECRET" -hex | awk '{print $2}')
curl -s -o /dev/null -w '%{http_code}\n' -XPOST localhost:8080/webhooks/payments \
  -H "Webhook-Timestamp: $OLD" -H "Webhook-Signature: v1=$SIG" -d "$BODY"
# expect: 401 — valid signature, expired timestamp

Confirm deduplication is independent of the signature. Send the same event twice with two freshly computed, both-valid signatures.

SELECT count(*) FROM processed_webhooks WHERE event_id = 'evt_test_1';   -- expect 1
SELECT count(*) FROM ledger_entries     WHERE event_id = 'evt_test_1';   -- expect 1

Failure scenarios and debugging

Failure Scenario Remediation Steps Observability Hooks
Signature verified against the re-serialised body, so every delivery fails once the sender changes serialiser or field order Capture the raw buffer before JSON middleware and verify against it. Add a test that sends a body with reversed key order and asserts a 200. webhook_signature_failure_total by sender; a step change to 100% right after a sender release is the signature
Timestamp tolerance set to 24 hours “to be safe”, leaving intercepted requests replayable all day Set it to 300 seconds and fix the underlying clock skew instead, per measuring and bounding NTP clock skew. webhook_timestamp_offset_seconds histogram; if p99 exceeds 30 s the receiver’s clock is the problem
Duplicate deliveries answered with 409, so the sender treats them as failures and retries harder Return 200 with a body indicating the duplicate. The sender did nothing wrong and must not be told to retry. webhook_responses_total{outcome="duplicate",status} — anything other than 200 here is a bug
Old secret revoked immediately on rotation, and retries of events first sent before the switch are rejected Keep both secrets valid for at least the sender’s maximum retry span (72 hours) and verify against all active secrets. webhook_signature_secret_used label; watch usage of the old secret decay to zero before revoking
Dedup window set to 1 hour while the sender retries for 72, so a delivery after a long receiver outage is applied twice Set dedup retention above the sender’s documented give-up time. Assert the configured value at startup against a constant. webhook_dedup_retention_seconds gauge compared against the sender’s retry span
Two windows, two jobs, wildly different lengths The signature tolerance is drawn as a narrow five-minute band bounding how long an intercepted request stays useful to an attacker. The deduplication window is drawn as a wide seventy-two hour band bounding how long the sender may keep retrying. Conflating the two either leaves requests replayable for days or leaves genuine retries able to apply a second effect. signature tolerance 5 minutes bounds attacker replay of a captured request deduplication window 72 hours — covers the sender's own retry schedule One value for both is wrong in one direction or the other, always.

SRE / observability checklist

  1. webhook_signature_failure_total — Counter labelled by sender and reason (mismatch, missing_headers, timestamp_outside_tolerance). Page on any sustained rate; it is either an attack or a broken integration.
  2. webhook_timestamp_offset_seconds — Histogram of now - Webhook-Timestamp. A p99 above 30 seconds means clock skew is eating the tolerance budget.
  3. webhook_responses_total{outcome} — Counter split by processed and duplicate. A duplicate ratio above 5% means the receiver is slow enough to trigger sender retries.
  4. webhook_dedup_retention_seconds — Gauge exported at startup, alerted if it falls below the sender’s documented retry span.
  5. webhook_signature_secret_used — Label recording which secret verified each delivery, so a rotation can be completed on evidence rather than on a calendar.
  6. event_id and sender log fields — On every delivery, so a duplicate can be traced to the original and to the sender’s own delivery attempt id.