Request Fingerprinting to Detect Payload Mismatch

An idempotency key binds a client’s intent to a server’s execution. Fingerprinting is what makes the binding honest: without it, a client that reuses one key across two different payloads gets whichever result the server happened to store first, with no indication that its second request was discarded. This runbook implements the payload-binding obligation described in API contract design for idempotent endpoints.

Prerequisites. You need a working key registration path — the atomic absent → pending transition from Redis SET NX or a unique constraint — and the ability to buffer the request body before the handler consumes it, which idempotency middleware & framework integration covers per framework.

---

Step 1 — Decide what belongs inside the fingerprint

The rule: include everything that changes what the operation does, exclude everything that varies between a request and its own retry.

The Authorization header deserves special mention. It is tempting to include it — surely a different credential is a different request? — but bearer tokens rotate, and a client that refreshes its token between attempts would find its retry rejected. The credential is already in the scoping tuple, which is the right place for it.

---

Step 2 — Canonicalise the JSON deterministically

Two JSON documents can be semantically identical and byte-different: key order, whitespace, unicode escaping, and number formatting all vary between languages and libraries. Hash the raw bytes and a client that switches from Python’s json.dumps to Go’s encoding/json will see every retry rejected.

The canonicalisation rules, which must be identical on both sides if the client ever computes the fingerprint itself:

1. Sort object keys by their UTF-16 code-unit sequence (the JCS / RFC 8785 rule, and what most sorted-serialisers already do).
2. Remove all insignificant whitespace — no spaces after : or ,, no newlines.
3. Serialise numbers in the shortest round-tripping form: 1 not 1.0, 1e3 normalised to 1000.
4. Do not escape non-ASCII characters; emit UTF-8 directly rather than \uXXXX.
5. Preserve array order — arrays are ordered by definition, and reordering them changes meaning.

import hashlib, json

def canonical(obj) -> bytes:
    # sort_keys handles rule 1; separators handles rule 2;
    # ensure_ascii=False handles rule 4. Python's repr of numbers satisfies rule 3.
    return json.dumps(obj, sort_keys=True, separators=(",", ":"),
                      ensure_ascii=False).encode("utf-8")

FINGERPRINT_VERSION = b"fp1"

def fingerprint(method: str, route: str, body: dict, semantic_headers: dict) -> bytes:
    h = hashlib.sha256()
    for part in (FINGERPRINT_VERSION, method.encode(), route.encode(),
                 canonical(body), canonical(semantic_headers)):
        h.update(part)
        h.update(b"\x1f")        # unit separator — no field-boundary ambiguity
    return h.digest()            # 32 bytes

// Go: encoding/json marshals map keys sorted, so a round trip through
// map[string]any is already canonical under rules 1, 2 and 4.
func Fingerprint(method, route string, body []byte, semantic map[string]string) [32]byte {
    var doc any
    _ = json.Unmarshal(body, &doc)
    canon, _ := json.Marshal(doc)

    h := sha256.New()
    for _, part := range [][]byte{[]byte("fp1"), []byte(method), []byte(route), canon} {
        h.Write(part)
        h.Write([]byte{0x1f})
    }
    keys := maps.Keys(semantic)
    slices.Sort(keys)
    for _, k := range keys {
        h.Write([]byte(k)); h.Write([]byte{0x1e}); h.Write([]byte(semantic[k])); h.Write([]byte{0x1f})
    }
    return [32]byte(h.Sum(nil))
}

The 0x1f unit separator between parts is not decoration. Without it, ("ab", "c") and ("a", "bc") hash identically, and a route named /v1/charge with a body starting s{ collides with /v1/charges and a body starting {.

---

Step 3 — Store 32 bytes, not the body

ALTER TABLE idempotency_keys
    ADD COLUMN fingerprint bytea NOT NULL,           -- exactly 32 bytes
    ADD CONSTRAINT fingerprint_len CHECK (octet_length(fingerprint) = 32);

A digest is constant-size regardless of payload, carries no personal data, and needs no index of its own — it is read only after the primary key lookup has already located the row. Storing the body instead would multiply the dedup store’s footprint by the mean payload size and put a second copy of customer data under the same retention and erasure obligations as the primary store.

---

Step 4 — Compare on every hit

import hmac

def compare(stored: bytes, computed: bytes) -> bool:
    # Constant-time: a fingerprint is not a secret, but the comparison sits on an
    # authenticated path and timing-independent equality costs nothing here.
    return hmac.compare_digest(stored, computed)

Reject with 422, not 409. A 409 says “try again later”; a 422 says “this request is wrong and retrying will not help”, which is exactly true of a key reused with new content.

---

Step 5 — Version the fingerprint

Canonicalisation rules change — a field gets excluded, a header becomes semantic, a number format is normalised. Every change invalidates every stored fingerprint, and without a version prefix that manifests as a wave of spurious 422s.

FINGERPRINT_VERSION = b"fp2"   # bump whenever the rules above change

def compare_with_migration(row, computed_v2: bytes, computed_v1: bytes) -> bool:
    if row.fingerprint_version == "fp2":
        return hmac.compare_digest(row.fingerprint, computed_v2)
    return hmac.compare_digest(row.fingerprint, computed_v1)   # legacy rows

Keep the dual-compare for one full key-retention window — 24 hours if that is your TTL — then delete the legacy branch. Any longer and it becomes permanent.

---

Verification and testing

Confirm canonicalisation is order-independent.

curl -s -XPOST localhost:8080/v1/charges -H "Idempotency-Key: $KEY" \
  -H 'Content-Type: application/json' -d '{"amount":4500,"currency":"gbp"}' -o /dev/null -w '%{http_code}\n'
curl -s -XPOST localhost:8080/v1/charges -H "Idempotency-Key: $KEY" \
  -H 'Content-Type: application/json' -d '{"currency":"gbp","amount":4500}' -o /dev/null -w '%{http_code}\n'
# expect: 201 then 201 (replay) — key order must NOT read as a mismatch

Confirm a real change is caught.

curl -s -XPOST localhost:8080/v1/charges -H "Idempotency-Key: $KEY" \
  -H 'Content-Type: application/json' -d '{"amount":9900,"currency":"gbp"}' \
  | jq -r '.code'
# expect: idempotency_key_payload_mismatch

Confirm cross-language agreement. Compute the digest in each runtime your clients use and assert equality — this is the test that catches a serialiser difference before a customer does.

python3 -c "print(__import__('hashlib').sha256(__import__('json').dumps({'b':1,'a':'é'},sort_keys=True,separators=(',',':'),ensure_ascii=False).encode()).hexdigest())"
node -e "const c=JSON.stringify(Object.fromEntries(Object.entries({b:1,a:'é'}).sort()));console.log(require('crypto').createHash('sha256').update(c,'utf8').digest('hex'))"
# expect: identical hex digests

Confirm the stored column is exactly 32 bytes.

SELECT octet_length(fingerprint) AS len, count(*)
  FROM idempotency_keys GROUP BY 1;
-- expect: a single row, len = 32

---

Failure scenarios and debugging

---

SRE / observability checklist

1. idempotency_payload_mismatch_total — Counter labelled by credential_id and route. Alert when one credential exceeds 5 in 10 minutes; that is a broken client, not network noise.
2. idempotency_fingerprint_duration_us — Histogram of canonicalisation plus hashing. Alert if p99 exceeds 2000 µs; a spike usually means an unbounded body slipped past the size limit.
3. idempotency_fingerprint_version — Label on the comparison counter, so a version migration is observable and the legacy branch can be removed on evidence.
4. request_body_bytes — Histogram on idempotent routes. Alert if p99 exceeds 256 KB, which is where fingerprinting cost stops being negligible.
5. idempotency_key and fingerprint_prefix log fields — Log the first 8 hex characters of the digest, never the body. It is enough to correlate a mismatch across two request logs and reconstruct what differed.

---

Related

• API Contract Design for Idempotent Endpoints — the parent page defining payload binding as one of the three contract guarantees.
• Designing the Idempotency-Key HTTP Header Contract — the scoping tuple that carries the credential, so the fingerprint does not have to.
• Returning 409 vs 200 for Replayed Requests — why a mismatch is 422 while an in-flight duplicate is 409.
• How to Generate Cryptographically Secure Idempotency Keys — the deterministic-key alternative, where the fingerprint is the key.