Chaos Testing Duplicate Request Handling

Part of: Chaos Engineering for Idempotency

This is a runbook, not a conceptual overview. It scripts a burst of concurrent, logically identical requests against a live payment endpoint, injects a fault on the dedup store mid-burst with Toxiproxy, and verifies that exactly one side effect committed. If your dedup logic has never been exercised at production-level concurrency, this is the fastest way to find the race condition before a customer does.

---

Problem Statement and Prerequisites

What you are proving: that N concurrent requests sharing one idempotency key result in exactly one committed side effect, even when the dedup store degrades mid-burst.

Prerequisites:

• You have read the parent chaos engineering for idempotency methodology and understand the hypothesis-injection-observation loop.
• Your endpoint already implements SETNX-based deduplication or an equivalent conditional-write pattern, with a PROCESSING → COMPLETE state machine stored against the idempotency key.
• Toxiproxy is installed and can proxy traffic between your application and its dedup store (Redis, DynamoDB, or Postgres).
• You have a staging environment where a duplicate charge is safe to produce (use a test payment processor account, never a live one).

---

The Race This Test Exposes

---

Step-by-Step Implementation

Step 1 — Fire concurrent duplicate requests with Python asyncio

Send 50 concurrent requests carrying the identical idempotency key. This is the minimum concurrency needed to reliably trigger the race between the winning SETNX and the losing readers.

import asyncio
import httpx
import uuid

ENDPOINT = "https://staging.example.com/v1/charges"
IDEMPOTENCY_KEY = str(uuid.uuid4())  # one key, shared by every request in the burst
CONCURRENCY = 50

async def fire_one(client: httpx.AsyncClient, i: int) -> tuple[int, str]:
    resp = await client.post(
        ENDPOINT,
        json={"amount": 2500, "currency": "usd"},
        headers={"Idempotency-Key": IDEMPOTENCY_KEY},
        timeout=5.0,
    )
    return resp.status_code, resp.json().get("charge_id", "")

async def main():
    async with httpx.AsyncClient() as client:
        results = await asyncio.gather(
            *(fire_one(client, i) for i in range(CONCURRENCY))
        )
    charge_ids = {r[1] for r in results}
    print(f"Unique charge_ids returned: {len(charge_ids)} (expect 1)")
    assert len(charge_ids) == 1, f"Duplicate side effect detected: {charge_ids}"

asyncio.run(main())

Step 2 — Fire concurrent duplicate requests with Go goroutines

Run the equivalent burst from a second language runtime to rule out client-library-specific connection pooling from masking or creating false races.

package main

import (
	"bytes"
	"fmt"
	"net/http"
	"sync"
)

const (
	endpoint    = "https://staging.example.com/v1/charges"
	concurrency = 50
)

func main() {
	idempotencyKey := "burst-key-4f21c9"
	results := make(chan string, concurrency)
	var wg sync.WaitGroup

	for i := 0; i < concurrency; i++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			body := bytes.NewBufferString(`{"amount":2500,"currency":"usd"}`)
			req, _ := http.NewRequest("POST", endpoint, body)
			req.Header.Set("Idempotency-Key", idempotencyKey)
			req.Header.Set("Content-Type", "application/json")

			resp, err := http.DefaultClient.Do(req)
			if err != nil {
				results <- "ERROR:" + err.Error()
				return
			}
			defer resp.Body.Close()
			results <- resp.Status
		}()
	}
	wg.Wait()
	close(results)

	for r := range results {
		fmt.Println(r)
	}
}

Step 3 — Scale the burst to production concurrency with k6

Client-level scripts validate correctness at 50 concurrent requests; k6 validates it at the concurrency a real retry storm produces — 500 to 2,000 virtual users hammering the same key.

import http from "k6/http";
import { check } from "k6";

const IDEMPOTENCY_KEY = "k6-burst-8a12f0";

export const options = {
  scenarios: {
    duplicate_burst: {
      executor: "shared-iterations",
      vus: 500,
      iterations: 500,
      maxDuration: "10s",
    },
  },
};

export default function () {
  const res = http.post(
    "https://staging.example.com/v1/charges",
    JSON.stringify({ amount: 2500, currency: "usd" }),
    {
      headers: {
        "Content-Type": "application/json",
        "Idempotency-Key": IDEMPOTENCY_KEY,
      },
    }
  );
  check(res, { "status is 200": (r) => r.status === 200 });
}

Step 4 — Inject a Toxiproxy fault on the dedup store mid-burst

While the k6 scenario is running, add a 1,500 millisecond latency toxic to the connection between the application and the dedup store. This reproduces the interleaving where some requests have already committed PROCESSING and others are still waiting on a slow SETNX call.

# Assumes a toxiproxy proxy named "redis_dedup" is already in front of the store
toxiproxy-cli toxic add redis_dedup --type latency --attribute latency=1500 --attribute jitter=300

# Run the k6 scenario now, while the toxic is active
k6 run duplicate_burst.js

# Remove the toxic immediately after the run completes
toxiproxy-cli toxic remove redis_dedup --toxicName latency_downstream

Step 5 — Verify dedup store state and assert exactly one side effect

After the burst completes, inspect the dedup store directly rather than trusting only the HTTP responses — a bug could return 200 to every caller while still writing two COMPLETE records under a different key variant.

# Redis: confirm exactly one key exists for this idempotency key, in COMPLETE state
redis-cli GET "idem:k6-burst-8a12f0"
# Expected: {"status":"COMPLETE","charge_id":"ch_...","amount":2500}

# Confirm no orphaned PROCESSING record remains after the verification window
redis-cli TTL "idem:k6-burst-8a12f0"
# Expected: a positive TTL, not -1 (no expiry set) or -2 (missing)

-- If the side effect writes to a ledger table, confirm exactly one row exists
-- for the idempotency key regardless of how many HTTP requests were sent.
SELECT idempotency_key, COUNT(*) AS row_count
FROM charges
WHERE idempotency_key = 'k6-burst-8a12f0'
GROUP BY idempotency_key
HAVING COUNT(*) <> 1;
-- Expected: zero rows returned. Any row here is a duplicate side effect.

---

Verification and Testing

Run these checks after every burst, in order:

# 1. Confirm the HTTP layer returned a consistent charge_id across all responses
grep -o '"charge_id":"[^"]*"' k6_output.json | sort -u | wc -l   # expect: 1

# 2. Confirm the dedup store has no lingering PROCESSING record past the TTL window
redis-cli GET "idem:k6-burst-8a12f0" | grep -q COMPLETE && echo "OK: settled"

# 3. Confirm the downstream ledger recorded exactly one charge
psql -c "SELECT COUNT(*) FROM charges WHERE idempotency_key = 'k6-burst-8a12f0';"
# expect: count = 1

If any check fails, do not re-run the burst against the same key — capture the dedup store’s raw state (redis-cli --scan for related keys, or a point-in-time export) before it expires, since the evidence of the race is time-limited.

---

Failure Scenarios and Debugging

---

SRE / Observability Checklist

1. duplicate_side_effect_rate — Counter incremented whenever more than one committed side effect is found for a single idempotency key during verification. Alert on any value greater than 0.
2. dedup_key_write_attempts_total — Counter labeled by idempotency_key and outcome (acquired / rejected). A burst of 500 requests should show exactly one acquired and 499 rejected.
3. processing_stuck_records_total — Gauge of records stuck in PROCESSING past TTL + P99 latency. Non-zero for more than 30 seconds indicates a finalize-path bug.
4. dedup_store_cpu_utilization — Gauge sampled during the burst; correlate spikes against the concurrency level from k6 to find the safe operating ceiling.
5. Trace span dedup.commit — Attach idempotency_key, outcome, and charge_id attributes so a single burst’s full fan-out can be reconstructed in one trace query.
6. Structured log field idempotency_key_state — Emit on every state transition (PROCESSING, COMPLETE, FAILED) so incident review can replay the exact sequence of writes during the burst.

---

Related

• Chaos Engineering for Idempotency — parent page covering the hypothesis-driven chaos method and the full fault taxonomy this runbook exercises one fault from
• Injecting Clock Skew to Validate TTL Safety — companion runbook for the clock-skew fault, using fencing tokens instead of a race on a single key
• Using Redis SETNX for Distributed Request Deduplication — the underlying dedup pattern this runbook validates under load
• Mitigating Thundering Herd During Retry Storms — server-side defenses for the retry-storm scenario surfaced when scaling to k6 concurrency
• Handling Stale Locks in Distributed Systems — fencing token mechanics referenced in the failure scenarios table above