Load Testing Duplicate Request Storms with k6

A retry storm is not “more traffic”. It is the same requests arriving many times within a few milliseconds of each other, which is a fundamentally different load shape and stresses a completely different part of the system. A load test that generates unique keys per virtual user never produces it. This runbook scripts one properly and sits under testing & verification of idempotent systems.

Prerequisites. k6 installed, a staging environment with a production-shaped deduplication store, and read access to the effects table for the post-run assertion.

Step 1 — Model the shape

Steady traffic versus a retry storm A chart of requests per second over time. A lower line labelled unique requests stays flat throughout. An upper line labelled total requests tracks just above it during the steady phase, then triples during a shaded storm window before returning to baseline. The gap between the two lines is shaded and labelled duplicates, rising from about two percent to about seventy percent of total volume. 6k 3k 0 time → dependency blip unique requests — flat total requests duplicates: 2% → 70% same keys, milliseconds apart Unique-key load tests only ever produce the flat line. The storm is the interesting part.

The deduplication store sees the total line while the handler sees the unique line. That divergence is the whole point: a store sized for steady traffic can be three to five times over capacity during a storm, and it is the first thing to fall over.

Step 2 — Build a shared key pool

k6 virtual users do not share JavaScript state, so the pool must be created once and loaded read-only.

// keys.js — generate the pool once, commit it, keep runs comparable.
import { randomString } from 'https://jslib.k6.io/k6-utils/1.4.0/index.js';
export function makePool(n) {
  return Array.from({ length: n }, (_, i) => `01HK6LOAD${String(i).padStart(16, '0')}`);
}
import { SharedArray } from 'k6/data';

// SharedArray loads ONCE per test run and is shared across all VUs, so every
// VU sees the same keys — which is what makes collisions possible at all.
const keys = new SharedArray('idempotency keys', () =>
  JSON.parse(open('./keys.json')));   // 50,000 pre-generated keys

Pool size sets the collision probability. With 50,000 keys and a 30% duplicate ratio at 2,000 requests per second, a given key is retried within a few hundred milliseconds of its first use — which is the timing that actually exercises the atomic registration path.

Step 3 — Script the storm

import http from 'k6/http';
import { check } from 'k6';
import { Counter, Rate, Trend } from 'k6/metrics';
import { SharedArray } from 'k6/data';

const keys = new SharedArray('keys', () => JSON.parse(open('./keys.json')));

const created   = new Counter('idem_created');
const replayed  = new Counter('idem_replayed');
const conflicts = new Counter('idem_conflict');
const dupRatio  = new Rate('idem_duplicate_ratio');
const regLatency = new Trend('idem_registration_ms');

export const options = {
  scenarios: {
    steady: {
      executor: 'constant-arrival-rate',
      rate: 2000, timeUnit: '1s', duration: '2m',
      preAllocatedVUs: 400, maxVUs: 1200,
      env: { DUPLICATE_RATIO: '0.02' },          // production baseline
    },
    storm: {
      executor: 'ramping-arrival-rate',
      startTime: '2m', startRate: 2000, timeUnit: '1s',
      preAllocatedVUs: 800, maxVUs: 4000,
      stages: [
        { target: 6000, duration: '30s' },       // dependency blips, clients retry
        { target: 6000, duration: '90s' },
        { target: 2000, duration: '30s' },       // recovery
      ],
      env: { DUPLICATE_RATIO: '0.70' },
    },
  },
  thresholds: {
    'idem_registration_ms': ['p(99)<15'],        // the store must stay fast
    'http_req_failed{expected_response:true}': ['rate<0.001'],
    'idem_conflict': ['count>0'],                // if zero, no duplicates collided
  },
};

export default function () {
  const ratio = Number(__ENV.DUPLICATE_RATIO);
  // A duplicate reuses a key from the low end of the pool, so collisions are
  // dense in time; a unique request takes one from the high end.
  const i = Math.random() < ratio
    ? Math.floor(Math.random() * keys.length * 0.3)
    : Math.floor(keys.length * 0.3 + Math.random() * keys.length * 0.7);
  const key = keys[i];

  const res = http.post(`${__ENV.BASE_URL}/v1/charges`,
    JSON.stringify({ amount: 4500, currency: 'gbp', source: 'card_load' }), {
      headers: { 'Content-Type': 'application/json', 'Idempotency-Key': key },
      tags: { name: 'POST /v1/charges' },
    });

  regLatency.add(Number(res.headers['X-Idempotency-Registration-Ms'] || 0));
  dupRatio.add(res.headers['Idempotent-Replay'] === 'true');

  if (res.status === 201 && res.headers['Idempotent-Replay'] === 'false') created.add(1);
  else if (res.status === 201) replayed.add(1);
  else if (res.status === 409) conflicts.add(1);

  check(res, {
    'status is 201 or 409': (r) => r.status === 201 || r.status === 409,
    'never 500': (r) => r.status < 500,
  });
}

The idem_conflict: count>0 threshold is doing real work: if it fails, no two requests ever collided and the entire run measured nothing about deduplication.

Step 4 — Assert the effect count

k6 run --out json=run.json \
  -e BASE_URL=https://staging.example.com storm.js | tee k6.log

# THE assertion. Status codes can look perfect while the handler ran twice.
psql -tA <<'SQL'
SELECT count(*) AS keys_with_duplicate_effects
  FROM (SELECT idempotency_key FROM charges
         WHERE created_at > now() - interval '10 minutes'
         GROUP BY idempotency_key HAVING count(*) > 1) d;
SQL
# expect: 0

# And the ratio that summarises the whole run in one number.
psql -tA -c "SELECT round(count(*)::numeric
                          / nullif(count(DISTINCT idempotency_key), 0), 4)
               FROM charges WHERE created_at > now() - interval '10 minutes'"
# expect: exactly 1.0000
# Confirm the duplicate ratio the script actually achieved matches the intent.
jq -r 'select(.metric=="idem_duplicate_ratio" and .type=="Point") | .data.value' run.json \
  | awk '{s+=$1; n++} END {printf "achieved duplicate ratio: %.3f\n", s/n}'
# expect: within a few points of DUPLICATE_RATIO; far below means the pool is too large

Step 5 — Fail the store mid-run

Measuring failover damage during a storm A chart plots duplicate effects per second against time during a load run. The line sits at zero through the steady and storm phases, spikes sharply during a shaded forced-failover window, and returns to zero once the new primary is serving. The integral of the spike is annotated as the number of duplicate charges a failover costs, which belongs in the degradation policy. dup/s failover forced recovered shaded area = duplicate charges this integer is your failover cost — put it in the degradation policy, not in an assumption nobody wrote down normal operation: 0 
k6 run -e BASE_URL=https://staging.example.com storm.js &
K6=$!

sleep 150                                        # mid-storm
redis-cli -p 26379 SENTINEL FAILOVER dedup-master     # or CLUSTER FAILOVER FORCE
wait "$K6"

psql -tA -c "SELECT count(*) FROM (
               SELECT idempotency_key FROM charges
                WHERE created_at > now() - interval '10 minutes'
                GROUP BY idempotency_key HAVING count(*) > 1) d"
# NOT expected to be zero — record whatever it is. That number is the honest
# answer to 'what happens when the dedup store fails over', and it belongs in
# the degradation policy alongside the fail-open/fail-closed decision.

Run this before choosing between Redis and a durable store, not after. It converts an architectural argument into a measured number.

Failure scenarios and debugging

Failure Scenario Remediation Steps Observability Hooks
Each VU generates its own key, so no duplicates ever collide and the run proves only handler throughput Load a SharedArray key pool and add the idem_conflict: count>0 threshold so a collision-free run fails loudly. idem_conflict count; zero means the test measured nothing about deduplication
Key pool far larger than the request volume, so the achieved duplicate ratio is a fraction of the configured one Size the pool so each key is reused within the store’s round-trip time. Verify with the achieved-ratio calculation in step 4. idem_duplicate_ratio from the run output versus DUPLICATE_RATIO
Run asserts only on status codes, and a broken store passes while executing handlers twice Query effect rows per distinct key after every run and fail the pipeline on any value above 1.0. charges ÷ distinct idempotency_key — the single number that summarises the run
Store sized for the steady rate is overwhelmed at the storm rate, and registration latency dominates the endpoint Size the deduplication store for peak total requests, not unique ones. The store sees every duplicate. idem_registration_ms p99 through the storm phase; alert above 15 ms
Load run leaves 50,000 live keys behind, and the next run’s “fresh” keys are all replays Namespace keys per run — include a run id in the pool — or flush the staging store between runs. Baseline idem_replayed at the start of a run; a non-zero value means leftover state
Pool size decides whether duplicates actually collide Three pool sizes against the same two thousand requests per second. A pool of five million keys yields an achieved duplicate ratio near zero regardless of configuration. Fifty thousand keys yields a ratio close to the configured thirty percent. Five hundred keys over-collides and produces unrealistic contention. The middle option is marked as the target. pool 5,000,000 achieved ratio ≈ 0.00 — the test proves nothing pool 50,000 achieved ≈ 0.30 — matches the configured ratio pool 500 over-collides

SRE / observability checklist

  1. idem_registration_ms — Trend of the store round trip, exported by the service as a response header. Threshold p99 under 15 ms through the storm.
  2. idem_duplicate_ratio — Rate of responses flagged Idempotent-Replay: true. Must land near the configured ratio or the test is not doing what it claims.
  3. idem_conflict — Counter of 409 responses. A zero count fails the run by design.
  4. Effect rows per distinct key — Post-run SQL. Must be exactly 1.0000 outside a deliberate failover experiment.
  5. redis_evicted_keys_total during the run — Must be zero. Eviction under storm load is a silently deleted guarantee.
  6. Failover duplicate count — Recorded per game day and tracked over time; a rising number means the topology or the lease configuration has drifted.