Writing a Duplicate Charge Runbook

A runbook is judged by one number: how long it takes an unfamiliar responder to stop the bleeding. Everything else — root-cause guidance, background, links — is secondary and belongs below the fold. This page gives the structure and the content, and sits under incident response for duplicate processing.

Prerequisites. The containment levers described below must actually exist and be tested; a runbook documenting a feature flag nobody has built is worse than none, because it consumes the responder’s first five minutes.

Step 1 — Containment first, above the fold

Runbook layout by time budget A page layout divided into three horizontal bands. The top band, marked zero to two minutes and shaded strongest, contains containment commands to be copied and run. The middle band, marked two to fifteen minutes, contains parameterised detection queries and the blast-radius summary. The bottom band, marked fifteen minutes onward, contains remediation authorisation, customer communication and root-cause background. A note states that anything requiring reading before acting belongs below the fold. 0–2 min · CONTAIN three copy-and-run commands, no prose between them flag flip · deploy rollback · gateway retry cap above the fold 2–15 min · SCOPE parameterised detection query, frozen into an incident table three numbers for the incident channel: charges, customers, value 15 min+ · REMEDIATE & EXPLAIN authorisation checklist · dry run · customer comms · root cause below the fold If the first thing on the page is a paragraph, the runbook is wrong. 
# RUNBOOK: Duplicate charges on /v1/charges

## STOP THE BLEEDING — run one of these now

    # 1. Fail the endpoint closed. Rejects new charges; stops duplicates dead.
    flagctl set charges.idempotency_required strict --reason "$INCIDENT_ID"

    # 2. If a deploy in the last 2 hours is suspected, roll it back instead.
    kubectl rollout undo deployment/payments-api -n payments

    # 3. If duplicates come from ONE client minting fresh keys, cap its retries.
    gwctl route charges set-retry-policy --credential "$CRED" --max-attempts 1

**Record the containment time now:** `date -u +%FT%TZ` → paste into the incident channel.

Three commands, no prose between them, the credential and incident id as the only variables. A responder should be able to act from this block without reading a sentence.

Step 2 — Commit the detection queries

Queries live in the repository next to the remediation script, not in the wiki page. A query in version control can be reviewed, tested against a snapshot, and fixed when a column is renamed.

-- runbooks/duplicate_charges/detect.sql
-- Usage: psql -v from="'2026-08-01 09:12:00Z'" -v to="'2026-08-01 14:41:00Z'" \
--             -v incident="'incident_2026_08_01'" -f detect.sql
\set table_name :incident '_duplicates'

CREATE TABLE :table_name AS
WITH windowed AS (
    SELECT id, customer_id, amount_minor, currency, idempotency_key,
           processor_charge_id, created_at,
           date_trunc('second', created_at)
             - (extract(second FROM created_at)::int % 10) * interval '1 second' AS bucket
      FROM charges
     WHERE created_at >= :from::timestamptz
       AND created_at <  :to::timestamptz
       AND status IN ('succeeded', 'captured')
), grouped AS (
    SELECT customer_id, amount_minor, currency, bucket, count(*) AS n,
           (array_agg(id ORDER BY created_at))[1] AS keep_id,
           array_agg(id ORDER BY created_at)      AS all_ids
      FROM windowed
     GROUP BY customer_id, amount_minor, currency, bucket
    HAVING count(*) > 1
)
SELECT customer_id, amount_minor, currency, n, keep_id,
       unnest(all_ids[2:]) AS reverse_id, 'pending'::text AS reversal_state
  FROM grouped;
-- The three numbers the incident channel needs within five minutes.
SELECT count(*)                    AS charges_to_reverse,
       count(DISTINCT customer_id) AS customers_affected,
       sum(amount_minor)/100.0     AS value_to_return
  FROM :table_name;

The 10-second bucket is the tunable that decides false positives. Document why it is 10 seconds — “two identical charges more than 10 s apart are plausibly a genuine repeat purchase for this product” — so a future responder can adjust it knowingly rather than guessing.

Step 3 — The authorisation checklist

No bulk write to financial data happens without this, and it belongs in the runbook as a literal checklist the responder ticks in the incident channel.

## Before any reversal is authorised

- [ ] Containment confirmed: `charges_created_total / idempotency_keys_registered_total`
      has returned to 1.000 for at least 10 minutes
- [ ] Detection query run and frozen into `<incident>_duplicates` (never remediate
      from a live query)
- [ ] Processor cross-check: exported window row count matches the incident table
      within 1%, or the difference is explained in the channel
- [ ] Twenty rows sampled and reviewed by a human; no genuine repeat purchases present
- [ ] Dry run executed: `reverse.py --run-id <incident> --dry-run` output attached
- [ ] Incident commander AND one finance approver have said "proceed" in the channel
- [ ] Support tooling frozen for the affected customers so agents do not refund in parallel

The dry-run output is the artefact the approvers actually read. Make the script print a one-line-per-charge projection — customer, amount, intended action, current processor state — so approval is a review rather than an act of faith.

# runbooks/duplicate_charges/reverse.py — committed, reviewed, dry-runnable.
python3 reverse.py --run-id incident-2026-08-01 --dry-run | tee dryrun.txt
wc -l dryrun.txt                      # must equal charges_to_reverse
grep -c 'would_refund\|would_void' dryrun.txt
grep -c 'manual_review'  dryrun.txt   # any value needs an owner before proceeding

Step 4 — Pre-write the customer communication

Drafting a message under time pressure produces either silence or a claim you cannot support. Template it, with placeholders that map to columns in the incident table.

Subject: We charged you {{ n }} time(s) in error on {{ date }} — refund issued

Hello {{ first_name }},

On {{ date }} a fault in our payment system charged your card {{ n }} times for the
same {{ description }}. You should have been charged once.

We have refunded {{ extra_amount }} to the card ending {{ last4 }}. Card refunds
usually appear within {{ refund_eta }}; your bank controls the exact timing.

Reference: {{ incident_public_ref }}. If the refund has not appeared by
{{ refund_deadline }}, reply to this message and we will chase it directly.

We are sorry. The specific fault has been fixed and we have added a monitor that
would have caught it before you did.

Send it after the reversal_audit rows exist, never before. Every factual claim in the message — the count, the amount, the refund reference — should be a column in a row you can produce on request. A message promising a refund that later fails is a second incident with the same customer.

Step 5 — Drill it quarterly

The quarterly runbook drill and its feedback loop Four stages in a cycle. Duplicates are injected in staging. An unfamiliar responder works the runbook while two checkpoints are timed: containment within two minutes and scoping within fifteen. Defects found, such as a renamed flag or a rotated credential, are recorded. Those defects are fixed in the runbook before the next quarter, closing the loop. inject 200 duplicates in staging unfamiliar responder works the runbook no help from the author time two checkpoints contained ≤ 2 min scoped ≤ 15 min record the actuals defects renamed flag, stale query fix before the next quarter — a drill that finds nothing was not a drill 
# runbooks/duplicate_charges/drill.sh — inject, then get out of the way.
set -euo pipefail
KEY_BASE="drill-$(date +%s)"
for i in $(seq 1 200); do
  for attempt in 1 2 3; do                       # deliberate triple-send
    curl -s -o /dev/null -XPOST "$STAGING/v1/charges" \
      -H "Idempotency-Key: ${KEY_BASE}-${i}" \
      -H 'Content-Type: application/json' \
      -d '{"amount":4500,"currency":"gbp","source":"card_drill"}' &
  done
done
wait
echo "drill duplicates injected at $(date -u +%FT%TZ) — hand the pager over"

Record two timings every drill: minutes to containment and minutes to a frozen incident table. Track them across quarters. A number that grows means the runbook has drifted from reality — usually a renamed flag, a rotated credential, or a query against a column that no longer exists.

Failure scenarios and debugging

Failure Scenario Remediation Steps Observability Hooks
Runbook opens with three paragraphs of context, and the responder spends four minutes reading while duplicates accumulate Move containment to the top and everything explanatory below the fold. Measure time-to-containment in the drill; above two minutes, restructure. Drill metric minutes_to_containment; a rising trend means the page has grown a preamble
Containment lever documented but never built, so the first command fails Test every lever in the quarterly drill and mark any that fails as a blocking defect. A documented, non-existent lever is worse than none. Drill checklist with pass/fail per lever; any failure blocks sign-off
Detection query pasted from the wiki references a column dropped three months ago Keep queries in the repository, run them against a staging snapshot in CI, and fail the build when a referenced column disappears. A CI job executing every runbook query with EXPLAIN against the current schema
Support agents refund manually while the batch job runs, double-refunding some customers Add “freeze support tooling for affected customers” to the authorisation checklist, and have the reversal job re-read processor state immediately before each action. reversal_state_changed_since_snapshot_total; any non-zero value pauses the job
Customer email sent before the audit rows exist, promising refunds that later fail Gate the communication step on reversal_audit row count matching the reversal count. Never send from the projection. A pre-send assertion comparing audit rows to the incident table; block on mismatch
The number the drill exists to produce A chart of minutes to containment across four quarterly drills. The first two sit near the two-minute target. The third rises because a feature flag was renamed, and the fourth rises further because a credential rotated. Each rise is annotated with the defect the drill uncovered, which is the drill's actual output. 2 min target Q1 Q2 Q3 flag renamed Q4 credential rotated

SRE / observability checklist

  1. minutes_to_containment — Recorded per drill and per real incident. The target is under 2 minutes; the trend matters more than any single value.
  2. minutes_to_frozen_scope — Time from detection to a materialised incident table. Target under 15 minutes.
  3. runbook_lever_test_pass_rate — Share of containment levers that worked in the last drill. Anything below 100% is a blocking defect.
  4. runbook_query_ci_failures — Count of committed runbook queries failing their schema check. Catches drift between drills.
  5. charges_created_total / idempotency_keys_registered_total — The detection signal the runbook’s first checkpoint depends on. Page above 1.001 sustained for 5 minutes.
  6. incident_id on every command run during the incident — Via the --reason flag on flag flips and the run_id on scripts, so the audit trail reconstructs itself.