Incident Response for Duplicate Processing

Every idempotency defence eventually fails once, and the difference between a bad afternoon and a regulatory problem is whether the response was rehearsed. This page is part of Observability & Operations for Idempotent Systems, and it covers what happens after the guarantee has already broken: finding the duplicates, bounding the damage, reversing safely, and proving afterwards that you reversed exactly what you said you did.

Duplicate-processing incidents have a distinctive shape. They are usually detected by customers rather than monitors, the damage is already done by the time anyone looks, and the remediation itself is a bulk write to production financial data under time pressure — which is to say, the remediation is the second-most dangerous thing that will happen that day.

---

Problem framing

The incident begins somewhere upstream: a deploy removed the NX flag, a Redis failover dropped the key space, a client library started minting a fresh key per attempt, a queue consumer’s offset commit stopped being atomic with its side effect. By the time the duplicate charges appear in a support queue, the causal window may be hours old and may still be open.

That gives the response two independent objectives that compete for the same responder. Containment stops new duplicates. Remediation reverses the existing ones. Containment always wins, because remediation against a still-producing source is unbounded work — you finish reversing yesterday’s duplicates while today’s accumulate behind you.

The gap between detection and containment is the one number a team can actually improve in advance, and it is improved by having the containment lever pre-built rather than by responding faster.

---

Guarantee model

Remediation makes a different promise from the system it is repairing. The original endpoint promised effectively-once execution; the remediation job promises exactly-once reversal of an immutable, pre-computed set. That reframing matters, because it means the remediation job must itself be idempotent — it will be re-run, probably by a tired person, probably twice.

Three properties define a safe remediation:

• Set immutability. The list of records to reverse is computed once and frozen. A job that re-derives its work list on each run will pick up records created after containment, or miss records whose state changed mid-run.
• Per-record idempotency. Reversing a record twice must be a no-op. The natural mechanism is the same one that failed upstream: an idempotency key on the reversal call, derived deterministically from the record being reversed.
• Auditability. Every reversal writes a row saying what was reversed, when, by which run, and with which outcome. Without it you cannot answer the only question that matters afterwards, which is whether a specific customer was made whole.

The guarantee breaks where the external system of record disagrees with yours. A charge your database shows as duplicated may have already been refunded manually by a support agent; a charge your database shows as single may have been captured twice by the processor. Reconciliation against the external system is not an optional final step — it is the only thing that makes the reversal claim true rather than merely plausible.

---

Phase 1 — Contain

Containment is a lever, not an investigation. Build the lever before the incident and make it a single, reversible action.

# Option A: fail the endpoint closed. Duplicates stop; so does legitimate traffic.
# Correct for money movement, where a rejected charge is far cheaper than a doubled one.
curl -XPOST "$FLAGS/v1/flags/charges.idempotency_required" -d '{"value":"strict"}'

# Option B: roll back the deploy that opened the window. Preferred when the window
# maps cleanly to one release, because it restores the guarantee rather than trading it.
kubectl rollout undo deployment/payments-api --to-revision="$LAST_GOOD"

# Option C: disable the retry path at the client edge, when the duplicates originate
# from a caller minting fresh keys rather than from the server losing them.
curl -XPOST "$GATEWAY/v1/routes/charges/retry-policy" -d '{"max_attempts":1}'

Record the containment timestamp immediately and in a durable place. Every subsequent query is bounded by it, and a containment time reconstructed later from chat scrollback is a query bound you cannot defend in a review.

---

Phase 2 — Scope the blast radius

Detection queries are the part of the runbook most worth writing in advance, because they are the part hardest to write correctly at 02:00. The identity you group by is a business decision, not a technical one: two charges for the same customer and amount ninety minutes apart are probably legitimate; two within four seconds are probably not.

-- Freeze the work list into an immutable incident table. Never remediate from a
-- live query — it moves under you between the count and the fix.
CREATE TABLE incident_2026_08_01_duplicates AS
WITH windowed AS (
    SELECT id, customer_id, amount_minor, currency, idempotency_key,
           processor_charge_id, created_at,
           -- 10-second bucket: tight enough to exclude genuine repeat purchases
           date_trunc('second', created_at)
             - (extract(second FROM created_at)::int % 10) * interval '1 second' AS bucket
      FROM charges
     WHERE created_at >= TIMESTAMPTZ '2026-08-01 09:12:00Z'   -- deploy
       AND created_at <  TIMESTAMPTZ '2026-08-01 14:41:00Z'   -- containment
       AND status IN ('succeeded', 'captured')
), grouped AS (
    SELECT customer_id, amount_minor, currency, bucket,
           count(*)                              AS n,
           min(created_at)                       AS first_at,
           (array_agg(id ORDER BY created_at))[1] AS keep_id,
           array_agg(id ORDER BY created_at)      AS all_ids
      FROM windowed
     GROUP BY customer_id, amount_minor, currency, bucket
    HAVING count(*) > 1
)
SELECT g.customer_id, g.amount_minor, g.currency, g.n, g.first_at, g.keep_id,
       unnest(g.all_ids[2:]) AS reverse_id,       -- keep the earliest, reverse the rest
       'pending'::text       AS reversal_state
  FROM grouped g;

-- The three numbers the incident channel needs within five minutes.
SELECT count(*)                        AS charges_to_reverse,
       count(DISTINCT customer_id)     AS customers_affected,
       sum(amount_minor) / 100.0       AS value_to_return
  FROM incident_2026_08_01_duplicates;

Cross-check the count against the processor before acting. A discrepancy between your table and theirs is itself a finding, and reversing on the strength of your own records alone is how a single incident becomes two.

# Stripe example: list charges in the window and diff against the incident table.
stripe charges list --created "gte=1785920000" --created "lt=1785939660" --limit 100 \
  | jq -r '.data[] | [.id, .metadata.idempotency_key, .amount, .status] | @csv' \
  > processor_window.csv

---

Phase 3 — Reverse safely

The remediation job is a batch process against production financial data. Treat it with the ceremony that deserves: dry run first, small batches, deterministic idempotency keys, an audit row per action, and a hard stop on unexpected states.

REVERSAL_RUN = "incident-2026-08-01"

def reverse_batch(rows, dry_run: bool = True, batch_size: int = 50) -> None:
    for row in rows[:batch_size]:
        charge = processor.charges.retrieve(row.processor_charge_id)

        if charge.refunded or charge.dispute is not None:
            audit(row, action="skip", reason="already_refunded_or_disputed"); continue

        # The reversal key is DERIVED, not random: re-running the job produces the
        # same key, and the processor's own idempotency turns the retry into a no-op.
        reversal_key = f"{REVERSAL_RUN}:reverse:{row.processor_charge_id}"

        if charge.status == "succeeded" and not charge.captured:
            action, call = "void", lambda: processor.charges.cancel(
                charge.id, idempotency_key=reversal_key)
        elif charge.captured:
            action, call = "refund", lambda: processor.refunds.create(
                charge=charge.id, reason="duplicate", idempotency_key=reversal_key)
        else:
            audit(row, action="manual_review", reason=f"unexpected_state:{charge.status}")
            continue

        if dry_run:
            audit(row, action=f"would_{action}", reason="dry_run"); continue

        result = call()
        audit(row, action=action, reason="duplicate", processor_ref=result.id)
        mark_reversed(row.reverse_id, run=REVERSAL_RUN, processor_ref=result.id)

-- The audit table. Written before the customer email, not after.
CREATE TABLE reversal_audit (
    id              bigserial PRIMARY KEY,
    run_id          text        NOT NULL,
    charge_id       uuid        NOT NULL,
    customer_id     uuid        NOT NULL,
    amount_minor    bigint      NOT NULL,
    action          text        NOT NULL,   -- void | refund | skip | manual_review
    reason          text        NOT NULL,
    processor_ref   text,
    created_at      timestamptz NOT NULL DEFAULT now(),
    UNIQUE (run_id, charge_id)              -- re-running the job cannot double-write
);

Run with dry_run=True first and diff the projected actions against the frozen list. The dry run’s output is what you show the incident commander before anyone authorises a write.

---

Edge cases and failure scenarios

---

Operational concerns

Pre-built levers. The single highest-value preparation is a feature flag that switches an endpoint to fail-closed, tested quarterly. The second is the detection query, parameterised by window and committed to the repository next to the runbook rather than pasted into an incident channel.

Batch sizing and rate limits. Processors rate-limit refund creation, commonly to a few requests per second per account. A 12,000-charge reversal at 5 requests per second is 40 minutes of continuous calls — plan for it, checkpoint after every batch, and make the job resumable from its audit table rather than from the top.

Storage budgeting. Incident tables are small but must outlive everything around them. A 12,000-row incident table with an audit row per action is well under 10 MB, and it needs the same retention as the underlying payment records — commonly seven years. Do not let it be swept up by a routine DROP TABLE of temporary analysis artefacts; name it with an incident_ prefix that your retention policy recognises.

Alert thresholds that would have caught it sooner. The metric that detects this class of incident is not an error rate — nothing errored. It is a ratio: charges created per distinct idempotency key. A healthy value is 1.00; anything above 1.001 sustained over five minutes means the guarantee is leaking. Pair it with the replay-rate signal from monitoring idempotency metrics & dashboards, because a replay rate that collapses to zero is the same event seen from the other side.

Customer communication. Draft the notification template in advance, with placeholders for the count, the window, and the refund timeline. Send it after the audit rows exist, so every claim in the message is backed by a queryable record. The message should state what happened, how many charges were affected for that customer, whether the money has already been returned, and when it will land — in that order.

Post-incident. The review should produce exactly two artefacts beyond the narrative: a test that reproduces the failure — a barrier-released concurrency case or a failover scenario, as in testing & verification of idempotent systems — and a monitor that would have fired before a customer did. An incident that produces neither will recur.

---

FAQ

What is the first action in a duplicate-charge incident?

Stop new duplicates being created, before you investigate why. That usually means flipping the endpoint to fail-closed, rolling back the deploy that removed the guarantee, or disabling the client retry path. Every minute spent diagnosing while the endpoint is still live grows the number of customers you must later refund, and remediation against a still-producing source never converges.

How do I find every duplicate without scanning the whole table?

Bound the query by the incident window first — from the causal event to containment — then group by the business identity: customer, amount, currency and a 10-second bucket, with HAVING count(*) > 1. Materialise the result into an immutable incident table so remediation, communication and the post-incident review all work from the same fixed list rather than re-running a query against moving data.

Should duplicate charges be voided or refunded?

Void if the charge has not yet settled, because a void leaves no trace on the customer’s statement and costs nothing in processing fees. Refund once it has settled. The decision is per-charge and depends on the processor’s settlement state, so the remediation job must query that state immediately before acting rather than trusting a snapshot taken an hour earlier.

How long should the incident record be retained?

At least as long as the financial retention obligation for the underlying transactions, which is commonly seven years for payment records. The incident and audit tables are the evidence of what was reversed and why, and they are the first thing an auditor or a disputing customer will ask for.

Can the remediation job itself create duplicates?

Yes, and it is the most likely place for a second incident. Protect it exactly as you would the original endpoint: derive the reversal idempotency key deterministically from the run ID and the charge ID, enforce a unique constraint on the audit table, and dry-run before every real execution. A remediation job with a random key per attempt is the same bug wearing a different hat.

---

Related

• Observability & Operations for Idempotent Systems — the parent section covering the metrics, traces and chaos work that keep incidents rare.
• Reconciling Duplicate Payments After an Incident — the three-way reconciliation between your ledger, the processor, and the incident audit table.
• Writing a Duplicate Charge Runbook — the on-call document itself: containment levers, parameterised queries, and the authorisation checklist.
• Monitoring Idempotency Metrics & Dashboards — sibling area covering the charges-per-key ratio and replay-rate signals that shorten detection.
• Testing & Verification of Idempotent Systems — sibling area on the regression tests every duplicate-processing post-incident review should produce.