Testing & Verification of Idempotent Systems

An idempotent endpoint that has never been tested under concurrency is an endpoint you hope is idempotent. This page is part of Observability & Operations for Idempotent Systems, and it covers how to convert that hope into evidence: the test matrix that matters, the invariants worth expressing as properties, and the harnesses that reproduce the conditions under which deduplication actually fails.

The distinguishing feature of idempotency bugs is that they are invisible to the tests most teams already write. A handler with a fatal check-then-write race passes every sequential test, every integration test, every staging soak with one client. It fails the first time two retries land in the same 3-millisecond window in production.

---

Problem framing

Deduplication correctness has three independent axes, and a test suite that covers only one gives false confidence.

The sequential axis asks whether a repeated request returns the stored response instead of executing again. This is what almost every existing test covers, and it is the least likely thing to be broken.

The concurrent axis asks whether two requests that both observe an absent key can both execute. This is where the real bugs live, because the guarantee reduces entirely to whether the absent → pending transition is one atomic operation, and nothing about a sequential test can tell you.

The crash axis asks what happens when the process dies between the side effect and the key write. Here the system is not wrong so much as ambiguous, and the test’s job is to pin down which of the two acceptable outcomes you chose — an orphaned pending key that reconciles, or a lost key that permits a second execution — and assert it deliberately.

---

Guarantee model

A test suite cannot prove idempotency in the mathematical sense; it can only fail to disprove it, with a confidence proportional to how adversarially it tried. What a well-built suite does provide is a precise statement of which guarantee you are claiming:

• Effect-count invariance — for any number of deliveries of the same key, the observable side effect happens exactly once. This is the property that matters to a customer.
• Response invariance — every delivery after the first returns the same status and body. This is the property that matters to a client SDK.
• Termination — no delivery blocks indefinitely waiting for another, and no key remains pending past its lease. This is the property that matters to your on-call rota.

The guarantee weakens exactly where the test conditions weaken. Testing against an in-memory fake proves nothing about atomicity. Testing with two threads proves almost nothing about a race whose window is a single network round trip. Testing without crash injection says nothing about the ambiguity described above. Each gap in the harness is a gap in the claim, and it is worth writing the claim down next to the suite so nobody later mistakes green for proven.

---

The four tests every idempotent endpoint needs

1. Sequential replay

The baseline. Call, call again, assert one effect and identical responses.

def test_sequential_replay(client, db):
    key = "01HXYZ3ABCDEFGHJKMNPQRSTVW"
    first = client.post("/v1/charges", json=PAYLOAD, headers={"Idempotency-Key": key})
    second = client.post("/v1/charges", json=PAYLOAD, headers={"Idempotency-Key": key})

    assert first.status_code == 201
    assert second.status_code == 201                       # NOT 200 — replay the original
    assert second.json() == first.json()                   # byte-identical body
    assert second.headers["Idempotent-Replay"] == "true"
    assert db.scalar("SELECT count(*) FROM charges WHERE key = :k", k=key) == 1

2. Concurrent duplicates released from a barrier

The test that finds real bugs. The point of the barrier is to make all requests reach the store within the same round trip, which two sequential goroutines will never do.

func TestConcurrentDuplicates(t *testing.T) {
    const N = 64
    for trial := 0; trial < 50; trial++ {          // a 1-in-20 race is still an incident
        key := ulid.Make().String()
        start := make(chan struct{})               // barrier: nothing runs until closed
        var wg sync.WaitGroup
        codes := make([]int, N)

        for i := 0; i < N; i++ {
            wg.Add(1)
            go func(i int) {
                defer wg.Done()
                <-start                            // all N goroutines unblock together
                codes[i] = post(t, "/v1/charges", key, payload)
            }(i)
        }
        close(start)
        wg.Wait()

        created, conflicts := tally(codes)
        require.Equal(t, 1, created, "trial %d: %d handlers executed", trial, created)
        require.Equal(t, N-1, conflicts)
        require.Equal(t, 1, chargeCount(t, key))   // the effect, not just the status codes
    }
}

Assert on the effect count, not only on status codes. A broken implementation can return one 201 and 63 409s while still having executed the handler twice, if the second execution lost a write race and its response was discarded.

3. Crash between effect and key write

Inject the crash at the exact seam the design is ambiguous about, then assert the recovery behaviour you chose.

def test_crash_after_effect_before_key_completion(client, db, redis, monkeypatch):
    key = "01HCRASHTEST0000000000000A"

    # Fail the completion write only — the charge has already been committed.
    def boom(*args, **kwargs):
        raise ConnectionError("redis died after the charge committed")
    monkeypatch.setattr(redis, "hset", boom)

    with pytest.raises(ConnectionError):
        client.post("/v1/charges", json=PAYLOAD, headers={"Idempotency-Key": key})

    assert db.scalar("SELECT count(*) FROM charges WHERE key = :k", k=key) == 1
    assert redis.hget(f"idem:{key}", "state") == b"pending"    # orphan, not absent

    # The chosen policy: the reconciler resolves the orphan, the client's retry replays.
    reconcile_orphaned_keys(max_age_seconds=0)
    retry = client.post("/v1/charges", json=PAYLOAD, headers={"Idempotency-Key": key})
    assert retry.status_code == 201
    assert db.scalar("SELECT count(*) FROM charges WHERE key = :k", k=key) == 1

4. Payload mismatch on a reused key

The contract test that catches a client bug rather than a server one.

test('same key + different body is rejected, not silently honoured', async () => {
  const key = '01HXYZ3ABCDEFGHJKMNPQRSTVW';
  const first = await api.post('/v1/charges', { amount: 4000 }, { key });
  const second = await api.post('/v1/charges', { amount: 4500 }, { key });

  expect(first.status).toBe(201);
  expect(second.status).toBe(422);
  expect(second.body.code).toBe('idempotency_key_payload_mismatch');
  expect(await chargeTotalFor(key)).toBe(4000);   // the second amount never landed
});

---

Property-based verification

Example-based tests check the cases you thought of. Property-based tests generate the ones you did not — particularly the interleavings. The generator produces a random delivery schedule; the assertion never changes.

from hypothesis import given, settings, strategies as st

@settings(max_examples=300, deadline=None)
@given(
    deliveries=st.integers(min_value=1, max_value=12),
    gaps_ms=st.lists(st.integers(min_value=0, max_value=40), min_size=0, max_size=12),
    kill_after=st.one_of(st.none(), st.integers(min_value=0, max_value=11)),
)
def test_effect_count_is_always_one(deliveries, gaps_ms, kill_after, client, db, redis):
    key = f"prop-{uuid.uuid4()}"
    responses = []
    for i in range(deliveries):
        if kill_after is not None and i == kill_after:
            simulate_store_failure(redis)            # crash injected mid-schedule
        responses.append(client.post("/v1/charges", json=PAYLOAD,
                                     headers={"Idempotency-Key": key}))
        if i < len(gaps_ms):
            time.sleep(gaps_ms[i] / 1000)
        restore_store(redis)

    successes = [r for r in responses if r.status_code < 300]
    assert db.scalar("SELECT count(*) FROM charges WHERE key = :k", k=key) <= 1
    if successes:
        assert all(r.json() == successes[0].json() for r in successes)

Note the <= 1 rather than == 1: under an injected store failure with a fail-closed policy, zero executions is a correct outcome. Encoding “exactly one or none, never two” is the honest invariant, and it is stronger than it looks — the bug you are hunting always produces two.

---

Implementation variants

The first two belong in the pull-request pipeline; the rest do not, because a 10-minute property run in the critical path gets deleted within a quarter. Wire them to a nightly job that files an issue on failure with the shrunk counterexample attached.

Containerised stores, not mocks. Use Testcontainers, docker compose, or an equivalent to bring up the real Redis or PostgreSQL. The entire guarantee is a claim about one store operation’s atomicity; a mock implements that operation inside the test process, where it is trivially atomic and therefore proves nothing. This is the single highest-leverage change most suites can make.

Seed determinism. Property-based and load harnesses must log their seed and accept it as an input. A race reproduced once and never again is a race you will argue about instead of fixing.

---

Edge cases and failure scenarios

---

Operational concerns

Runtime budgeting. A 64×50 concurrency test against a containerised Redis costs roughly 3,200 requests and completes in 1 to 3 minutes on ordinary CI hardware. That is affordable per commit. The same test at 512×200 costs 102,400 requests and belongs nightly. Decide the split explicitly, because the failure mode of an over-long pull-request suite is that someone marks it skip.

Store lifecycle. Give each test its own key namespace — prefix every key with the test name and a random suffix — rather than flushing the store between tests. Flushing serialises the suite and hides cross-test leakage that a shared namespace would reveal.

Index and data volume. Load tests that generate millions of keys against a PostgreSQL-backed store will expose index bloat that a small suite never touches, which is genuinely useful signal. Run pgstatindex before and after the load run and record the leaf fragmentation delta; a sharp rise means the key format is fighting the B-tree, as UUIDv4 vs UUIDv7 vs ULID explains.

Alert thresholds for the tests themselves. Track the concurrency suite’s pass rate over the last 100 runs. A rate below 100% is not flakiness to be tuned away — it is the suite reporting a real race at a lower frequency than production will hit it. Page the owning team on the second failure in a week.

Relationship to chaos work. These harnesses verify the code; chaos engineering for idempotency verifies the deployment. They are complementary and neither substitutes for the other: a suite that proves the handler is atomic says nothing about what a Redis failover does to in-flight keys, and a game day that survives a failover says nothing about a race that needs 64 simultaneous callers to appear.

---

FAQ

Why does a passing unit test not prove an endpoint is idempotent?

Because the common bug is a race, and a sequential test never creates one. Calling the handler twice in a row exercises the completed-key path; it never exercises two requests observing an absent key simultaneously. A test that does not run the duplicates concurrently, against a real store, cannot distinguish a correct atomic registration from a check-then-write that is broken under load.

What is the core invariant an idempotency property test should assert?

That for any sequence of N deliveries of the same key, the observable side-effect count is at most one — exactly one when no failure was injected — and every successful response is byte-identical to the first. Generate the sequence length, the interleaving, and the failure injections randomly; assert those same two properties every time.

Should idempotency tests use a real Redis or PostgreSQL, or a mock?

A real one, in a container. The entire guarantee rests on the atomicity of one store operation, and a mock reimplements that atomicity in the test process where it is trivially correct. Testing against a mock verifies that you called the right function, not that the function does what you need under concurrency.

How many concurrent duplicates should a test fire?

Enough to beat the store’s round-trip time — typically 32 to 128 requests released from a single barrier. A pair of goroutines or threads usually serialises by accident and proves nothing. Run the assertion loop 50 times per CI run, because a race that fails one time in twenty is still a production incident.

Do I need to test the client’s retry logic as well as the server’s deduplication?

Yes, and it is frequently the weaker half. A client that mints a fresh key on every attempt defeats a perfectly correct server, and the only place that shows up is a test asserting the key is stable across retries. Cover it alongside the backoff schedule described in retry logic & backoff fundamentals.

---

Related

• Observability & Operations for Idempotent Systems — the parent section covering metrics, tracing, chaos work and incident response.
• Property-Based Testing for Idempotent Handlers — building generators, oracles and shrinkers for delivery schedules in Python and Go.
• Load Testing Duplicate Request Storms with k6 — a scripted retry storm with a realistic duplicate ratio and effect-count assertions.
• Chaos Engineering for Idempotency — sibling area covering failure injection against real deployments rather than test harnesses.
• Monitoring Idempotency Metrics & Dashboards — the production signals these tests are ultimately protecting.