Exactly-Once vs Effectively-Once Semantics Explained

“Exactly-once” is the most oversold phrase in distributed systems, and the confusion it causes is not academic — it shows up as a payment charged twice because someone believed a broker’s marketing over its documentation. This page sits under exactly-once vs at-least-once delivery and pins down what each term can and cannot promise.

Prerequisites. You should understand why a client timeout is ambiguous and have seen an atomic deduplication registration in practice.

Step 1 — Separate delivery from processing

The two words do different work, and almost every argument about exactly-once is two people using one phrase for both.

Delivery is about the wire: how many times a message crosses from sender to receiver. Processing is about the effect: how many times the receiver’s side effect is applied.

Delivery count versus effect count The upper band, labelled delivery layer, shows a sender transmitting the same message three times because acknowledgements were lost. The lower band, labelled processing layer, shows a deduplication gate collapsing those three deliveries into a single application of the effect, with two rejected as duplicates. A caption notes that the delivery count is unbounded and the effect count is exactly one. Delivery layer — count is unbounded sender ack lost ×2 delivery 1, 2, 3 — same message receiver 3 messages in at-least-once is the ceiling here Processing layer — count is exactly one dedup gate atomic register + effect, one commit duplicate 2 → rejected duplicate 3 → rejected 1 effect effectively-once = at-least-once delivery + deduplication

Exactly-once delivery is impossible over an unreliable network. The two-generals argument is short: the sender cannot know its message arrived without an acknowledgement, the acknowledgement can be lost, an acknowledgement of the acknowledgement can also be lost, and no finite exchange terminates with both parties certain. Any protocol that claims otherwise has moved the uncertainty somewhere you are not looking.

Exactly-once processing — usually written effectively-once to keep the distinction visible — is entirely achievable, because the receiver can remember what it has already done.

Step 2 — Locate the deduplication boundary

Effectively-once holds inside a boundary and stops at its edge. The boundary is wherever the deduplication record and the side effect commit together.

-- Inside the boundary: one transaction, both writes. This is effectively-once.
BEGIN;
  INSERT INTO processed_events (event_id) VALUES ($1);   -- unique constraint
  UPDATE balances SET amount = amount - $2 WHERE account_id = $3;
COMMIT;
-- A duplicate hits the unique violation, the whole transaction rolls back,
-- and the balance is untouched. The two facts cannot disagree.
# Outside the boundary: two systems, no shared transaction. This is at-least-once.
redis.set(f"idem:{event_id}", "done", nx=True, px=86_400_000)
stripe.charges.create(amount=amount, source=source)   # a crash here loses the charge/key link

The second snippet is not wrong — it is the normal shape of real systems — but its guarantee is weaker and must be described as such. The window between the two calls is the whole exposure, and closing it is what the transactional outbox pattern exists to do.

Step 3 — Identify every escaping effect

An effect escapes the boundary when it cannot be rolled back by the transaction that guards it. The audit is mechanical: list every line in the handler that touches something outside the transactional store.

Effect Escapes? Consequence of a duplicate Mitigation
UPDATE in the same database No None — rolled back with the dedup row Nothing needed
INSERT into an outbox table in the same database No None Publisher dedupes downstream
Direct publish to Kafka mid-transaction Yes Duplicate event, consumers see it twice Outbox instead of direct publish
Call to an external payment API Yes Duplicate charge Pass an idempotency key to the processor
Sending an email or SMS Yes Duplicate message to a human Dedupe at the sender with a message key
Writing to S3 or object storage Partially Overwrite is harmless if the key is deterministic Deterministic object keys, never uuid4()
Incrementing a Prometheus counter Yes Slightly inflated metric Acceptable — do not engineer around it

The last row matters. Not every escaping effect is worth closing; the engineering judgment is which ones a customer or an auditor could ever observe.

Step 4 — Write the guarantee down

A guarantee with unstated preconditions is a guarantee nobody can verify. Write it as a sentence with the conditions attached.

Charges API. Effectively-once execution per (credential, route, Idempotency-Key) for 24 hours, provided the PostgreSQL primary is available and the client resends an identical key and payload. Outside that window, or when the dedup store is unreachable, the endpoint fails closed and returns 503. Charges submitted to the upstream processor carry a derived idempotency key, so a crash between our commit and their capture resolves to a single charge on reconciliation.

Every clause in that paragraph is testable, and each one names a failure mode someone would otherwise discover in production. Compare with “our API is idempotent”, which is true and useless.

Step 5 — Instrument the assumptions

Three signals that a stated guarantee has stopped holding Three panels. The first tracks effect rows divided by distinct keys, healthy at exactly one and alarming above one point zero zero one. The second tracks keys stuck in the pending state, healthy near zero and alarming when the age exceeds the lease. The third tracks deduplication store availability, where any unavailability means the guarantee's precondition is not met. effects ÷ distinct keys 1.000 — healthy leak page above 1.001 sustained 5 min orphaned pending keys stuck age > lease means the effect/key link broke dedup store availability 99.99% any gap = the stated precondition unmet 
# The single most valuable idempotency metric: effects created per distinct key.
# Healthy is exactly 1.0. Anything above it means the guarantee is leaking.
sum(rate(charges_created_total[5m]))
  / sum(rate(idempotency_keys_registered_total[5m]))

Verification and testing

Confirm the boundary really is atomic by killing the process between the two writes and asserting they agree.

def test_dedup_and_effect_commit_together(db, killer):
    event_id = "evt_atomicity_probe"
    killer.after_statement("INSERT INTO processed_events")   # crash mid-transaction
    with pytest.raises(ProcessKilled):
        handle_event(event_id, amount=4500)

    assert db.scalar("SELECT count(*) FROM processed_events WHERE event_id=:e",
                     e=event_id) == 0
    assert db.scalar("SELECT count(*) FROM ledger_entries WHERE event_id=:e",
                     e=event_id) == 0        # neither, never one

Confirm an escaping effect is actually deduplicated downstream.

# Replay the same event twice and count charges at the processor, not locally.
for i in 1 2; do publish_event evt_escape_probe; done
sleep 5
stripe charges list --limit 10 \
  | jq '[.data[] | select(.metadata.event_id=="evt_escape_probe")] | length'
# expect: 1

Failure scenarios and debugging

Failure Scenario Remediation Steps Observability Hooks
A broker advertising “exactly-once” is trusted end-to-end, and a handler’s external API call is duplicated on rebalance Read the broker’s guarantee as scoped to its own storage. Pass a derived idempotency key to every external call so the guarantee extends past the broker’s boundary. external_call_duplicate_total derived from the processor’s own idempotent-replay signal; alert on any non-zero rate
Dedup row and side effect are written to different databases, and a crash leaves the effect without the row Move the dedup row into the same database as the effect, or adopt an outbox so the escaping publish is derived from a committed row rather than issued alongside it. idempotency_phantom_effect_total from a reconciliation job; idempotency_pending_age_seconds above the lease
The stated guarantee says 24 hours but the store’s TTL was reduced to 1 hour during a capacity incident and never restored Assert the configured TTL in a startup check against the published contract value and fail the deploy on mismatch. idempotency_configured_ttl_seconds gauge compared against a constant in the alert expression
Effect count per key drifts above 1.0 slowly after a deploy, unnoticed because no request errored Add the ratio query above as a standing alert. Nothing in this failure mode produces an error, so error-rate monitoring cannot see it. charges_created_total / idempotency_keys_registered_total — page above 1.001 sustained for 5 minutes
The five clauses a written guarantee needs Five labelled blocks forming one sentence: the scope of the key, the window it is honoured for, what a repeat call receives, what happens when the store is unavailable, and where the guarantee stops applying. Each is annotated as independently testable, in contrast to the phrase "our API is idempotent", which is neither. scope (cred, route, key) window 24 hours replay original status, verbatim degradation fail closed, 503 boundary external calls excluded Each clause is independently testable and names a failure mode. "Our API is idempotent" is true, and tells an integrator nothing.

SRE / observability checklist

  1. effect_per_key_ratio — Derived gauge of effect rows over distinct registered keys. Page above 1.001 sustained for 5 minutes; this is the definitive leak detector.
  2. idempotency_pending_age_seconds — Histogram of how long keys sit in pending. Alert above 120 seconds, which means an effect and its record have decoupled.
  3. idempotency_configured_ttl_seconds — Gauge exported at startup and compared against the published contract. Catches silent configuration drift.
  4. external_call_idempotent_replay_total — Counter of downstream calls the processor reported as replays. A healthy value is small and non-zero; exactly zero usually means keys are not being propagated at all.
  5. guarantee_precondition_met — Boolean gauge combining store availability and TTL correctness, so a dashboard can show at a glance whether the written guarantee currently holds.