Exactly-Once vs At-Least-Once Delivery

Part of: Idempotency Fundamentals & API Guarantees

Every distributed system that sends a message, publishes an event, or calls a remote API must pick a delivery semantic — and that choice is not a implementation detail, it is a contract that ripples through every downstream consumer. Get it wrong and you either lose money-relevant events silently or process a payment twice. This page defines the three canonical delivery semantics precisely, proves why one of them cannot exist at the transport layer, and shows exactly how production systems manufacture its effect anyway through idempotent consumers and deduplication stores.

---

Problem Framing: Why Delivery Semantics Are a Spectrum, Not a Switch

The Two Generals Problem

Two generals command armies on opposite hills, facing a fortified city between them. They can only win by attacking simultaneously; attacking alone means annihilation. They can only communicate by sending a messenger across the valley, and the messenger might be captured. General A sends “attack at dawn.” To be sure General B received it, A wants an acknowledgment. But the acknowledgment can also be lost — so B wants an acknowledgment of the acknowledgment, and so on, forever. No finite exchange of messages over an unreliable channel can give both parties simultaneous certainty that the other will act. This is not an engineering limitation to be optimized away; it is a proof by induction that terminates in the same unresolved state at every step.

Map this directly onto a service calling another service, or a producer publishing to a broker: the sender can never be mathematically certain the receiver processed the message, because the acknowledgment itself travels over the same unreliable network. This is why exactly-once delivery — a guarantee enforced entirely by the transport — is impossible on any network that can drop, delay, or duplicate packets, which is every network that exists.

The Three Semantics, Defined Precisely

• At-most-once delivery. The sender transmits once and does not retry. A message is delivered zero or one times. Failure mode: silent loss. No duplicates are possible because nothing is ever resent.
• At-least-once delivery. The sender retries until it receives an acknowledgment, using exponential backoff with jitter to avoid overwhelming the receiver. A message is delivered one or more times. Failure mode: duplication — the ack can be lost even after the receiver successfully processed the message, causing a resend of a message that already took effect.
• Exactly-once processing. Not a delivery guarantee at all — an application-level outcome. The transport still delivers at-least-once (with all its duplicates), but the receiver is idempotent: it recognizes a duplicate by a stable identifier and discards it before any side effect executes a second time. The message may arrive five times; the ledger entry, the email, the inventory decrement happens once.

Conflating “exactly-once delivery” with “exactly-once processing” is the single most common vendor-marketing error in this space. Kafka’s “exactly-once semantics,” AWS SQS FIFO’s “exactly-once processing,” and similar guarantees are all the second kind — at-least-once transport plus application-layer deduplication, engineered carefully enough that the seams don’t show.

---

Guarantee Model

Exactly-once processing provides: for any message or request carrying a stable, unique identifier, the protected side effect executes at most once, regardless of how many times the transport redelivers it. This guarantee holds only when all three of the following are true simultaneously:

1. Transport delivers at-least-once (never fewer than one attempt reaches the consumer eventually).
2. Every message carries an identifier that is stable across redeliveries — the same logical event always presents the same ID, never a freshly generated one per retry.
3. The consumer checks and records that identifier in a durable dedup store atomically, before or within the same transaction as the side effect — never as a separate, non-atomic step.

The guarantee breaks under these conditions:

1. Dedup store TTL shorter than the maximum redelivery delay. If the broker or client can redeliver a message 6 hours after the first attempt but the dedup key expired after 1 hour, the duplicate is treated as new and reprocessed.
2. Non-atomic check-then-act. Reading “have I seen this ID?” and then writing “mark as seen” as two separate operations leaves a race window where two concurrent duplicate deliveries both read “not seen” and both proceed.
3. Unstable identifiers. If a retry generates a new message ID instead of reusing the original (a common client-library bug), deduplication has nothing to key on and silently fails.

---

Choosing a Semantic for Your System

Not every code path needs the strongest guarantee. Choosing at-most-once, at-least-once, or exactly-once processing is a deliberate trade-off between loss risk, duplication risk, and implementation cost.

The rule of thumb: if a duplicate side effect is observable and costly — money moves, inventory changes, an irreversible external action fires — build exactly-once processing. If a duplicate is merely wasteful, at-least-once alone is sufficient and considerably cheaper to operate.

---

Core Algorithm: Delivery Semantics Compared

The diagram below traces the same message through all three semantics side by side — sender, network, and receiver — showing exactly where each model diverges on loss, retry, and duplication.

---

Implementation Variants

Variant 1 — Idempotent Consumer with Dedup Store Check

The consumer checks a durable store for the message’s unique identifier before executing any side effect, and records it atomically as part of the same operation. This is the general-purpose pattern for HTTP APIs, webhooks, and stream consumers alike — see idempotent consumer patterns for event streams for the full stream-specific treatment.

# Python: idempotent consumer using Redis as the dedup store
import redis

r = redis.Redis(host="localhost", port=6379, decode_responses=True)

def handle_message(message_id: str, payload: dict) -> None:
    # SET NX with a TTL wider than the maximum possible redelivery delay
    reserved = r.set(f"dedup:{message_id}", "PROCESSING", nx=True, ex=86400)
    if not reserved:
        status = r.get(f"dedup:{message_id}")
        if status == "DONE":
            return  # already processed — discard duplicate silently
        raise RetryableError("processing in flight, redeliver later")

    apply_side_effect(payload)
    r.set(f"dedup:{message_id}", "DONE", ex=86400)

The full atomic reservation pattern — including the race between concurrent duplicate deliveries — is covered in using Redis SETNX for distributed request deduplication.

Variant 2 — Durable Dedup Store on a Relational Backend

For workloads that need an audit trail alongside deduplication (payments, ledger postings), a unique constraint on the message identifier column gives the same atomicity guarantee without a separate cache tier:

-- PostgreSQL: unique constraint enforces exactly-once processing
CREATE TABLE processed_messages (
    message_id   TEXT PRIMARY KEY,
    processed_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- Attempt to claim the message; 0 rows inserted = already processed
INSERT INTO processed_messages (message_id) VALUES ($1)
ON CONFLICT (message_id) DO NOTHING;
-- Application checks rows_affected: 1 = proceed, 0 = duplicate, discard

Variant 3 — Transactional Outbox for Dual-Write Safety

When processing a message also requires publishing a downstream event, writing both the dedup record and the outgoing event in one database transaction eliminates the classic dual-write gap. The transactional outbox pattern commits the business state, the dedup marker, and the pending outbound event atomically; a relay process publishes from the outbox afterward, and the downstream consumer’s own dedup layer absorbs any relay-side duplicates.

Variant 4 — Kafka Transactional Read-Process-Write (EOS)

Apache Kafka’s exactly-once semantics wrap the read offset, the processing output, and the offset commit in a single atomic transaction, so a consumer crash mid-batch cannot produce a partial commit:

// Java: transactional producer performing read-process-write
Properties props = new Properties();
props.put("transactional.id", "order-processor-1");
KafkaProducer<String, String> producer = new KafkaProducer<>(props);
producer.initTransactions();

try {
    producer.beginTransaction();
    for (ConsumerRecord<String, String> record : records) {
        String result = process(record.value());
        producer.send(new ProducerRecord<>("orders-processed", result));
    }
    producer.sendOffsetsToTransaction(currentOffsets(records), consumerGroupId);
    producer.commitTransaction();
} catch (Exception e) {
    producer.abortTransaction();
}

Downstream consumers must set isolation.level=read_committed so they never observe records from an aborted transaction. This is the strongest exactly-once processing guarantee available without leaving the Kafka cluster, because the input offset, the output record, and the offset commit are atomic with respect to each other. See achieving exactly-once processing in Kafka consumers for the complete runbook, including the fallback pattern required when the consumer writes to a non-Kafka external system.

Summary Comparison

---

Edge Cases and Failure Scenarios

---

Operational Concerns

TTL Sizing

The dedup store TTL must exceed the maximum possible redelivery window, not the average one. For Kafka, this means accounting for consumer group rebalance stalls and retention policy; for HTTP webhook retries, align to the sender’s documented maximum retry duration (commonly 72 hours for payment gateways — see webhook delivery guarantees). A TTL that expires before the last possible retry silently converts exactly-once processing back into at-least-once with duplicates.

Index and Storage Layout

For relational dedup tables, index the message identifier as the primary key (not a secondary index) so the uniqueness check and the write happen in one atomic operation. Add a partial index on processed_at to support efficient purge jobs:

CREATE INDEX idx_processed_recent ON processed_messages (processed_at)
WHERE processed_at > now() - INTERVAL '7 days';

Memory Budgeting

A Redis dedup key with a UUID identifier and a short status value typically costs 80–150 bytes. At a sustained rate of 5,000 messages/second with a 24-hour TTL, expect roughly 432 million keys resident at peak — budget approximately 43–65 GB of Redis memory, or shard the keyspace across multiple nodes if that exceeds a single instance’s capacity.

SRE Alert Thresholds

• duplicate_processing_total — alert on any non-zero rate; indicates the dedup layer failed to catch a redelivery
• dedup_store_miss_rate — alert if it exceeds 0.1% over a 5-minute window; a rising rate signals TTL misconfiguration or store outages
• kafka_transaction_abort_rate — alert if greater than 1% of transactions over 10 minutes; sustained aborts indicate coordinator instability
• consumer_group_rebalance_count — alert if more than 3 rebalances in 5 minutes; frequent rebalancing widens the at-least-once redelivery window
• stuck_processing_records_total — alert on any record stuck in PROCESSING beyond TTL + processing_p99, indicating a crashed consumer left a dangling claim

---

Related

• Idempotency Fundamentals & API Guarantees — parent section covering the full engineering contract for idempotent APIs, key lifecycle, and failure boundary mapping.
• Achieving Exactly-Once Processing in Kafka Consumers — the complete runbook for transactional read-process-write with initTransactions, offset commits, and duplicate injection testing.
• Idempotent Consumer Patterns for Event Streams — general-purpose stream consumer deduplication independent of any single broker.
• Using Redis SETNX for Distributed Request Deduplication — the atomic reservation pattern behind the dedup store variant above.
• Implementing the Transactional Outbox Pattern — solving the dual-write problem when a processed message must also emit a downstream event.
• Message Queue Deduplication Patterns — broker-agnostic deduplication strategies spanning RabbitMQ and Kafka.
• Consensus Algorithms for Deduplication — Raft- and ZooKeeper-backed logs that extend exactly-once processing guarantees across multiple coordinating nodes.