Message Queue Deduplication Patterns

Part of: Backend Implementation & Storage Patterns

Every message broker that offers at-least-once delivery — Kafka, RabbitMQ, Amazon SQS, Google Pub/Sub — will, under normal operation, deliver the same message more than once. A consumer crashes after processing but before committing an offset; a visibility timeout expires while a handler is still running; a network blip causes an ack to be lost in flight. The broker cannot distinguish “the consumer never got this” from “the consumer got this and I never heard back,” so it does the only safe thing: it redelivers. This page covers why that forces dedup work onto the consumer, what each broker does and does not provide natively, where the dedup key comes from, and three production-grade implementation variants with code and a failure-scenario table.

If you have not yet read Idempotent Consumer Patterns for Event Streams and Exactly-Once vs At-Least-Once Delivery, start there — this page assumes you understand why “exactly-once delivery” is not achievable at the transport layer and covers the storage-layer mechanics of the consumer-side dedup store instead. For a hands-on runbook covering RabbitMQ and Kafka specifically, see Deduplicating RabbitMQ and Kafka Deliveries.

---

Problem Framing: Why Broker Guarantees Force Consumer-Side Dedup

At-least-once delivery is a deliberate trade-off, not a bug. A broker that tried to guarantee exactly-once delivery at the transport layer would need a distributed transaction spanning the network link, the consumer’s ack, and the broker’s own commit log — and no such transaction can survive an arbitrary partition without blocking. So every mainstream broker picks at-least-once and pushes deduplication to the application:

• Kafka redelivers when a consumer’s offset commit has not caught up with a partition it has already read from — after a crash, a rebalance, or a manual seek() to an earlier offset.
• RabbitMQ redelivers any message whose consumer disconnects, nacks, or fails to ack before the channel closes; requeued messages go back to the front or back of the queue depending on configuration and can be redelivered indefinitely if consumers keep failing.
• Amazon SQS redelivers a message if it is not deleted before its visibility timeout expires, or up to maxReceiveCount times before it moves to a dead-letter queue.
• Google Pub/Sub redelivers on ack-deadline expiry, identically in spirit to SQS’s visibility timeout.

None of these mechanisms know whether your handler actually finished the side effect (charged a card, sent an email, updated a ledger) before the ack was lost. The only entity that can answer that question is a dedup store that records “this message id (or business key) has already been handled” — independent of the broker’s own delivery bookkeeping.

---

Guarantee Model

A correctly implemented message-queue dedup layer provides: at-least-once delivery from the broker, exactly-once processing of side effects at the consumer, subject to three conditions:

1. The dedup key is claimed before the handler produces any externally visible side effect, using an atomic check-and-set operation (SETNX, a unique-constraint INSERT, or a conditional write).
2. The dedup store’s window is at least as long as the broker’s maximum possible redelivery window.
3. The dedup store itself does not lose the claimed key before the redelivery window closes.

Where this breaks down:

• Ack-then-crash ordering. If the handler commits its side effect and then crashes before writing the dedup key, the next redelivery reprocesses the message — the dedup store and the side effect must be transactionally coupled, or the dedup write must happen strictly before the side effect (accepting a rare false-negative dedup instead of a duplicate side effect).
• Dedup store outage. If the dedup store is unreachable, you must choose between blocking consumption (safe, but stalls the queue) and processing without a dedup check (fast, but risks duplicates) — there is no third option.
• Redelivery window underestimation. A dedup TTL shorter than the broker’s actual redelivery window silently reopens the duplicate-processing gap once the key expires.

---

Core Protocol: Filtering Redeliveries Through a Dedup Store

The diagram below shows the canonical consumer-side flow: every delivery — first attempt or redelivery — passes through the dedup store before the handler runs.

The critical ordering constraint is visible in the diagram: the dedup key check happens before Run Handler, and the ack happens after Persist Key. Acking before the key is durably persisted reopens the exact gap this pattern exists to close — if the process crashes between ack and persist, the message is gone from the broker but never recorded as handled, and there is no redelivery left to catch it.

---

Where the Dedup Key Comes From

Two sources are used in production, and picking the wrong one is the most common implementation mistake:

Message id (broker-assigned or client-supplied). Kafka delivers (topic, partition, offset) as a durable, unique triple; SQS assigns a MessageId (and, for FIFO queues, accepts a client-supplied MessageDeduplicationId); RabbitMQ has no built-in message id, so producers must set the AMQP message_id property explicitly. Message-id dedup is cheap and exact — the same physical delivery never yields two different keys — but it does not catch duplicate business events published as two separate messages (e.g., a producer retry that generates a new message id for the same logical payment).

Derived business key. A key computed from domain fields — order_id, idempotency_key header, or a hash of the normalized payload — catches duplicates regardless of how many times or under what message id the event was (re)published. This is the same key-derivation approach used in Redis SETNX-based request deduplication, applied at the consumer instead of the API gateway. Business-key dedup is mandatory whenever an upstream producer might retry a publish with a new message id, which is common in any system built on exponential backoff with jitter.

The safe default for financial and other side-effecting consumers is to dedup on both: the message id catches broker-level redelivery cheaply, and the business key catches producer-level republishing. Store both in the same dedup record so a single lookup can short-circuit either failure mode.

---

Broker-Native Dedup vs Application Dedup Store

Brokers vary widely in what they offer natively, and none of them cover the full problem:

The practical conclusion: broker-native dedup, where it exists, is a narrow optimization for producer-retry noise. It is never a substitute for an application-level dedup store, because none of these mechanisms survive a consumer crash-and-restart or a deliberate replay from an earlier offset — both of which are routine operational events, not exotic failures.

---

Implementation Variants

Variant 1 — Broker-Native Dedup (Kafka Idempotent Producer / SQS FIFO)

Use broker-native dedup as a first line of defense against producer-side retry noise, never as the only layer.

# Kafka producer config: idempotent producer
enable.idempotence: true
acks: all
max.in.flight.requests.per.connection: 5
retries: 2147483647

# SQS FIFO: client-supplied dedup id derived from the business event
import boto3, hashlib, json

sqs = boto3.client("sqs")

def publish_order_event(order_id: str, payload: dict):
    dedup_id = hashlib.sha256(f"order:{order_id}".encode()).hexdigest()
    sqs.send_message(
        QueueUrl="https://sqs.us-east-1.amazonaws.com/000000000000/orders.fifo",
        MessageBody=json.dumps(payload),
        MessageGroupId="orders",
        MessageDeduplicationId=dedup_id,  # SQS dedups within a 5-minute window
    )

Variant 2 — Redis SETNX Dedup Store

The fast path for high-throughput consumers. The consumer claims the dedup key atomically before running the handler, mirroring the Redis cache-based deduplication pattern used at the API layer.

import redis

r = redis.Redis(host="localhost", decode_responses=True)
DEDUP_TTL_SECONDS = 86_400  # exceeds the broker's max redelivery window

def handle_delivery(message_id: str, business_key: str, payload: dict) -> bool:
    dedup_key = f"mq:dedup:{business_key or message_id}"
    claimed = r.set(dedup_key, "PROCESSING", nx=True, ex=DEDUP_TTL_SECONDS)
    if not claimed:
        return True  # duplicate — ack and skip

    try:
        process(payload)
        r.set(dedup_key, "DONE", ex=DEDUP_TTL_SECONDS)
        return True  # ack
    except Exception:
        r.delete(dedup_key)  # release so a legitimate redelivery can retry
        return False  # nack / requeue

Variant 3 — Database Unique-Constraint Inbox

The durable, transactional option: the dedup record and the business-side write commit atomically in the same database transaction, giving true exactly-once processing without a second network round-trip to Redis.

-- Inbox table: one row per handled message, enforced by the primary key
CREATE TABLE consumer_inbox (
    dedup_key    TEXT PRIMARY KEY,
    consumed_at  TIMESTAMPTZ NOT NULL DEFAULT now()
);

BEGIN;
INSERT INTO consumer_inbox (dedup_key) VALUES ($1);   -- fails with 23505 if already seen
UPDATE ledger SET balance = balance + $2 WHERE account_id = $3;
COMMIT;
-- On a 23505 unique_violation: ROLLBACK and ack without reprocessing.

// Java: consume, insert into inbox, and update state in one transaction
@Transactional
public boolean handleDelivery(String dedupKey, Payload payload) {
    try {
        jdbcTemplate.update(
            "INSERT INTO consumer_inbox (dedup_key) VALUES (?)", dedupKey);
    } catch (DuplicateKeyException e) {
        return true; // already handled — ack, skip business logic
    }
    applyBusinessLogic(payload);
    return true; // commit covers both the inbox row and the business write
}

This variant is the natural pairing with PostgreSQL unique constraints vs. application-level checks and with the transactional outbox pattern when the consumer must also publish a downstream event as part of the same unit of work. Teams on DynamoDB should instead use conditional writes for atomic idempotency, which provides the same claim-before-process guarantee without a relational database.

Variant Comparison

---

Edge Cases and Failure Scenarios

---

Operational Concerns

Dedup Window vs. Redelivery Window

The single most important operational number on this page is the relationship between two durations:

dedup_window_seconds  >=  broker_max_redelivery_window_seconds × 1.3

For SQS, the maximum redelivery window is maxReceiveCount × visibility_timeout — with maxReceiveCount = 5 and a visibility_timeout of 30 seconds, that is 150 seconds before the message dead-letters, so a dedup TTL of 300 seconds is comfortably safe. For RabbitMQ, there is no hard ceiling on redelivery count by default; size the dedup TTL to the longest realistic consumer outage you plan to tolerate — commonly 86400 seconds (24 hours) for services with an SLA around consumer recovery time. For Kafka, the relevant bound is not topic retention but the longest delay before a stalled consumer group is detected and restarted — typically governed by session.timeout.ms (measured in seconds) escalated to an on-call page within 900 seconds, so a dedup TTL of 3600 seconds to 86400 seconds covers realistic operational recovery times.

Getting this backwards — a dedup window shorter than the redelivery window — is the single most common production incident in this pattern: everything works in testing (where redeliveries happen within seconds) and then silently reopens the duplicate gap once a real outage stretches past the TTL.

Storage Budgeting

Each dedup record costs roughly 100-300 bytes (Redis) or one row (~150 bytes with index overhead in Postgres). At 5,000 messages/second sustained throughput with an 86400 second window, the steady-state key count is 5,000 × 86,400 ≈ 432 million keys. At 200 bytes per key that is roughly 86 GB — large enough to require a dedicated Redis cluster with maxmemory-policy noeviction (never evict live dedup keys) or a partitioned Postgres table with a background purge job:

-- Purge job: run every 300 seconds, keep the table bounded
DELETE FROM consumer_inbox WHERE consumed_at < now() - INTERVAL '25 hours';

Reduce cost by storing only the dedup key and a terminal-state flag (no payload), and by partitioning the inbox table by day so old partitions can be dropped in O(1) instead of scanned and deleted row-by-row.

SRE Alert Thresholds

• mq_dedup_hit_rate — expected baseline depends on redelivery frequency; alert if it exceeds 5× baseline for 5 minutes (signals a stuck consumer or retry storm upstream).
• dedup_store_error_rate — alert if > 1% over 60 seconds.
• poison_message_dlq_total — alert on any non-zero rate; poison messages never self-resolve.
• consumer_lag (Kafka) / ApproximateNumberOfMessagesVisible (SQS) — rising lag alongside a flat dedup-hit rate indicates the handler itself is slow, not duplicating.
• dedup_ttl_miss_total — any non-zero value means the dedup window is undersized relative to actual redelivery behavior; treat as a page, not a warning.

---

Related

• Deduplicating RabbitMQ and Kafka Deliveries — copy-pasteable runbook for both brokers in Python, Node.js, Java, and Go, with forced-redelivery verification steps.
• Backend Implementation & Storage Patterns — parent section covering the full storage-layer approach to idempotency and deduplication.
• Using Redis SETNX for Distributed Request Deduplication — the same atomic-claim pattern applied at the API gateway layer instead of the consumer.
• DynamoDB Conditional Writes for Deduplication — the equivalent claim-before-process pattern for teams standardized on DynamoDB.
• Idempotent Consumer Patterns for Event Streams — the fundamentals-level treatment of why consumers, not brokers, own the exactly-once contract.
• Exactly-Once vs At-Least-Once Delivery — the delivery-guarantee taxonomy this page’s guarantee model builds on.