SQS FIFO Deduplication ID vs Application-Level Dedup

SQS FIFO advertises exactly-once delivery, and it delivers on that within a narrow, precisely defined scope: a five-minute window, on the producer side. Everything outside that window — a producer retrying after six minutes, a consumer whose visibility timeout expires — is still at-least-once and still needs a deduplication table. This guide sits under message queue deduplication patterns.

Prerequisites. An SQS FIFO queue, and the framing from exactly-once vs effectively-once semantics.

Step 1 — What the five-minute interval covers

Three duplicate scenarios and which layer catches each Three horizontal scenarios. In the first, a producer resends within five minutes and SQS silently discards the duplicate. In the second, a producer resends after six minutes and SQS accepts it as a new message because the deduplication interval has elapsed. In the third, the consumer's visibility timeout expires before deletion and SQS redelivers the same message, which the deduplication id does not prevent. Only the first is caught by SQS; the other two require a consumer-side deduplication table. 1 · Producer resends at t+90 s same MessageDeduplicationId, inside the 5-minute interval → SQS discards it silently. Caught by the broker. ✓ SQS FIFO 2 · Producer resends at t+6 min same MessageDeduplicationId, interval has elapsed and is NOT configurable → SQS accepts it as a new message. Broker cannot help. your table 3 · Visibility timeout expires before DeleteMessage consumer was slow or crashed; the deduplication id is irrelevant here → SQS redelivers the same message. Broker cannot help. your table Two of the three duplicate sources are invisible to the broker. The group id decides your parallelism Two arrangements. A single constant message group id forces strict ordering across the whole queue, so one message is processed at a time no matter how many consumers run. Per-account group ids let messages for different accounts proceed in parallel while preserving order within each account. group = "all" one at a time — consumers idle group = account_id parallel across accounts, ordered within each

The deduplication ID is a producer-side mechanism. It answers “did this producer already send this message?” and nothing else. Consumer-side redelivery — by far the more common duplicate source in practice — is unaffected by it.

Step 2 — Set an explicit deduplication ID

SQS offers content-based deduplication, which hashes the message body. Do not use it for anything where two legitimately identical messages can occur.

import boto3

sqs = boto3.client("sqs")

# GOOD: explicit id derived from the business event, stable across producer retries.
sqs.send_message(
    QueueUrl=QUEUE_URL,
    MessageBody=json.dumps(event),
    MessageDeduplicationId=event["id"],        # ULID from the producing service
    MessageGroupId=event["account_id"],        # ordering scope — see step 3
)
# RISKY: content-based deduplication collapses two genuine identical events.
# "Customer bought the same £3.50 coffee twice in one minute" becomes one message.
sqs.create_queue(
    QueueName="payments.fifo",
    Attributes={"FifoQueue": "true", "ContentBasedDeduplication": "true"},
)

An explicit ID must be stable across the producer’s own retries and unique across distinct events. A ULID or UUIDv7 minted once per business intent satisfies both, and its 26 or 36 characters fit comfortably inside the 128-character limit.

Step 3 — Choose the message group ID carefully

This is where FIFO queues most often disappoint on throughput. Messages sharing a MessageGroupId are delivered in strict order and processed one at a time; a single group ID serialises the entire queue.

Group ID choice Ordering guarantee Parallelism When it is right
Constant ("all") Total order across everything 1 consumer, effectively Almost never
Per tenant Ordered within a tenant One per tenant Multi-tenant, cross-entity ordering matters
Per account or entity Ordered per account One per account The usual right answer
Per message (unique) None Unlimited You did not need FIFO 
# The narrowest ordering unit the business actually requires.
MessageGroupId=event["account_id"]        # ledger entries per account stay ordered

FIFO queues cap at 300 transactions per second per API action, or 3,000 with batching of 10. High-throughput mode raises this substantially but requires many distinct group IDs to use it — another reason to pick a narrow ordering unit.

Step 4 — Add the consumer-side table

from psycopg.errors import UniqueViolation

def handle_batch(messages, conn) -> None:
    for msg in messages:
        event = json.loads(msg["Body"])
        try:
            with conn.transaction():
                conn.execute(
                    """INSERT INTO processed_events (consumer_group, event_id, source)
                       VALUES (%s, %s, 'sqs')""",
                    (GROUP, event["id"]),
                )
                apply_effect(event, conn)      # same transaction as the marker
            outcome = "processed"
        except UniqueViolation:
            outcome = "duplicate"              # a redelivery — safe to delete

        # Delete AFTER the transaction commits, never before.
        sqs.delete_message(QueueUrl=QUEUE_URL, ReceiptHandle=msg["ReceiptHandle"])
        metrics.inc("sqs_messages_handled_total", outcome=outcome)

Delete after the commit, not before. Deleting first turns a crash into lost data; deleting after turns it into a redelivery that the marker table absorbs — which is the trade you want.

Size the visibility timeout above the handler’s p99 duration, with headroom:

# Extend the visibility while a long handler is still working, rather than
# setting a single huge timeout that delays recovery from a real crash.
def heartbeat(receipt_handle, every_s=20, extend_to_s=60):
    while not done.is_set():
        sqs.change_message_visibility(
            QueueUrl=QUEUE_URL, ReceiptHandle=receipt_handle,
            VisibilityTimeout=extend_to_s)
        done.wait(every_s)

Step 5 — Verify both layers

Confirm the broker deduplicates inside the window.

ID=$(uuidgen)
for i in 1 2; do
  aws sqs send-message --queue-url "$QUEUE_URL" --message-body '{"id":"'"$ID"'"}' \
    --message-deduplication-id "$ID" --message-group-id acct_7 \
    --query MessageId --output text
done
# expect: the SAME MessageId twice — the second send was discarded

Confirm it does not deduplicate outside the window.

aws sqs send-message --queue-url "$QUEUE_URL" --message-body '{"id":"'"$ID"'"}' \
  --message-deduplication-id "$ID" --message-group-id acct_7 --query MessageId --output text
sleep 310                                   # just past 5 minutes
aws sqs send-message --queue-url "$QUEUE_URL" --message-body '{"id":"'"$ID"'"}' \
  --message-deduplication-id "$ID" --message-group-id acct_7 --query MessageId --output text
# expect: two DIFFERENT MessageIds — only your table stops the double effect
psql -c "SELECT count(*) FROM ledger_entries WHERE event_id = '$ID'"   # expect 1

Confirm redelivery is absorbed.

# Force a visibility-timeout redelivery by killing the consumer mid-handler.
kill -STOP "$(pgrep -f sqs-consumer)"       # freeze past the visibility timeout
sleep 45
kill -CONT "$(pgrep -f sqs-consumer)"
psql -c "SELECT event_id, count(*) FROM ledger_entries GROUP BY 1 HAVING count(*) > 1"
# expect: zero rows

Failure scenarios and debugging

Failure Scenario Remediation Steps Observability Hooks
Content-based deduplication enabled, and two genuinely identical events collapse into one Disable it and set an explicit MessageDeduplicationId derived from the producing event’s id. Backfill the missing event manually. Producer send count versus NumberOfMessagesSent; a persistent shortfall means silent collapsing
A single constant MessageGroupId, so the queue processes one message at a time and lag grows without bound Change the group ID to the narrowest ordering unit — usually an account or entity id — and redeploy consumers. ApproximateAgeOfOldestMessage climbing while consumers sit idle; NumberOfMessagesReceived flat at ~1 per poll
Visibility timeout shorter than the handler’s p99, so every slow message is redelivered and processed twice Raise the timeout above p99 with headroom, or heartbeat with ChangeMessageVisibility. The marker table absorbs the effect, but the wasted work is real. sqs_messages_handled_total{outcome="duplicate"} rate above 1%; ApproximateReceiveCount above 1
DeleteMessage called before the database transaction commits, so a crash loses the event entirely Delete after the commit returns. A redelivery is recoverable; a deleted-but-unprocessed message is not. events_lost_total from a producer-side sequence-gap detector; consumer lag that never recovers
Producer retries with a 10-minute backoff, so genuine duplicates land outside the 5-minute interval and rely entirely on the consumer table This is expected, not a bug — but verify the marker table’s retention exceeds the producer’s total retry span. processed_events_oldest_age_seconds compared against the producer’s documented give-up time
Delete after commit, never before A message is received, the marker and effect are written in one transaction, the transaction commits, and only then is DeleteMessage called. A crash before the delete causes a redelivery that the marker absorbs. Deleting first would turn the same crash into permanent loss of the event. receive marker + effect one transaction, COMMIT DeleteMessage crash here → redelivery, absorbed Reversing the order converts a recoverable redelivery into unrecoverable loss.

SRE / observability checklist

  1. sqs_messages_handled_total{outcome} — Counter split processed and duplicate. A duplicate rate above 1% points at the visibility timeout, not at the producer.
  2. ApproximateAgeOfOldestMessage — CloudWatch gauge. Growth with idle consumers is the message-group-ID serialisation signature.
  3. ApproximateReceiveCount — Per-message attribute; log it. A p99 above 1 means messages are routinely being redelivered.
  4. ApproximateNumberOfMessagesNotVisible — Gauge of in-flight messages. A value pinned near the consumer count means handlers are slower than the visibility timeout.
  5. sqs_delete_after_commit_violations_total — Assertion counter from a test hook confirming deletion always follows the commit. Should be structurally zero.
  6. event_id, MessageId and ApproximateReceiveCount log fields — On every handled message, so a duplicate can be attributed to the producer or to a redelivery.