Deduplicating SQS and SNS Message Deliveries

Part of: Idempotent Consumer Patterns for Event Streams

SQS and SNS are two of the most common message transports in AWS-native architectures, and each has a distinct deduplication story. FIFO queues offer built-in deduplication, but only within a 5-minute rolling window. Standard queues offer none at all — they trade dedup and ordering for higher throughput and are documented as strictly at-least-once. SNS fan-out multiplies the problem, since each subscribed queue receives its own independent copy of every message. This is a focused runbook: it assumes you already understand the broader idempotent consumer pattern — deriving a stable dedup key and checking it inside an inbox before executing business logic — and the idempotency guarantee model it builds on. All code below is copy-pasteable and independently verifiable.

The core mechanism: FIFO’s 5-minute window vs. an unbounded app-level store

SQS FIFO deduplication is not permanent — it is a rolling window scoped to 5 minutes per MessageGroupId. A retry that arrives after the window closes is treated as a brand-new message, even with an identical MessageDeduplicationId. This is the detail that catches teams off guard: FIFO dedup absorbs fast producer-side retries, but it does nothing for a consumer-side failure that causes reprocessing an hour later, and it does nothing at all for standard queues. The diagram below shows both halves: a FIFO queue deduplicating two sends inside the window, then admitting a third send after the window expires, followed by an application-level DynamoDB check that catches what the queue could not.

SQS FIFO dedup window and DynamoDB application-level fallback Producer sends a message with a fixed deduplication id at t=0s and SQS FIFO enqueues it as msg_id=abc. A resend at t=120s, still inside the 5-minute window, is dropped and SQS returns the same msg_id=abc. A resend at t=400s, after the window has expired, is treated as new and enqueued as msg_id=xyz. SQS delivers msg_id=xyz to the consumer, which attempts a conditional PutItem in DynamoDB keyed on the business id. DynamoDB rejects it with ConditionalCheckFailedException because the business event was already recorded, so the consumer skips business logic and acknowledges. Producer SQS FIFO Queue Consumer DynamoDB send(dedup_id=X) t=0s enqueued, msg_id=abc — 120s later, within 5-min window — send(dedup_id=X) t=120s dedup: same msg_id=abc — 400s later, window expired — send(dedup_id=X) t=400s enqueued NEW, msg_id=xyz deliver msg_id=xyz PutItem If Not Exists(order_id) ConditionalCheckFailedException duplicate — skip business logic, ack anyway

Problem statement and prerequisites

What you are implementing: deduplication that survives beyond SQS FIFO’s 5-minute window, covers SQS standard queues (which offer no dedup at all), and correctly scopes state per subscriber when SNS fans a message out to multiple queues.

Prerequisites:

  • You understand the idempotent consumer pattern — specifically how to derive a stable dedup key and check it inside an inbox before executing business logic.
  • You are familiar with the broader idempotency guarantee model and why at-least-once transports push the burden of exactly-once effects onto the consumer.
  • You have an AWS account with permissions to create SQS queues, SNS topics, and a DynamoDB table (or an equivalent local test environment via localstack).

Step-by-step implementation

Step 1 — Create a FIFO queue with deduplication enabled

aws sqs create-queue \
  --queue-name orders.fifo \
  --attributes '{
    "FifoQueue": "true",
    "ContentBasedDeduplication": "true",
    "VisibilityTimeout": "30"
  }'

ContentBasedDeduplication hashes the message body automatically; set it to false and pass an explicit MessageDeduplicationId per send if your payload contains volatile fields (timestamps, trace ids) that would otherwise defeat content hashing.

import boto3

sqs = boto3.client("sqs", region_name="us-east-1")
queue_url = sqs.get_queue_url(QueueName="orders.fifo")["QueueUrl"]

resp = sqs.send_message(
    QueueUrl=queue_url,
    MessageBody='{"order_id": "ord_123", "action": "capture"}',
    MessageGroupId="ord_123",              # required for FIFO; scopes ordering + dedup
    MessageDeduplicationId="ord_123-capture-v1",
)
print(resp["MessageId"])

Step 2 — Verify the 5-minute window boundary

Send the same MessageDeduplicationId twice within 5 minutes and confirm SQS returns the same MessageId both times; then wait past the window and confirm it does not.

# First send
aws sqs send-message --queue-url "$QUEUE_URL" \
  --message-body '{"order_id":"ord_123"}' \
  --message-group-id ord_123 \
  --message-deduplication-id ord_123-capture-v1
# Immediate resend (within 5 min) — MessageId matches the first response
aws sqs send-message --queue-url "$QUEUE_URL" \
  --message-body '{"order_id":"ord_123"}' \
  --message-group-id ord_123 \
  --message-deduplication-id ord_123-capture-v1

Step 3 — Implement application-level dedup for standard queues

Standard queues provide no dedup at all, so every consumer must check a durable store keyed on a business identifier before executing logic — the same inbox pattern used for any event stream, backed here by a DynamoDB conditional write.

import boto3
import json
from botocore.exceptions import ClientError

dynamodb = boto3.resource("dynamodb")
inbox = dynamodb.Table("sqs_inbox")

def handle_message(sqs_message: dict) -> None:
    body = json.loads(sqs_message["Body"])
    dedup_key = f"orders#{body['order_id']}#{body['action']}"
    try:
        inbox.put_item(
            Item={"dedup_key": dedup_key, "received_at": int(time.time())},
            ConditionExpression="attribute_not_exists(dedup_key)",
        )
    except ClientError as e:
        if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
            return  # duplicate — ack without re-executing business logic
        raise
    apply_business_logic(body)
// Node.js (AWS SDK v3): application-level dedup for a standard SQS queue
const { DynamoDBClient } = require("@aws-sdk/client-dynamodb");
const { DynamoDBDocumentClient, PutCommand } = require("@aws-sdk/lib-dynamodb");

const ddb = DynamoDBDocumentClient.from(new DynamoDBClient({ region: "us-east-1" }));

async function handleMessage(sqsMessage) {
  const body = JSON.parse(sqsMessage.Body);
  const dedupKey = `orders#${body.order_id}#${body.action}`;

  try {
    await ddb.send(new PutCommand({
      TableName: "sqs_inbox",
      Item: { dedup_key: dedupKey, received_at: Date.now() },
      ConditionExpression: "attribute_not_exists(dedup_key)",
    }));
  } catch (err) {
    if (err.name === "ConditionalCheckFailedException") {
      return; // duplicate — ack without re-executing business logic
    }
    throw err;
  }
  await applyBusinessLogic(body);
}

Step 4 — Scope dedup state per subscriber for SNS fan-out

SNS delivers an independent copy of every published message to each subscribed queue. If two queues subscribe to the same topic, each must maintain its own sqs_inbox (or a shared table partitioned by consumer group), because deduplicating globally would silently drop legitimate deliveries intended for the other subscriber.

# Partition the dedup key by subscriber so fan-out queues never collide
def handle_fanout_message(sqs_message: dict, consumer_group: str) -> None:
    body = json.loads(json.loads(sqs_message["Body"])["Message"])  # unwrap SNS envelope
    dedup_key = f"{consumer_group}#orders#{body['order_id']}#{body['action']}"
    try:
        inbox.put_item(
            Item={"dedup_key": dedup_key, "received_at": int(time.time())},
            ConditionExpression="attribute_not_exists(dedup_key)",
        )
    except ClientError as e:
        if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
            return
        raise
    apply_business_logic(body)

Step 5 — Align visibility timeout with processing time

A message becomes visible to other consumers again if it is not deleted before the queue’s VisibilityTimeout expires, producing a redelivery that looks identical to a genuine duplicate. Set the timeout to comfortably exceed your P99 processing latency, and extend it explicitly for long-running handlers rather than relying on a single large static value.

# Verify current visibility timeout
aws sqs get-queue-attributes --queue-url "$QUEUE_URL" \
  --attribute-names VisibilityTimeout
# Extend visibility mid-processing for a handler expected to run long
sqs.change_message_visibility(
    QueueUrl=queue_url,
    ReceiptHandle=receipt_handle,
    VisibilityTimeout=60,
)

Verification and testing

Confirm FIFO dedup within the window:

aws sqs send-message --queue-url "$QUEUE_URL" --message-group-id g1 \
  --message-deduplication-id d1 --message-body '{"x":1}'
aws sqs send-message --queue-url "$QUEUE_URL" --message-group-id g1 \
  --message-deduplication-id d1 --message-body '{"x":1}'
# Expected: both calls return the same MessageId

Confirm the DynamoDB fallback rejects a duplicate business key:

aws dynamodb put-item --table-name sqs_inbox \
  --item '{"dedup_key": {"S": "orders#ord_123#capture"}}' \
  --condition-expression "attribute_not_exists(dedup_key)"
# Second identical call must fail with ConditionalCheckFailedException
aws dynamodb put-item --table-name sqs_inbox \
  --item '{"dedup_key": {"S": "orders#ord_123#capture"}}' \
  --condition-expression "attribute_not_exists(dedup_key)"

Check for messages stuck past visibility timeout:

aws sqs get-queue-attributes --queue-url "$QUEUE_URL" \
  --attribute-names ApproximateNumberOfMessagesNotVisible ApproximateNumberOfMessages

Failure scenarios and debugging

Failure Scenario Remediation Steps Observability Hooks
Retry arrives after the 5-minute FIFO window and is treated as a new message Rely on an application-level DynamoDB or Redis dedup store keyed on a business id for anything that must survive beyond 5 minutes; never treat FIFO dedup as a permanent guarantee late_duplicate_detected_total counter on the app-level check; CloudWatch NumberOfMessagesSent spike correlated with retry logic
Processing exceeds VisibilityTimeout, so SQS redelivers the same message to a second worker while the first is still running Extend visibility explicitly via ChangeMessageVisibility for long-running handlers; set the base timeout to P99_processing_latency × 1.5 at minimum CloudWatch ApproximateNumberOfMessagesNotVisible; log field visibility_extended=true; alert if extension calls fail
SNS fan-out dedup state accidentally shared globally, causing one subscriber’s processing to suppress delivery to another Partition the dedup key by consumer group / subscriber id, never by message id alone inbox_duplicate_hits_total labeled by consumer_group; alert if one subscriber’s hit rate is unexpectedly zero
DynamoDB conditional write throttled (ProvisionedThroughputExceededException) during a delivery burst Do not fall back to unchecked processing on throttling — let the message become visible again and retry the conditional write with backoff; provision on-demand capacity or increase write capacity units dynamo_throttle_total CloudWatch metric; inbox_write_retry_total application counter
Content-based dedup misses because the payload includes a volatile field (timestamp, trace id) that changes on every retry Switch to explicit MessageDeduplicationId derived from stable business fields instead of ContentBasedDeduplication; strip volatile fields before hashing if computing the id application-side dedup_key_scheme label on send calls; alert if content-based dedup hit rate for an event type is unexpectedly zero

SRE / observability checklist

  1. inbox_duplicate_hits_total — Counter, labeled by consumer_group and dedup_key_scheme (FIFO-native vs. DynamoDB fallback). Alert if it exceeds 5× baseline over 5 minutes.
  2. ApproximateNumberOfMessagesNotVisible — CloudWatch SQS metric. A sustained rise indicates messages are stuck mid-processing and approaching redelivery.
  3. dynamo_conditional_check_failed_total — Counter around every PutItem/UpdateItem dedup check; a non-zero baseline is expected (it’s the dedup working), but a sudden spike signals a redelivery storm.
  4. dynamo_throttle_total — CloudWatch ThrottledRequests on the inbox table; alert on any non-zero rate during peak traffic.
  5. DLQ depth (ApproximateNumberOfMessagesVisible on the dead-letter queue) — alert if > 0, since every DLQ arrival represents an event that exhausted maxReceiveCount and needs manual or automated remediation.
  6. Structured log fields on every dedup decision — message_id, dedup_key, consumer_group, outcome (new | duplicate | error) — indexed for incident triage when reconciling a suspected double-processing event.