DynamoDB vs Redis for Idempotency at Scale

Both stores offer the one primitive deduplication needs — an atomic conditional write — so the choice comes down to four other properties: what happens to an acknowledged key when infrastructure fails, how expiry actually behaves, what it costs at your volume, and how much latency you are willing to add to a payment path. This guide sits under DynamoDB conditional writes for deduplication.

Prerequisites. A working atomic registration in at least one store, and a documented answer to what your endpoint does when the store is unreachable.

Step 1 — Decide whether a lost key is acceptable

This question decides the outcome more often than cost or latency does.

Cost follows traffic, or it does not A daily traffic curve peaks in business hours and falls overnight. DynamoDB on-demand cost tracks that curve exactly. A Redis cluster sized for the peak is drawn as a flat line at the peak level, paid for through the overnight trough as well. For spiky traffic the area between them favours DynamoDB; for flat traffic the two converge. DynamoDB on-demand — follows traffic Redis sized for peak — paid 24/7 Spiky traffic favours on-demand; steady high throughput favours provisioned capacity. What an acknowledged write means in each store Two panels. On the left, a DynamoDB write fans out to three availability zones and the acknowledgement is returned only after a quorum of them has durably stored it, so a zone failure cannot lose the key. On the right, a Redis write is acknowledged by the primary immediately, and replication to the replica happens afterwards, so a primary failure before replication loses the key. DynamoDB — synchronous, 3 AZs client AZ-a stored AZ-b stored AZ-c stored ACK arrives only after durable quorum an acknowledged key survives a zone loss cost: 8–15 ms p99 in-region Redis — asynchronous replication client primary ACK later… replica ACK precedes replication a failover loses 1–50 ms of keys benefit: 0.3–1 ms p99

If a lost key means a duplicate charge, DynamoDB’s durability is worth its latency. If it means a duplicate email or a re-sent push notification, Redis’s speed is worth its window. Decide this before comparing anything else, because it is the only property neither store can be configured into.

Step 2 — The atomic primitive, side by side

# DynamoDB: a conditional PutItem. Rejected with ConditionalCheckFailedException.
try:
    table.put_item(
        Item={
            "pk": scoped_key,
            "fingerprint": fingerprint,
            "state": "pending",
            "expires_at": int(time.time()) + 86_400,   # TTL attribute, epoch seconds
        },
        ConditionExpression="attribute_not_exists(pk) OR expires_at < :now",
        ExpressionAttributeValues={":now": int(time.time())},
        ReturnValuesOnConditionCheckFailure="ALL_OLD",  # get the existing item free
    )
    outcome = "won"
except ClientError as e:
    if e.response["Error"]["Code"] != "ConditionalCheckFailedException":
        raise
    existing = e.response["Item"]                       # replay from this
    outcome = "replay"
# Redis: SET NX, or a Lua script for the richer state machine.
won = r.set(scoped_key, payload, nx=True, px=86_400_000)
outcome = "won" if won else "replay"

Note ReturnValuesOnConditionCheckFailure="ALL_OLD" — it returns the conflicting item as part of the failed write, saving a second round trip on the replay path. Without it, DynamoDB’s latency disadvantage roughly doubles for duplicates.

The OR expires_at < :now clause is not optional on DynamoDB, and it is the trap in step 4.

Step 3 — Model the cost

# Worked example: 2,000 registrations/sec, 24 h TTL, ~400-byte items.
rps, ttl_h, item_bytes = 2_000, 24, 400
writes_per_month = rps * 2_592_000                        # 5.18 billion
live_items       = rps * ttl_h * 3600                     # 172.8 million
stored_gb        = live_items * item_bytes / 1e9          # ~69 GB

# DynamoDB on-demand: 1 WCU per KB written; duplicates add a read on failure.
dynamo_write_usd = writes_per_month / 1e6 * 1.25          # ~$6,480
dynamo_store_usd = stored_gb * 0.25                       # ~$17
# Redis: sized for peak, paid 24/7 — 3 shards × r7g.2xlarge with replicas.
redis_usd = 6 * 0.60 * 730                                # ~$2,630
print(f"DynamoDB ≈ ${dynamo_write_usd + dynamo_store_usd:,.0f}/mo, Redis ≈ ${redis_usd:,.0f}/mo")
Traffic shape Cheaper store Why
Steady, high throughput Redis Provisioned capacity is fully used; DynamoDB pays per request regardless
Spiky, low duty cycle DynamoDB on-demand No idle spend; Redis must be sized for the peak all month
Very large key space, low rate DynamoDB Storage is far cheaper than RAM per GB
Latency-critical payment path Redis 0.3–1 ms versus 8–15 ms p99; the cost difference is usually second-order

Cost modelling here is directional, not authoritative — run it with your own item sizes and current regional prices before deciding. The shape of the answer is stable even as the numbers move.

Step 4 — TTL semantics differ sharply

This is the most common DynamoDB deduplication bug. DynamoDB’s TTL is a background sweeper, and AWS documents deletion as typically occurring within 48 hours of expiry — not at it. Expired items remain readable, countable and, crucially, still block a conditional write on attribute_not_exists(pk).

# WRONG: a key expired 30 hours ago still blocks the write.
ConditionExpression="attribute_not_exists(pk)"

# RIGHT: treat an expired item as absent, explicitly.
ConditionExpression="attribute_not_exists(pk) OR expires_at < :now"
# Reads must filter too — never trust absence to mean expired.
item = table.get_item(Key={"pk": scoped_key}).get("Item")
if item and item["expires_at"] < int(time.time()):
    item = None            # logically expired, physically still present

Redis has the opposite property: expiry is effectively immediate from the client’s perspective, and there is no lag to compensate for. In exchange, a Redis key can also disappear early under memory pressure unless maxmemory-policy is noeviction, as Redis Cluster vs Sentinel covers.

Step 5 — Partition-key design

DynamoDB spreads load by partition key, and a poorly chosen one creates a hot partition that throttles regardless of provisioned capacity.

# BAD: credential as the partition key. One large tenant saturates one partition.
{"pk": credential_id, "sk": f"{route}#{key}"}

# GOOD: the full scoped key as the partition key. Uniform by construction,
# because the idempotency key itself is high-entropy.
{"pk": f"idem#v1#{credential_id}#{route}#{key}"}

The deduplication access pattern is a pure point lookup — you never range-scan idempotency keys — so there is no reason to use a sort key at all. Putting the whole scoped key in pk gives perfectly uniform distribution for free.

If you do need to enumerate a tenant’s keys for an incident, add a sparse GSI on credential_id with a low projection rather than restructuring the primary key. The GSI costs writes but keeps the hot path uniform.

Verification and testing

Confirm the expiry clause actually works. Insert an item whose TTL has already passed and assert a fresh registration wins.

aws dynamodb put-item --table-name idempotency \
  --item '{"pk":{"S":"idem#v1#probe"},"expires_at":{"N":"1"},"state":{"S":"completed"}}'
python3 -c "from store import register; print(register('idem#v1#probe','fp'))"
# expect: won — an expired item must not block the write

Confirm concurrency.

KEY="idem#v1#concurrency-$RANDOM"
seq 64 | xargs -P 64 -I{} python3 -c "
from store import register; print(register('$KEY','fp'))" | sort | uniq -c
# expect: 1 × won, 63 × replay

Confirm no hot partition under load.

aws cloudwatch get-metric-statistics --namespace AWS/DynamoDB \
  --metric-name ThrottledRequests --dimensions Name=TableName,Value=idempotency \
  --start-time "$(date -u -d '1 hour ago' +%FT%TZ)" --end-time "$(date -u +%FT%TZ)" \
  --period 60 --statistics Sum
# expect: all zeros during a sustained load test

Failure scenarios and debugging

Failure Scenario Remediation Steps Observability Hooks
attribute_not_exists(pk) used alone, so logically expired keys block registrations for up to 48 hours Add OR expires_at < :now to every condition expression and filter expired items on read. idempotency_expired_blocked_total from a comparison of item age against TTL; alert on any value
Credential used as the partition key, and the largest tenant throttles a single partition while the table is far under capacity Move the full scoped key into pk. Add a sparse GSI if enumeration is needed. ThrottledRequests above zero while ConsumedWriteCapacityUnits sits well below provisioned
Redis chosen for a payment path, and a failover loses 40 ms of keys, producing duplicate charges Either move to DynamoDB, or keep Redis as a negative cache in front of a relational unique constraint that is the authoritative record. Duplicate charges per distinct key measured across a forced failover — the number belongs in the degradation policy
DynamoDB latency added to a synchronous checkout path, pushing p99 past the budget Register asynchronously where the operation allows it, or front DynamoDB with a Redis read-through cache for the replay path only — never for registration. idempotency_registration_duration_ms p99 split by store; checkout latency_seconds p99
On-demand billing adopted for steady 2,000 rps traffic, and the bill is several times a provisioned equivalent Switch to provisioned capacity with autoscaling once the traffic shape is known to be steady. Re-model quarterly. ConsumedWriteCapacityUnits variance over 30 days; a flat line means provisioned is cheaper
Partition key choice decides whether you throttle Two partition distributions. Using the credential as the partition key puts every key for the largest tenant into one partition, which saturates while the rest sit idle. Using the full scoped key, which contains the high-entropy idempotency key, spreads writes evenly across all partitions. pk = credential one partition throttles, the rest idle pk = scoped key even — the key supplies the entropy

SRE / observability checklist

  1. idempotency_registration_duration_ms{store} — Histogram split by backend. DynamoDB p99 above 25 ms or Redis p99 above 5 ms indicates a problem in that store, not in the application.
  2. ConditionalCheckFailedException rate — DynamoDB’s native duplicate signal. The healthy band is the same 0.5%–2% replay rate as any other store.
  3. ThrottledRequests — Must be zero. Any value with spare capacity means a hot partition.
  4. idempotency_expired_blocked_total — Counter of registrations blocked by an item past its TTL. Non-zero means the condition expression is missing the expiry clause.
  5. redis_evicted_keys_total — Must be zero on a Redis-backed store; an eviction is a silently deleted guarantee.
  6. idempotency_store label on every metric — So a migration between stores is observable and can be rolled back on evidence.