Redis Cluster vs Sentinel for Deduplication State

Idempotency keys are unusual Redis data: losing one does not degrade performance, it silently removes a correctness guarantee. That changes how you should evaluate the two high-availability topologies, because the usual framing — availability and throughput — misses the question that matters most, which is what happens to acknowledged writes during a failover. This decision guide sits under Redis cache-based deduplication.

Prerequisites. You should have a working atomic registration and a documented degradation policy for when the store is unreachable.

---

Step 1 — Size the key space

The topology choice usually falls out of arithmetic rather than preference.

# Live keys = peak registrations/sec × TTL seconds.
peak_rps, ttl_hours, bytes_per_key = 2_000, 24, 180
live_keys  = peak_rps * ttl_hours * 3600            # 172,800,000
raw_bytes  = live_keys * bytes_per_key              # ~31 GB
with_overhead = raw_bytes * 1.35                    # ~42 GB — hash + expiry structures
print(f"{live_keys:,} keys, ~{with_overhead / 1e9:.0f} GB")

That last row is the one teams skip. Neither topology makes an acknowledged write durable, so a system that genuinely cannot tolerate a lost key needs a unique constraint in the transactional database as the authoritative record, with Redis as a fast negative cache in front of it.

---

Step 2 — Quantify the lost-write window

Both topologies replicate asynchronously and both lose the window. The differences are in how quickly failover completes and how many keys are affected:

WAIT 1 100 after each registration blocks until one replica acknowledges or 100 ms elapses. It shrinks the window materially and does not close it, because a replica that acknowledged can itself fail before promotion. Treat it as risk reduction with a latency cost — typically 1–3 ms added at p99 — not as a durability guarantee.

pipe = r.pipeline(transaction=False)
pipe.set(scoped, payload, nx=True, px=ttl_ms)
pipe.execute_command("WAIT", 1, 100)      # best-effort replication barrier
registered, replicas = pipe.execute()
if registered and replicas < 1:
    metrics.inc("idempotency_unreplicated_write_total")   # visible, not silent

---

Step 3 — Hash tagging on Cluster

Cluster shards by slot, and any multi-key operation — including a Lua script — must stay inside one slot.

def scoped(credential_id: str, route: str, key: str) -> str:
    # The braces delimit the slot-hashed substring. All keys for one operation
    # share a slot, so Lua scripts and MULTI both remain legal.
    return f"idem:v1:{{{credential_id}:{route}:{key}}}"

redis-cli -c CLUSTER KEYSLOT 'idem:v1:{cred_7:POST/v1/charges:01HXYZ}'
redis-cli -c CLUSTER COUNTKEYSINSLOT 5432       # watch for a hot slot

A subtle trap: tagging on the credential alone — idem:v1:{cred_7}:route:key — puts every key for your largest customer on one shard. Include the key itself in the tag so distribution stays even, unless you have a specific reason to co-locate.

---

Step 4 — Configure eviction correctly

# redis.conf for a deduplication instance
maxmemory 48gb
maxmemory-policy noeviction        # NOT allkeys-lru — see below
appendonly yes
appendfsync everysec
save ""                            # AOF only; RDB snapshots stall on large datasets

allkeys-lru is the default instinct and it is wrong here. Under memory pressure it silently deletes idempotency keys to make room, which removes the guarantee without any signal. noeviction makes the instance return an error on write instead, which your fail-closed path surfaces as a 503 — visible, alertable and correct.

Size maxmemory at roughly 70% of instance RAM. Redis needs headroom for the copy-on-write fork during AOF rewrite, and an OOM kill during a rewrite loses far more than the eviction you were avoiding.

---

Step 5 — Test a real failover under load

# Drive steady load with a fixed duplicate ratio, then force a failover mid-run.
k6 run --vus 200 --duration 5m idempotency_load.js &

sleep 120
redis-cli -p 26379 SENTINEL FAILOVER mymaster        # Sentinel
# or, on Cluster:
redis-cli -c -h shard1-replica CLUSTER FAILOVER FORCE

wait
psql -c "SELECT count(*) FROM (
           SELECT idempotency_key FROM charges
            GROUP BY idempotency_key HAVING count(*) > 1) d;"
# This integer is the real, measured cost of a failover for your workload.

Record the number. It belongs in the degradation policy alongside the fail-open/fail-closed decision, because it is the honest answer to “what happens when Redis fails over” — and “nothing” is not that answer.

---

Failure scenarios and debugging

---

SRE / observability checklist

1. redis_evicted_keys_total — Must be exactly zero on a deduplication instance. Any increment means the guarantee is being deleted to make room.
2. redis_master_repl_offset minus slave_repl_offset — Gauge of replication lag in bytes. Alert above 1 MB; it is the size of the lost-write window.
3. idempotency_unreplicated_write_total — Counter of registrations where WAIT returned zero replicas. A rising rate predicts failover damage.
4. redis_moved_redirections_total — Cluster only. A sustained rate means the client’s slot map is stale.
5. redis_memory_used_bytes versus maxmemory — Alert at 80%. With noeviction, hitting the ceiling turns into write errors and a fail-closed endpoint.
6. idempotency_store_unavailable_total — Counter of the fail-closed path. This is what a failover looks like from the application’s side, and it should correlate exactly with the Redis events.

---

Related

• Redis Cache-Based Deduplication — the parent page on Redis as a deduplication substrate and its guarantees.
• Redis Lua Scripts for Atomic Idempotency State Machines — the scripts that require the hash tagging described here.
• Redis vs PostgreSQL for Idempotency State — the prior question of whether Redis should hold the authoritative record at all.
• Chaos Testing Duplicate Request Handling — running the step 5 failover experiment as a scheduled game day.