etcd vs ZooKeeper vs Redis for Distributed Locking

Before comparing lock services it is worth asking whether you need one. Deduplicating a request requires a single atomic conditional write — SET NX or a unique constraint — not a lock. A lock service earns its place when a handler coordinates across several resources, or when exactly one instance must own a partition or a cron job. This guide sits under distributed lock acquisition patterns and assumes you have established that need.

Prerequisites. Familiarity with lease semantics from lock timeout & lease management, and with fencing tokens, which decide this comparison.


Step 1 — Efficiency lock or correctness lock?

Efficiency locks versus correctness locks A single question at the top asks what happens if two holders run at once. The left branch, labelled efficiency lock, covers cases such as regenerating a cache or sending a duplicate report where the outcome is merely wasteful; any lock service including Redis is adequate. The right branch, labelled correctness lock, covers debiting a balance or issuing a payout where a duplicate corrupts state; a consensus-backed service plus a resource-enforced fencing token is required. What if two holders run at once? answer this before comparing anything Efficiency lock duplicate work is merely wasteful cache regeneration · report rendering idempotent batch reprocessing Redis is fine · lowest latency wins Correctness lock duplicate work corrupts state debiting a balance · issuing a payout allocating a limited resource consensus + fencing token, always

Step 2 — The acquisition primitive

// etcd: a lease plus a compare-and-swap on create-revision. The Put's revision
// is a fleet-wide monotonic fencing token, produced by Raft.
lease, _ := cli.Grant(ctx, 30)                                  // 30-second TTL
resp, err := cli.Txn(ctx).
    If(clientv3.Compare(clientv3.CreateRevision(key), "=", 0)).
    Then(clientv3.OpPut(key, holderID, clientv3.WithLease(lease.ID))).
    Commit()
token := resp.Header.Revision
keepAlive, _ := cli.KeepAlive(ctx, lease.ID)                    // heartbeat renewal
// ZooKeeper via Curator: an ephemeral sequential znode. The sequence number is
// the fencing token; the ephemeral node disappears when the session ends.
InterProcessMutex mutex = new InterProcessMutex(client, "/locks/settlement");
if (mutex.acquire(5, TimeUnit.SECONDS)) {
    try {
        Stat stat = client.checkExists().forPath("/locks/settlement");
        long token = stat.getCzxid();       // consensus-assigned transaction id
        doWork(token);
    } finally {
        mutex.release();
    }
}
-- Redis: SET NX with a lease, plus INCR for a token. Atomic within one shard.
if redis.call('SET', KEYS[1], ARGV[1], 'NX', 'PX', tonumber(ARGV[2])) then
  return redis.call('INCR', KEYS[2])
end
return 0

The three differ in what happens when the service itself fails. etcd and ZooKeeper cannot grant the same lock twice, because a grant is a committed consensus entry and a new leader has it. Redis can, because a promoted replica may never have received the grant.


Step 3 — Fencing-token quality is the deciding factor

Property etcd ZooKeeper Redis (single) Redis (Redlock)
Token source Raft revision zxid / sequential znode INCR counter INCR on each node
Strictly monotonic across failover Yes Yes No No
Double-grant possible on failover No No Yes Yes (contested)
Session/lease expiry mechanism Lease + keep-alive Session heartbeat Key TTL Key TTL per node
Automatic release on client death Yes, lease expiry Yes, ephemeral node Only on TTL Only on TTL
Typical acquire latency (in-region) 3–10 ms 5–15 ms 0.3–1 ms 2–5 ms
Cluster size for HA 3 or 5 3 or 5 1 + replica 5 independent
Operational weight Moderate High (JVM, ensemble) Low Moderate
Already running it? If you run Kubernetes Often with Kafka/HBase Almost always

The rows that matter for a correctness lock are the first three. A token that can regress is not a fence; it merely looks like one, and the failure surfaces as data corruption during an incident rather than as an error anyone can see.

That does not make Redis useless for correctness locks. It makes Redis-plus-a-resource-fence viable, where the resource itself records the highest token it has accepted — because then a regressed token is rejected by the database, not trusted by the lock service. What is not viable is Redis alone.


Step 4 — Latency versus operational cost

# Measure acquisition latency for your own workload rather than trusting a table.
etcdctl --endpoints=$ETCD check perf --load=s        # etcd's own benchmark
redis-cli --latency-history -i 5                     # Redis round-trip baseline
# A fair comparison: full acquire + release cycle under contention.
def bench(lock_factory, holders=32, iterations=200):
    latencies = []
    with ThreadPoolExecutor(holders) as pool:
        def one(_):
            t0 = time.perf_counter()
            lease = lock_factory.acquire("bench", lease_ms=5_000)
            lock_factory.release(lease)
            latencies.append((time.perf_counter() - t0) * 1000)
        list(pool.map(one, range(iterations * holders)))
    latencies.sort()
    return latencies[len(latencies) // 2], latencies[int(len(latencies) * 0.99)]

The honest framing: etcd costs roughly 5–10 ms more per acquisition than Redis. If a lock is taken once per batch job that is irrelevant. If it is taken once per request at 2,000 requests per second, it is a design constraint — and usually a sign the design should be using an atomic conditional write instead of a lock.

If you already run Kubernetes, you already run etcd. Using it for application locks is tempting and usually wrong: application lock churn competes with the control plane for the same Raft log. Run a separate etcd cluster for application locks, and budget for it.


Step 5 — Verify liveness during leader election

Client behaviour during a lock-service leader election A timeline shows the lock service leader failing, an election window of roughly one to five seconds during which acquisition requests fail, and a new leader taking over. Two client behaviours are contrasted: failing closed by rejecting the work, which is correct, and proceeding without the lock, which silently removes the guarantee. leader fails election: 1–5 s, no grants new leader grants resume, tokens continue upward Fail closed — correct acquire() raises; the job is deferred availability dips, correctness holds Proceed anyway — silent failure "lock service down, carry on" two holders, no error, corrupted state
# etcd: kill the leader and measure the acquisition gap.
LEADER=$(etcdctl --endpoints=$ETCD endpoint status -w json \
         | jq -r '.[] | select(.Status.leader == .Status.header.member_id) | .Endpoint')
./lock_probe --interval 100ms --duration 60s &        # logs every acquire attempt
sleep 10 && ssh "$LEADER" sudo systemctl stop etcd
wait
awk '/FAIL/ {n++} END {print "failed acquisitions:", n}' lock_probe.log
# expect: a contiguous run of failures ~1–5 s long, then recovery

# And the important assertion: no work proceeded without a lock.
grep -c 'proceeded without lock' app.log       # expect: 0

Failure scenarios and debugging

Failure Scenario Remediation Steps Observability Hooks
Redis chosen for a correctness lock without a resource-enforced fence, and a failover grants the same lock twice Add a fence_token column and enforce it in the write’s WHERE clause, or move the lock service to etcd. Replication alone never closes this. lock_fence_rejected_total — zero while lock_double_grant_suspected_total is non-zero means no fence exists
Client treats a lock-service outage as “proceed without the lock” Fail closed: an unavailable lock service means the work is deferred, not done unlocked. Make the acquire call throw rather than return a nullable lease. lock_acquire_error_total alongside a proceeded_without_lock_total counter that must be structurally zero
Application locks placed in the Kubernetes control-plane etcd, and lock churn slows the API server Run a dedicated etcd cluster for application locks. Never share the control plane’s Raft log. etcd_server_proposals_committed_total rate correlated with application deploy activity; API-server latency
ZooKeeper session timeout shorter than a GC pause, so the ephemeral node vanishes while the holder still runs Set the session timeout above the JVM’s p999 pause, and add a fence so a resumed holder cannot write regardless. zk_session_expired_total; jvm_gc_pause_seconds p999 against the configured timeout
Lock held across an external HTTP call, so a slow dependency holds the lock for its full timeout and everything queues Acquire, do local work, release; call external services outside the lock with their own idempotency key. lock_hold_duration_seconds p99; alert above 1 s for request-path locks
What a failover does to the fencing token Two token sequences plotted across a failover event. The consensus revision continues to increase because the grant was a committed log entry the new leader already has. The Redis counter regresses because the increment had not replicated, so a token value already handed out is issued a second time — which makes the fence useless exactly when it is needed. failover consensus revision — never regresses Redis counter — reissues used values

SRE / observability checklist

  1. lock_acquire_duration_seconds — Histogram per service. etcd p99 above 50 ms or Redis p99 above 5 ms indicates the lock service is degraded.
  2. lock_acquire_error_total — Counter of failed acquisitions. Expected to spike briefly during leader election; a sustained rate is an outage.
  3. proceeded_without_lock_total — Must be structurally zero. Any value means a code path fails open on a correctness lock.
  4. lock_hold_duration_seconds — Histogram. A p99 near the lease duration means holders are routinely close to expiry and fence rejections are imminent.
  5. lock_fence_rejected_total — Counter of stale writes rejected at the resource. Non-zero proves the fence works and that holders do freeze.
  6. etcd_server_has_leader / zk_server_state — Gauge of quorum health. A quorum without a leader grants nothing, and clients must fail closed.