Consistent Hashing Failure Modes

MEDIUM with RF=3 (successor serves instantly).

Constraint: each node holds 1/100 of total keys (5M keys). The node crashes (OOM, hardware failure, bad deploy). Without replication (RF=1), those 5M keys are lost. Every request misses the cache and hits the database. At 115K ops/sec with 1% affected, that is 1,150 extra DB queries/sec. With RF=3, the clockwise successor already holds the data and takes over seamlessly. Detection: heartbeat stops for 30 seconds, coordinator marks node dead, increments epoch. Gateways refresh within 5 seconds.

One overloaded node degrades latency for all keys it serves, not just the hot key.

Constraint: consistent hashing distributes keys evenly, but not request volume. A viral social media post is cached under one key. That key receives 80% of all reads: $115K \times 0.8 = 92K$ reads/sec to one node while 99 others handle ~230 reads/sec each. The hot node's CPU maxes out, latency degrades from 1ms to 50ms, and the node starts timing out requests. Virtual nodes do not help: they balance key count, not access patterns. Detection: per-node CPU divergence. One node at 90% while peers average 30%. Cache latency p99 for keys on that node exceeds 50ms.

Live traffic degrades during rebalancing but recovers once transfer completes (~42-84 seconds).

Constraint: adding a node transfers 5.25 GB from existing donors at 1 Gbps. The donor nodes must serve live traffic while simultaneously streaming data to the new node. At peak (347K ops/sec), each donor handles 3.5K ops/sec. Adding the I/O load of streaming 5.25 GB saturates the donor's NIC. Live request latency degrades from 1ms to 20ms. What breaks: cache hit ratio drops as donors slow down, and more requests time out and fall through to the database. Detection: donor node network utilization exceeds 80%. Cache latency for keys on donor nodes spikes.

Divergent rings serve different data for the same keys. Silent data inconsistency until detected.

Constraint: the ring spans two datacenters (DC-A and DC-B) for geographic availability. The network link between DCs fails. Each DC believes the other's nodes are dead (heartbeats stop). DC-A's coordinator removes DC-B's 50 nodes from the ring. DC-B's coordinator removes DC-A's 50 nodes. Both DCs rebalance independently, creating two divergent rings. Writes to the same key in both DCs create conflicting values. Detection: sudden loss of exactly 50% of nodes (all in one DC) is a partition signal, not mass failure.

Stale ring causes misrouting for up to 5 seconds. With retry and replica fallback, most requests succeed on the second attempt.

Constraint: gateways poll the ring coordinator every 5 seconds. A node fails and is removed from the ring. The coordinator updates the epoch. A gateway has not yet polled and still has the old ring (epoch N-1). For the next 5 seconds, this gateway routes keys from the failed node's range to the dead node. Reads fail (connection refused) and fall back to replicas (if RF>1). Writes to the dead node are lost unless the gateway retries to the successor. Detection: gateway logs show connection errors to the dead node IP. Ring epoch mismatch in gateway metrics.

Cascading failure can take down the database, turning a cache expiry into a full system outage.

Constraint: a popular key with TTL=3600s expires at T. At T+0, 10,000 concurrent requests arrive, all find a miss, all query the database with the same query. The database, sized for 5% miss rate (5,750 queries/sec), gets 10,000 identical queries in one second. Queue fills. Other queries are blocked. Latency spikes from 10ms to 5 seconds. Cascading failure: more entries expire while the DB is slow, causing more misses. Detection: miss rate spikes 10x in 5 seconds. DB query queue exceeds 1,000.