News Feed Failure Modes

Affects timeline freshness for all users, not only the celebrity's followers. A 5-minute delay makes the platform feel broken.

Constraint: fanout-on-write capped at 10K followers. What breaks: misconfiguration lets a celebrity with 30M followers trigger push fanout into 30M timeline caches. At 100K writes/sec per shard, that is 300 seconds of sustained writes for one tweet. Every other tweet sits in the Kafka fanout queue, unable to drain. Detect: consumer lag above 1M messages, fanout latency above 30s, Redis write throughput at 100% on multiple shards. Recover: circuit breaker switches ALL users to fanout-on-read until queue drains below 100K. Impact: all users' tweets delayed 5+ minutes.

Cascading load on the database can trigger a full outage if rebuilds are not throttled.

Constraint: Redis holds 10M timeline caches per node for sub-10ms reads. What breaks: a node restarts after a crash. Those 10M users now have empty timelines. Each miss triggers a cold rebuild: query followees, fetch tweets per shard, merge, sort, populate. Each rebuild takes 200-500ms and hits the DB. If 1M users open the app within 5 minutes, the DB gets 3,333 QPS of multi-shard reads on top of normal load. Detect: miss rate spiking from 2% to 50%+, DB CPU rising, timeline p99 jumping from 50ms to 500ms+. Recover: RDB snapshots reload in 30 seconds, recovering 95%+ of caches.

Temporary inconsistency (2-5 seconds). Filtered at read time as a safety net. No data loss.

Constraint: follows sync across MySQL and Redis via async Kafka events. What breaks: user A unfollows B. The API deletes from MySQL and removes B from A's following set. But the Kafka event to update B's followers set lags 3 seconds. B tweets in that window. Fanout reads B's followers from Redis, still sees A, pushes B's tweet into A's timeline. A sees a tweet from someone they unfollowed. Detect: unfollowed-account tweet reports, MySQL follows vs Redis SET cardinality delta above 0.1% for 30+ seconds. Recover: read-time filter drops tweets from users not in the current following set.

Duplicate IDs cause INSERT failures (lost tweets). Out-of-order IDs cause timeline misordering. Both erode user trust.

Constraint: Snowflake IDs depend on monotonic timestamps across generator nodes. What breaks: a node's NTP sync drifts by 2 seconds, generating IDs with past timestamps that sort before IDs from other nodes. If the clock jumps forward after correction, the node reuses a timestamp, creating duplicate IDs. Two tweets sharing an ID cause a primary key collision on INSERT. Detect: PK collision errors, Snowflake timestamp diverging from wall-clock by 500ms+, NTP drift alerts. Recover: affected node halts ID generation until the clock catches up. Impact: lost tweets and timeline misordering.

Users experience stale timelines for several minutes. During major events, the backlog compounds rapidly.

Constraint: 12 Kafka partitions process fanout writes. What breaks: one consumer pod crashes, K8s takes 60s to reschedule. The dead consumer's 4 partitions accumulate $4/12 \times 5{,}787 \times 60 = 115{,}740$ unprocessed tweets. Each fans out to 200 followers: 23M pending writes. Replacement needs 4 min to drain at 100K writes/sec. Detect: consumer lag above 50K, pod restarts, fanout latency above 60s. Recover: auto-scaling spins up replacements, cooperative-sticky rebalancing reassigns partitions in 5 seconds. Impact: stale timelines for minutes, compounding during major events.

Expected behavior in an eventually consistent system. Noticeable only if the user posts and immediately checks another device.

Constraint: eventual consistency for fanout with a 2-5 second target. What breaks: user A posts a tweet, written to MySQL and published to Kafka. User B refreshes 1 second later. The fanout consumer has not processed A's tweet yet (latency is 2-5 seconds). B sees a stale feed, refreshes 10 seconds later, tweet appears. Detect: fanout latency p50 at 2s and p99 at 8s, or timelines served within 1 second of a tweet missing it 90% of the time. Recover: write-through inserts A's tweet into A's own timeline immediately; client-side optimistic insertion shows it locally before fanout confirms.