Chat System Anti-Patterns

Polling for new messages every few seconds wastes bandwidth and adds latency. At 500M users polling every 5 seconds, that is 100M HTTP requests/sec for mostly empty responses.

Why: HTTP polling is simpler to implement: standard request-response, no connection state, works with any load balancer. Candidates default to what they know. But at 500M users polling every 5 seconds: $500M / 5 = 100M$ requests/sec. With 200 bytes of HTTP headers per request, that is 20 GB/sec of wasted bandwidth for mostly empty responses ('no new messages'). Message delivery latency averages half the polling interval (2.5 seconds), unacceptable for a chat system targeting 200ms.

A single auto-increment counter for all messages creates a serialization bottleneck at 231K/sec. Per-conversation counters eliminate cross-conversation contention entirely.

Why: A global counter gives total ordering: every message has a unique, strictly increasing number. It is the simplest mental model. But at 231K messages/sec, every increment serializes behind a single lock. Even Redis INCR, which handles 100K ops/sec per shard, needs 3 dedicated shards. More critically, the counter becomes a single point of failure: if it goes down, no messages can be ordered. Candidates do not realize that chat messages only need ordering within a conversation, not across all conversations.

Broadcasting every status change to all contacts generates 417M events/sec at morning peak. Lazy presence with subscriber-only push reduces this by 100x.

Why: It feels correct: when User A comes online, notify all 500 contacts immediately so their UI updates. The feature works perfectly in testing with 10 users. But at morning peak, 500M users come online over 10 minutes. Each triggers 500 notifications: $500M \times 500 / 600\text{s} = 417M$ presence events/sec. No pub/sub system handles this. Even Kafka at 10M/sec would need 42 partitions just for presence, and consumers could not keep up.

MySQL's B-tree indexes require random disk I/O for each insert. At 231K writes/sec, replication lag grows to minutes. Cassandra's LSM-tree handles append-only writes at O(1).

Why: MySQL is the default choice for structured data. A messages table with indexes on (conversation_id, created_at) seems straightforward. At low volume, it works. But at 231K inserts/sec, each insert triggers a B-tree page split. Random I/O kills throughput. MySQL replication lag grows from milliseconds to minutes. Secondary indexes double the write cost. Cassandra's LSM-tree converts random writes to sequential appends, handling 231K writes/sec on a modest cluster.

Without a connection registry, every message is broadcast to all 10K gateway servers. That is 2.31B network calls/sec. A Redis lookup costs 1ms and eliminates 9,999 wasted calls per message.

Why: Broadcasting avoids maintaining a connection registry. Every server receives every message and checks if the recipient is connected locally. This works with 10 servers. At 10,000 servers and 231K messages/sec: $231K \times 10{,}000 = 2.31B$ network calls/sec. Each server receives 231K messages/sec regardless of how many recipients it holds. CPU is wasted on 99.99% of messages that have no local recipient.

Writing to Cassandra before acking the sender adds 50ms of p99 latency to delivery. Kafka decouples delivery from persistence, keeping ack time at 12ms.

Why: It feels safe: persist the message before telling the sender it was delivered. Guarantees durability. But Cassandra write latency is 5ms at p50 and 50ms at p99. Adding 50ms to the delivery path pushes total latency above 60ms at p99, and during compaction storms, Cassandra writes can spike to 200ms+. The sender's UI freezes until the ack arrives. Users perceive the chat as 'laggy'.

Without dedup, a lost ack causes the sender to retry and the recipient sees the same message twice. Idempotency keys in Redis prevent duplicate delivery with zero user impact.

Why: In the happy path, messages are delivered exactly once. Candidates test the happy path and move on. But network is unreliable. The server delivers to User B, writes to Cassandra, but the ack to User A is lost in transit. A's client retries after 3 seconds. Without dedup, B sees the message twice. At 0.1% retry rate and 231K messages/sec, that is 231 duplicate messages per second, roughly 20M duplicates per day visible to users.

Gateways hold WebSocket connections in memory. Round-robin load balancing sends a message to a random gateway that does not hold the recipient's connection. Use the connection registry for targeted routing.

Why: Stateless services are the default in modern architectures. Load balancers distribute requests evenly via round-robin. But gateways are inherently stateful: each holds a set of WebSocket connections. A message for User B must reach the specific gateway holding B's connection. Round-robin sends the message to any of 10,000 gateways, and 9,999 of them cannot deliver it. The message is lost or must be re-routed, adding 10-50ms of latency.