TRICKYwalkthrough
Timeline Cache
Where does the pre-computed timeline actually live, and why do we store IDs instead of full objects? The constraint: a viral post might appear in 10 million timelines.
We cannot afford that duplication. Each user gets a Redis Sorted Set where members are post IDs (not full post objects) and scores are Snowflake IDs, which double as timestamps for chronological ordering.
“If we store the full post object (roughly 1 KB with text, metadata, and author info) in each timeline, that single viral post consumes 10 TB of cache memory.”
Storing only post IDs keeps each entry at roughly 8 bytes. The sorted set is trimmed to a maximum of 800 entries per user, because users rarely scroll past 800 posts, and keeping more wastes memory without improving the experience.
When a user requests their timeline, our system reads the top 20 post IDs via ZREVRANGE, then hydrates each ID into a full post object by fetching from a separate post store (a dedicated Redis cluster or Memcached). This separation matters: one copy of the viral post lives in the post store, and 10 million timelines hold an 8-byte pointer instead of the full object.
We chose Redis Sorted Sets (not plain Lists) because sorted sets give us score-based range queries and automatic deduplication, while lists would require manual dedup when fanout-on-write inserts the same post ID twice during a retry. Trade-off: we gave up the simplicity of LPUSH/LPOP for the deduplication and range query guarantees of sorted sets.
Implication: 200 million DAU with 800 entries at 8 bytes per entry requires roughly 1.28 TB of Redis memory for timeline caches alone. We partition this across hundreds of Redis shards using consistent hashing on user ID, ensuring that adding or removing a shard only remaps 1/N of the keys.
What if the interviewer asks: 'Why not store full objects and skip the hydration step?' Because a single post update (edit, delete, like count change) would require updating that post in every timeline that contains it. With ID-only storage, we update the post once in the post store and every timeline read gets the latest version automatically.