STANDARDwalkthrough
Campaign Fanout
Marketing schedules one message to "all users active in the last 30 days": a single API call that must become 100 million device-level sends. If the goal is delivering the wave within 10 minutes, the pipeline must sustain of fanout on top of normal traffic.
Option one: synchronous expansion at the API. The campaign service queries the segment and writes 100M messages into the queue in one transaction.
“How do we expand one logical message into 100M physical ones?”
The segment query alone scans hundreds of millions of user rows; the API call times out and a retry double-sends. Option two: two-stage fanout.
Stage one: the campaign service resolves the segment into recipient chunks of 10,000 user IDs each (10,000 chunks for 100M) and publishes each chunk as a single compact message. Stage two: fanout workers consume chunks, look up each user's active tokens and preferences, apply per-user rate limits and quiet hours, and emit per-device messages to the P2 topic.
We chose two-stage. The chunk layer gives natural parallelism (10,000 independent units), checkpointed progress (a crashed worker re-reads one chunk, not the campaign), and a place to apply preference filtering close to the data.
The numbers: chunk resolution takes one indexed segment scan; 200 fanout workers each processing 850 sends/sec sustain the 167K/sec target. The trade-off: two-stage delivery is eventually consistent with the segment.
A user who opts out after chunk resolution but before their chunk processes can still receive the push; we re-check preferences at send time to shrink that window to seconds. What if the interviewer asks: why chunks of 10,000 and not 1 or 1M?
One message per user makes the campaign topic as hot as the send topic (no savings); 1M-user chunks make retry blast radius enormous. 10K balances checkpoint granularity against message overhead.
Related concepts