Design a News Feed

High-Level Architecture

Key Numbers

Capacity Estimation

The arithmetic that motivates every architectural choice:

# Inputs (Twitter-scale)
DAU                = 250,000,000
posts per user/day = 2          (avg, very skewed)
followers per user = 200        (avg, very skewed)
feed opens / day   = 10
posts per feed     = 100        (cached candidate set)

# Writes
posts/sec  = 250M * 2 / 86400  =  ~5,800 posts/sec average
peak       = 5,800 * 3         = ~17,000 peak QPS

# Pure fanout-on-write
fanout writes/day = 250M * 2 * 200       = 100B/day
fanout writes/sec = 100B / 86400         = 1.16M writes/sec average
peak                                     = ~3.5M writes/sec
storage (post_id 8B + score 8B)         = 16 bytes/entry
daily fanout bytes                      = 100B * 16 = 1.6 TB/day

# But 0.01% of users have 500M followers each
1000 celebrity posts/day * 500M followers = 500B writes/day
   ALONE -- 5x the rest of the system put together.

# Reads
feed_opens/sec = 250M * 10 / 86400 = ~29,000 opens/sec avg
peak                               = ~87,000 opens/sec
posts hydrated/sec at peak         = 87,000 * 100 = 8.7M posts/sec

# Storage
post body avg 280 chars + media URLs ~ 1 KB
total posts/day            = 500M
posts storage/day          = 500GB
posts storage/year         = ~180 TB (just text + metadata)
plus media (S3, separate)  = ~4 PB/day at FB scale

The numbers force three decisions: (1) you cannot fanout celebrity posts; (2) the inbox cache must be in RAM (Redis), not on disk; (3) you must shed load on the read path with aggressive caching.

Push (Fanout-on-Write) vs Pull (Fanout-on-Read)

The classic Twitter answer — documented in their 2013 blog "The Infrastructure Behind Twitter" — is hybrid:

• For users with < ~10,000 followers: fanout on write. The post is pushed into each follower's Redis-backed home timeline (sorted set keyed by tweet ID).
• For users with ≥ ~10,000 followers ("celebrities"): fanout on read. The post is stored only in the author's user timeline. When a follower opens their feed, the system reads the home timeline from Redis and queries the user timelines of every celebrity they follow, merging the results.

The threshold is tuned per platform — Facebook used a higher one historically (~100K) because Facebook posts are richer (less frequent, more expensive to merge). Some implementations don't use a hard cutoff but a soft one: every post above N followers triggers fanout to active followers (logged in within X days) only, leaving cold users to be served via pull on demand.

The Inbox: Redis Sorted Sets

The home timeline cache is a Redis sorted set per user. Each member is a post_id; the score is a timestamp (or, for ranked feeds, a precomputed score). Operations:

# Push to follower inbox (called by fanout worker)
ZADD home:{user_id} <timestamp> <post_id>
ZREMRANGEBYRANK home:{user_id} 0 -801   # keep only 800 most recent

# Read top-N for feed open
ZREVRANGE home:{user_id} 0 99 WITHSCORES

# Per-user cap is critical: without ZREMRANGEBYRANK,
# memory blows up. 800 entries x 16 bytes = 12.8 KB / user
# 250M users x 12.8 KB = 3.2 TB cluster-wide

# Sharding: hash(user_id) -> Redis node
# Shard locality: keep all of user's data co-located
# Replication: 1 primary + 2 replicas per shard

# Eviction policy: allkeys-lru
# Cold inactive users get evicted; their feed is rebuilt
# from posts table on next open (acceptable -- rare event)

Twitter's original "Timeline Service" used a custom in-memory store (an evolved memcached called TFE) for exactly this pattern; modern designs typically use Redis Cluster or a managed equivalent like ElastiCache. The cap (Twitter caps at ~800 entries per home timeline) means a user who hasn't opened the app in days only sees the most recent 800 posts they would have, not the full historical backfill — a deliberate product choice.

The Celebrity Path: Merge at Read Time

# Pseudo-code for hybrid feed assembly
def get_home_feed(user_id, limit=100):
    # 1. Read pushed posts (fanout-on-write inbox)
    pushed = redis.zrevrange(f"home:{user_id}", 0, limit*2-1, withscores=True)

    # 2. List celebrity authors this user follows
    celebs = graph.celebrities_followed_by(user_id)  # cached, max ~50

    # 3. Read each celebrity's outbox in parallel
    celeb_posts = []
    with ThreadPoolExecutor(max_workers=20) as pool:
        for celeb_id in celebs:
            celeb_posts.extend(pool.submit(redis.zrevrange,
                f"user:{celeb_id}:posts", 0, 50, withscores=True).result())

    # 4. Merge by score (timestamp or ranked score)
    candidates = heapq.merge(pushed, celeb_posts, key=lambda x: -x[1])

    # 5. Take top-K, send to ranker
    return list(itertools.islice(candidates, 500))

# Total latency budget:
#   Step 1: 5 ms (one Redis hit)
#   Step 2: 1 ms (cached graph lookup)
#   Step 3: 20 ms (N parallel Redis hits, bounded by max followees-of-celebs)
#   Step 4: 5 ms
#   Step 5: 5 ms
#   = ~36 ms before ranking

The number of celebrities any user follows is bounded by social reality (people don't follow more than ~50 mega-accounts), which makes the merge cost predictable. Without that bound, the celebrity path would itself become unbounded — which is why some platforms also restrict who can become a "celebrity" account (verified, public figure designations) or pre-aggregate celebrity posts into a global hot-list.

The Ranking Pipeline

Modern feeds run a multi-stage funnel that turns thousands of candidates into ~30 ranked posts:

Twitter open-sourced parts of "the algorithm" in 2023 — their ranking pipeline matches this shape almost exactly: a SimClusters-based candidate generation, a "light ranker" using a real-time feature store, and a "heavy ranker" neural model. Facebook's analogous pipeline was described in their 2017 paper "DLRM: Deep Learning Recommendation Model."

Database Choices

Cassandra is the canonical choice for posts because the data shape fits its strengths exactly: partitioned by user_id (one shard per author), clustered by timestamp descending, writes never update existing rows (just append a new post). Reads are always "latest N posts by user X," which becomes a single-shard, single-partition seek. Twitter built Manhattan as a Cassandra-style store with operational improvements; Facebook built TAO as a graph-aware cache layer over MySQL.

Twitter's Home Timeline, Specifically

Twitter's blog and engineering talks are unusually open about their architecture. The shape, as of 2023:

• Tweet creation → written to Manhattan (their custom key-value store, conceptually similar to Cassandra). User timeline (the author's own posts) is materialized.
• Tweet event → Kafka → fanout service.
• Fanout queries the social graph (followers of author). For each follower with < 10K followees and recently active, push the tweet ID into their Timeline Service (TLS) home timeline cache — an in-memory store derived from Redis-style ZSETs.
• Heavy hitters (accounts with millions of followers) skip fanout. Their tweets are pulled at read-time.
• On feed open, the Home Mixer service: reads the TLS home timeline, fetches candidate tweets from celebrity outboxes, queries other candidate sources (Out-of-Network, e.g. SimClusters-recommended tweets from accounts you don't follow), batches them through the light ranker, then the heavy ranker, applies diversification, returns ~30 tweets.
• Hydration happens last: take the final 30 tweet IDs, pull tweet bodies and media URLs in parallel, pull engagement counts (likes/retweets) from a separate counters service, return to the client.

Tradeoffs & Failure Modes

• Fanout backlog under bursts. A celebrity who suddenly gains followers can spike fanout queue depth. Solution: rate-limit per-author fanout, prioritize active follower delivery, accept eventual consistency for inactive users.
• Hot author shards. A celebrity's user_timeline is read by millions on every feed open. Solution: replicate hot-author timelines to many cache nodes, pre-warm during traffic spikes (live events).
• Inbox cache eviction storms. If a Redis node fails and its replicas can't keep up, millions of users see empty feeds while the cache rebuilds. Solution: use replicated sorted-sets, slow-rebuild from Cassandra in background.
• Ranking model drift. A trained model is good for a few days; user behavior shifts. Continuous online training pipelines (Spark/Flink) are mandatory; A/B harnesses to evaluate new models before rollout are a separate problem of comparable size.
• Privacy / blocking. A user blocks another. The blocked user's posts must not appear in feed. Either filter at fanout time (small inboxes contaminate) or filter at hydration time (read-time cost). Most platforms filter at hydration since blocks are rare and the post-blacklist join is cheap.
• Edited / deleted posts. Posts in inboxes become stale. Either store only IDs and re-fetch at read time (always fresh, more reads), or push edits/deletes through Kafka to invalidate (more writes, lower read amplification). Most large platforms do the former.
• Mute / unfollow churn. Frequent follow/unfollow doesn't backfill or remove the inbox; future posts are correctly filtered, but historical posts in the cache are not. Acceptable to most products.
• Cold-start ranking. A brand-new user has no engagement history; the heavy ranker has nothing to score from. Fall back to popularity-based defaults until ~10 interactions accumulate, then transition to personalized model.