Design a News Feed

A News Feed (or Home Feed) is the constantly updating list of stories, posts, and updates that users see when they open a social application. It is the heart of platforms like Facebook, LinkedIn, Instagram, X (Twitter), and Threads. Designing a feed system that is personalized, fresh, and fast at the scale of billions of users is a classic and challenging system design interview question. This article explores the architecture, algorithms, and trade-offs behind building a modern, scalable news feed.

Step 1: Requirements Clarification

Functional Requirements

• Personalized home feed – Each user sees a unique, ranked list of posts from accounts they follow, as well as recommended content.
• Infinite scrolling – Users can continuously load older posts without explicit page numbers.
• Support diverse content – Text, images, videos, links.
• Engagement actions – Users can like, comment, share, and hide posts directly from the feed.
• Feed refresh – Users can pull to refresh and see new content.
• Pagination – Stateful, cursor-based pagination for consistent, non-duplicative scrolling.
• Real-time updates (optional) – New posts can be inserted at the top of the feed while the user is browsing.

Non-Functional Requirements

• Low latency – Feed must load in under 200ms. Users abandon slow apps.
• High availability – 99.95%+ uptime. Feed unavailability directly impacts engagement and revenue.
• Horizontal scalability – System must handle billions of users and trillions of posts.
• High read throughput – Feeds are read orders of magnitude more than they are written.
• Eventual consistency – A very slight delay (seconds) in seeing a new post is acceptable.
• Fault tolerance – Partial failures (e.g., ranking service) should not disable the entire feed.
• Freshness – New posts should appear in the feed within seconds.
• Personalized ranking – Posts should be ordered by relevance to the user, not just time.

Step 2: Capacity Estimation

Assume a large social platform:

• Registered users: 2 billion
• Daily active users (DAU): 500 million
• Feed requests per second: Average 5 reads per user per session → ~30,000 reads/sec steady; peak 10x → 300,000 reads/sec.
• Posts created per second: 10,000 average, 50,000 peak.
• Average follows per user: 200.
• Average feed size displayed: 20 posts per page.
• Storage for post metadata: ~500 TB/year, media multiple PB/year.
• Cache size for hot feeds: Terabytes of in-memory data.

News Feeds are among the most read-heavy systems on the Internet. A single post may be read millions of times, which drives the architectural focus on optimizing reads.

Step 3: API Design

REST APIs with cursor-based pagination for consistency.

• Get Home Feed
  GET /api/v1/feed?limit=20 (initial load)
  Returns: { posts: [PostObject, ...], next_cursor: "encrypted_cursor_string" }

• Load More
  GET /api/v1/feed?cursor={next_cursor}&limit=20

• Refresh Feed (pull-to-refresh)
  GET /api/v1/feed?since={last_post_id}

• Like Post POST /api/v1/posts/{id}/like

• Hide Post POST /api/v1/posts/{id}/hide

• Publish Post POST /api/v1/posts (triggers feed generation)

Pagination uses an opaque cursor (e.g., a timestamp + post ID encoded) instead of OFFSET, because feeds are continuously changing. OFFSET would cause skipped or duplicated items.

Step 4: High-Level Architecture

The architecture separates write-time feed pre-computation from read-time serving.

• Feed Service orchestrates feed delivery. It queries the pre-built feed from cache, applies ranking, and returns posts.
• Ranking Service scores posts based on user affinity, recency, and engagement probability.
• Social Graph Service provides the list of followees for fan-out.
• Post Service manages post creation and storage.
• Message Queue decouples post creation from feed fan-out and ranking updates.
• Feed Cache (e.g., Redis) holds the pre-computed list of post IDs for each user.
• Feed Store is the durable database backing the cache.

Step 5: Feed Generation Strategies

The core decision is when to assemble the list of posts that go into a user's feed.

Fan-out on Write (Push)

When a user publishes a post, the system immediately writes that post's ID into the feed caches of all their followers.

• Advantages: Feed reads are extremely fast—just a single cache lookup. Low read latency.
• Disadvantages: Write amplification: a user with 10 million followers causes 10 million cache writes. That's slow and wastes resources, especially for inactive followers. Also, updating the feed for a new follow requires backfilling historical posts, which can be heavy.

Fan-out on Read (Pull)

At read time, the system queries the recent posts of every account the user follows, merges them, and ranks them.

• Advantages: No write amplification; storage efficient. Easier to incorporate real-time ranking signals.
• Disadvantages: Read latency is higher. If a user follows 1000 accounts, the system must query up to 1000 timelines, which can be slow.

Hybrid Feed Generation

Modern platforms use a combination:

• For regular users (the vast majority), they use fan-out on write: when they post, the post is pushed to the feeds of their followers.
• For celebrity or ultra-popular users (millions of followers), they do NOT fan-out on write. Instead, during feed read, the system pulls the celebrity's recent posts from a dedicated timeline cache and merges them into the pre-built feed.
• This approach optimizes both write efficiency and read latency.

Strategy	Write Amplification	Read Latency	Celebrity Handling	Consistency
Fan-out on Write	High	Low	Poor	High
Fan-out on Read	Low	High	Good	Low
Hybrid	Balanced	Low	Excellent	Balanced

Step 6: Feed Ranking

A chronological feed is simple but quickly becomes irrelevant for users following hundreds of accounts. Ranking orders posts by predicted interest.

Reverse chronological feeds are used for timelines (user's own posts) and some platforms like Twitter's "Latest" mode, but the home feed usually employs ML ranking.

The ranking pipeline conceptually involves:

1. Candidate generation: Fetch the list of post IDs from the feed cache (already filtered by followees).
2. Feature retrieval: For each candidate post, fetch features: user affinity to author, recency, post type (text/image/video), engagement rate, content quality signals.
3. Scoring: A machine learning model (often a lightweight logistic regression or gradient boosted trees model, or a neural network for larger platforms) predicts the probability of positive engagement (like, comment, share, meaningful interaction).
4. Sorting by score.
5. Re-ranking with diversity and business rules: Ensure no two posts from the same author appear consecutively; insert advertisements at predetermined slots; apply content moderation filters; boost posts from "close friends".

Ranking is pre-computed asynchronously to keep feed reads fast. The Ranking Service continuously updates scores and reorders the feed cache.

Step 7: Feed Storage Design

The feed system uses multiple storage layers.

• Feed Cache (Hot Storage): A distributed in-memory store (e.g., Redis) mapping user_id to a sorted list of post_ids with scores (for ranking). This is the primary source for feed reads. Data is partitioned by user_id across a Redis cluster.
• Feed Store (Cold Storage): A durable database (e.g., Cassandra, DynamoDB) that permanently stores the feed entries. It serves as the source of truth and for rebuilding caches. Each row represents a single entry: (user_id, post_id, created_at, score).
• Post Store: Sharded by post_id or author_id, stores full post content and metadata. Accessed for rendering.
• Timeline Store: A separate store for user timelines (all posts by a specific user), used for fan-out on read and celebrity pull.

Storage	Use Case	Technology
Feed Cache	Serving feed reads	Redis, Memcached
Feed DB	Durable feed storage	Cassandra, DynamoDB
Post DB	Post content	PostgreSQL (sharded), Cassandra
Timeline DB	User-specific posts	Cassandra, HBase
Media	Images/videos	Object Storage + CDN

Step 8: Caching Strategy

Caching is the single most critical performance optimization.

• Feed Cache: Each active user has their pre-built feed (list of post IDs) in Redis. This serves reads with sub-millisecond latency.
• User Cache: User profile, followees list, and account settings are cached to accelerate fan-out.
• Post Cache: Recently accessed post content is cached in a key-value store to avoid hitting the Post DB on every render.
• Edge Cache: For static assets and media, a CDN caches at global points of presence.
• Cache invalidation: When a post is deleted or a user is blocked, the system must remove the post ID from affected feed caches. This is handled by a separate invalidation worker that subscribes to relevant events.
• Cache warming: When a user logs in, the feed service pre-emptively loads their feed into cache if not present. For inactive users, the cache is not warmed, saving resources.
• Celebrity cache: Popular celebrity posts are cached separately in a small, highly replicated cache, as they are pulled by millions of reads.

Step 9: Handling Celebrity Users

A user with millions of followers breaks the fan-out on write model. The hybrid strategy specifically addresses them:

• Identification: Users with follower count exceeding a threshold (e.g., 1 million) are marked as "heavy" users.
• Separation: Their posts are NOT fan-out written to follower feeds. Instead, they are stored in a dedicated, small celebrity post cache.
• Feed Merge at Read: When building a follower's feed, the Feed Service:
  1. Retrieves the pre-built feed (regular followees) from the user's feed cache.
  2. Also queries the celebrity post cache for recent posts from any celebrities the user follows.
  3. Merges the two lists and applies ranking.
• Optimization: Since many users follow the same celebrity, the celebrity cache is extremely hot. It is aggressively replicated or partitioned to handle the load.

This technique, pioneered by Twitter and adapted by many, is fundamental to scalability.

Step 10: Timeline vs News Feed

These two concepts are often confused but serve different purposes.

Feature	Timeline	News Feed
Content	Posts by the user	Posts from others the user follows
Ownership	Owned by the user	Aggregated for the user
Storage	Sharded by author_id	Sharded by reader_id (in fan-out on write)
Ranking	Chronological	Algorithmic
Read Pattern	Typically lower read volume	Extremely high read volume
Personalization	None	Highly personalized

The Timeline is simpler to implement; the News Feed is the complex, scaled system described here.

Step 11: Scalability Strategies

• Horizontal scaling: Feed Service, Ranking Service, and Graph Service are stateless and scale out behind load balancers.
• Feed partitioning: Feed caches and databases are sharded by user_id (reader) for fan-out on write; feed reads hit a single shard.
• Sharding posts: Post DB sharded by author_id to quickly retrieve a user's timeline.
• Read replicas: For relational databases storing user profiles and settings.
• CDN for media: All static and media assets served from CDN edge nodes.
• Queue-based processing: Kafka handles the firehose of new posts, enabling asynchronous fan-out and ranking updates without slowing the write path.
• Geo-distributed deployment: The system is deployed in multiple regions. A user's feed cache lives in their primary region for low latency. Celebrity caches are replicated globally.

Step 12: Reliability and Failure Handling

• Queue failures: If the fan-out worker fails, posts remain in Kafka for replay. No data is lost.
• Cache failures: If a user's feed cache shard goes down, the Feed Service falls back to assembling the feed from the feed database (pull model) until the cache recovers. This is slower but keeps the system operational.
• Feed rebuilding: In case of catastrophic cache failure, feeds can be rebuilt by replaying the stream of recent posts for a user.
• Partial ranking degradation: If the ML Ranking Service is slow or down, the system serves a default chronological feed instead of a ranked one. The feed remains functional.
• Retry and circuit breaker: For transient failures, services retry with backoff; persistent failures trip circuit breakers to prevent cascading failures.
• Graceful degradation: The feed can be served without images if the CDN is down, or without rich metadata if the Post Service is slow.

Step 13: Security and Privacy

• Authentication & Authorization: All requests pass through auth. Users can only see feeds they are authorized for.
• Privacy filtering: Feed queries must respect privacy settings (e.g., posts shared with "Friends only"). This filtering can happen at query time or be baked into the feed cache (by partitioning feeds into "Friends" and "Public" caches per user). Complex privacy models favor runtime filtering.
• Blocked users: If user A blocks user B, B's posts must be removed from A's feed cache and A must not see them. An invalidation service handles this.
• Content moderation: Offensive posts are flagged by automated systems and removed from feeds. A moderation pipeline processes reports.
• Abuse detection: Rate limiting on feed reads and post creations prevents scraping and spam.

Real-World Example: Facebook-like News Feed

Let's trace the lifecycle of a post from creation to feed display in a hybrid model.

1. Alice posts. The Post Service persists the post and fires an event.
2. Feed Worker fans out the post ID to the feed caches of Alice's regular followers.
3. Ranking Service asynchronously updates scores.
4. When Bob requests his feed, the Feed Service gets his pre-built list from cache, merges with any celebrity posts, fetches post details, applies final ranking, and returns the feed. The entire read path is extremely fast.

Trade-offs

• Fan-out on Write vs Read: Write fan-out makes reads lightning fast but increases storage costs and write amplification. Read fan-out keeps writes simple but adds read latency. The hybrid model balances both.
• Feed freshness vs latency: Real-time fan-out (on write) gives near-instant feed updates but requires heavy infrastructure. A slight delay (seconds) reduces cost significantly and is often acceptable.
• Cache size vs storage cost: Storing full feed lists in RAM is expensive. Evicting inactive users and keeping only "hot" users in cache reduces cost but requires falling back to disk for dormant users.
• Ranking complexity vs response time: Complex ML models improve relevance but take longer to score. Pre-computing scores and caching them is standard; final lightweight re-ranking can happen at read time.
• Personalization vs scalability: Every additional personalization feature (e.g., affinity-based weighting) increases the amount of data stored per user. Efficient feature storage and retrieval are critical.

Common Mistakes

• Using OFFSET pagination: In a dynamic feed, it leads to duplicates and gaps. Cursors are mandatory.
• Generating feeds synchronously on every read: Won't scale beyond a trivial number of users. Pre-computation (fan-out on write) is necessary.
• Ignoring celebrity users: A naive push model will crash under the load of a single celebrity post.
• Not caching feeds: Directly querying the post database and social graph for every feed read will result in unacceptably high latency.
• Recomputing rankings on every request: Drains database and CPU resources. Rankings must be pre-computed and stored.
• Tight coupling between feed and post services: Post creation should be fast and decoupled from feed fan-out. Use events.

Interview Perspective

Interviewers want to see that you can balance the extreme read/write asymmetry. Expect questions like:

• Design Facebook News Feed.
• What is fan-out on write vs fan-out on read?
• How do you handle celebrity users?
• How do you rank posts?
• How do you cache feeds?
• How do you paginate feeds?

Demonstrate understanding of the hybrid fan-out, the need for pre-computation, and the decoupling of writes from reads.

Summary

A scalable News Feed is a masterpiece of distributed systems engineering. It balances the enormous read-to-write ratio of social media through hybrid fan-out strategies, caches pre-built feeds in massive distributed caches, and employs asynchronous ranking pipelines to personalize content. Careful handling of celebrity users, cursor-based pagination, and robust failure handling ensure the system remains fast and reliable even under peak load. The News Feed is not just a feature; it is the central nervous system of modern social platforms.

Further Reading

• Design a Social Network
• Design a Recommendation System
• Design a Timeline
• Design a Chat System
• Caching Explained
• Database Sharding Explained
• CQRS Pattern Explained
• Event-Driven Architecture
• Scalability Explained