Skip to main content

Design a Recommendation System

Recommendation systems are the engines behind the content we consume daily. YouTube’s next video, Netflix’s homepage, TikTok’s For You feed, Instagram’s Explore page, LinkedIn’s job suggestions, and Amazon’s “Customers who bought this also bought” are all powered by sophisticated recommendation pipelines. Building such a system at scale requires not just machine learning models but a robust distributed architecture that can generate personalized results from millions or billions of items in under 200 milliseconds. This article explores the full architecture, from user behavior collection to online serving, covering the key design decisions and trade-offs.

Step 1: Requirements Clarification​

Functional Requirements​

  • Personalized recommendations – Users receive items (videos, posts, products, people) tailored to their interests.
  • Home feed / explore – A continuous, ranked stream of recommended content.
  • Related content – “Similar items” or “more like this” suggestions.
  • People / friend recommendations – Connect users with others they may know or want to follow.
  • Trending content – Surface currently popular items, often with a locality or topic context.
  • Cold-start recommendations – Provide reasonable suggestions for new users or newly published content with no interaction history.
  • Real-time recommendations – Reflect recent user activity (e.g., after watching a video, the next recommendations adapt).
  • Multi-device support – Consistent experience across phone, web, and TV.

Non-Functional Requirements​

  • Low latency – Recommendations must load with the page or feed, ideally < 200ms (P95).
  • High throughput – Handle millions of recommendation requests per second at peak.
  • Horizontal scalability – Grow with the user base and catalog size.
  • High availability – The recommendation service is critical; fallback mechanisms (e.g., pre-computed static recommendations) are needed.
  • Personalization – Results must be relevant to the individual user.
  • Fault tolerance – Failures in the ML pipeline should not cause complete loss of recommendations.
  • Freshness – New items and recent user actions should quickly influence recommendations.
  • Explainability (optional) – In some contexts (e.g., why a product is recommended), the system may provide reasons.

Step 2: Capacity Estimation​

Assume a large consumer platform:

  • Total users: 1 billion
  • Daily active users (DAU): 300 million
  • Recommendation requests per second: average 50,000; peak 300,000
  • Candidate pool size: 100 million active items (videos, posts, products)
  • Average number of recommendations returned: 20
  • User action events ingested: 500,000 events/sec (views, likes, clicks)
  • Feature updates per second: 100,000 writes to the online feature store
  • Storage: feature store holding profiles and item metadata – tens of terabytes

Latency targets: P50 < 100ms, P95 < 200ms, P99 < 500ms for the entire recommendation pipeline.

Step 3: API Design​

Recommendation services expose a simple API for clients.

  • Get Recommendations
    GET /api/v1/recommendations?type=feed&limit=20&cursor={token}
    Returns { items: [ItemObject...], next_cursor }

  • Get Similar Items
    GET /api/v1/similar?item_id={id}&limit=10

  • Get Related Users
    GET /api/v1/connections?user_id={id}&limit=10

  • Record User Action (internal, via event stream)
    POST /internal/events with payload { user_id, item_id, action, timestamp }

  • Refresh Recommendations
    GET /api/v1/recommendations?type=feed&since={last_item_id}

Responses include item metadata (title, image URL, creator) so the client can render without additional calls. Idempotency is not critical for reads but the event recording API uses idempotency keys to avoid double-counting.

Step 4: High-Level Architecture​

The system is composed of offline (model training, feature engineering) and online (serving, real-time inference) components.

  • Recommendation Service orchestrates the request: candidate retrieval, ranking, post-processing.
  • Candidate Generation reduces the pool from millions to hundreds.
  • Ranking Service scores each candidate using ML models.
  • Feature Store provides real-time user, item, and context features.
  • Model Serving hosts trained models for low-latency inference.
  • Event Collection & Stream Processing ingest user actions and update features and feedback loops.
  • Offline Training Pipeline uses historical data to train and evaluate models.
  • Prediction Cache stores recently computed recommendations to reduce load.

Step 5: User Behavior Collection​

User interactions are the fuel for recommendations. These are collected as events:

  • Implicit feedback: clicks, views, watch time, dwell time, scroll depth.
  • Explicit feedback: likes, ratings, shares, purchases, follows.

Events are sent from clients to the Event Collection Service (via HTTP or a dedicated SDK) and then published to a distributed log like Apache Kafka. From there, a Stream Processing layer (e.g., Apache Flink, Spark Streaming) computes real-time aggregates (e.g., a user’s last N viewed items, a content’s recent popularity) and writes them to the online Feature Store. This data also lands in a data lake for offline training.

Step 6: Candidate Generation​

Candidate generation retrieves a manageable subset (hundreds) of items from the full catalog (millions or billions). This must be fast and computationally light.

Common retrieval strategies:

StrategyDescriptionProsCons
Collaborative Filtering“Users who liked X also liked Y.” Compute item-item or user-user similarity matrices.Simple, effective for established items.Cold start for new items/users; computationally heavy to compute for all pairs.
Content-BasedMatch item attributes (genre, tags, embeddings) to user profiles.Works for new items with metadata.Requires rich content features; may be less serendipitous.
Embedding SimilarityUse two-tower neural network embeddings: one tower for users, one for items. At serving time, find items with nearest neighbor embeddings to the user vector.State-of-the-art for large catalogs; efficient with approximate nearest neighbor (ANN) indexes like FAISS.Requires training the two-tower model; embeddings may drift.
Graph TraversalExplore the user-item bipartite graph; use random walks (e.g., Node2Vec) or graph neural networks to generate candidates.Captures complex relationships.Complex to scale.
Trending / PopularServe currently popular items, often localized.Simple, handles cold start for new users.Not personalized.

Modern large-scale systems usually combine multiple candidate sources. For example, a user’s feed might consist of 50% two-tower embedding candidates, 20% collaborative filtering, 15% trending, and 15% content-based. The Candidate Generation Service merges and deduplicates these streams into a single pool of a few hundred candidates for ranking.

Step 7: Ranking Pipeline​

After candidate generation, the Ranking Service scores each candidate with a more sophisticated model to produce the final order. The pipeline often has multiple stages:

  1. Filtering: Remove items the user has already seen, items blocked by moderation, or items violating business rules.
  2. Lightweight Scoring: A simple model (e.g., logistic regression) quickly prunes the list from hundreds to maybe 50 candidates, using lightweight features.
  3. Heavy Scoring: A deep learning model (e.g., DNN, Wide & Deep, DLRM) scores the remaining candidates using hundreds of user and item features. This is the most latency-sensitive step.
  4. Business Rules & Re-ranking: Apply diversity rules (e.g., no two videos from the same creator in a row), boost fresh content, insert advertisements, and ensure fairness.
  5. Final Sort: Sort by final score and return the top K items.

To meet latency targets, heavy scoring must be extremely optimized—batched inference, model quantization, GPU acceleration, or pre-computed scoring where possible.

Step 8: Feature Store​

A Feature Store is an essential component of production ML systems. It bridges offline training and online serving by providing consistent features.

  • Online Feature Store: Low-latency (millisecond) key-value store (e.g., Redis, DynamoDB) serving features for real-time inference. Stores user features (last 10 watched videos, aggregated stats), item features (popularity, category), and context features (time of day, device).
  • Offline Feature Store: Data warehouse/lake (e.g., Hive, BigQuery) storing historical features for model training. The offline store is used to generate training datasets with point-in-time correctness.
  • Feature Engineering: Transformations computed via stream processing (for real-time features) and batch pipelines (for historical/aggregate features).
  • Consistency: Online and offline stores must use the same feature definitions to avoid training-serving skew.

A dedicated Feature Store service simplifies retrieval: the Ranking Service requests “user_id=123, features=[...” and gets a vector back in under 1ms.

Step 9: Online Inference​

Model Serving hosts the trained ranking models and provides a low-latency inference API. Common patterns:

  • Dedicated model servers: TensorFlow Serving, Triton Inference Server, or Seldon Core. Models are loaded from the Model Registry.
  • Batched inference: The server can internally batch prediction requests from different users to maximize GPU utilization.
  • Model versioning: Multiple model versions can be served simultaneously; an experiment framework can split traffic (A/B test) to evaluate new models.
  • Prediction caching: For non-personalized or semi-static recommendations (e.g., “similar items” for a popular product), predictions can be pre-computed and stored in a cache. Real-time personalized feed items typically cannot be cached due to uniqueness.

Latency from ranking inference should be tightly controlled; often the model is compressed (quantization, pruning) or executed on specialized hardware.

Step 10: Feedback Loop​

Recommendation systems are never static. They must learn from user reactions to improve.

  • Implicit feedback loop: User actions (clicks, watch time) are streamed back, aggregated, and used to update real-time features (e.g., “user clicked category X 3 times today”). These updated features influence future recommendations immediately.
  • Explicit feedback: Less frequent but high-signal.
  • Online learning: Some systems update model parameters incrementally from the stream, adapting to trends (e.g., a breaking news event). More commonly, models are retrained in batches daily or hourly using recent data.
  • Retraining pipeline: Offline pipelines periodically train new models on fresh data, evaluate them against key metrics (CTR, engagement), and if successful, promote them to the Model Registry for serving.
  • Model drift detection: Monitor prediction distributions; if they diverge from training, trigger retraining.

The feedback loop ensures the system gets smarter over time and adapts to shifting user interests.

Step 11: Cold Start Problem​

New users and new items have no historical interactions, making personalization difficult.

New User Cold Start

  • Serve trending or popular content initially.
  • Use context (device, location, time) to bootstrap.
  • Ask for explicit preferences during onboarding (e.g., select genres you like).
  • Use demographic similarity to existing users.

New Item Cold Start

  • Use content-based features (title, tags, creator) to match with users who liked similar items.
  • Inject the item into a small percentage of relevant feeds to gather initial engagement data (“explore”).
  • Leverage creator authority: if a popular creator posts new content, it can be shown to their followers first.

A good recommendation system gracefully handles cold start by blending personalized and non-personalized strategies.

Step 12: Scalability Strategies​

  • Horizontal scaling: All online services (Recommendation, Candidate Gen, Ranking) are stateless and scale out.
  • Feature Store sharding: User features sharded by user_id; item features sharded by item_id.
  • Candidate Cache: Popular candidate sets (e.g., trending, top-n for a genre) are cached in Redis.
  • Model Serving replicas: Load balanced across many pods/nodes; auto-scale based on prediction QPS.
  • Distributed inference: The two-tower embedding model can be split: item embeddings can be pre-computed and stored in an ANN index (like FAISS) on a separate cluster. The user embedding is computed online and the lookup is a fast nearest neighbor search.
  • Regional deployment: A global platform deploys full stacks in each region, with user data localized to meet privacy regulations and reduce latency.
  • Asynchronous pipelines: Event processing, feature engineering, and model training run on separate, scalable compute clusters (Spark, Flink).

Step 13: Reliability and Observability​

  • Quality Metrics: Click-through rate (CTR), conversion rate, watch time, user retention. Tracked per model version and per recommendation surface.
  • Latency metrics: P50, P95, P99 for the entire recommendation endpoint and each sub-component.
  • Feature freshness: Monitor the lag between event occurrence and feature store update.
  • Model health: Input feature distributions, prediction distributions. Alert on anomalies.
  • Fallback: If the ranking model times out, serve pre-ranked popular or editorial recommendations. If the candidate generator fails, fall back to a static hot list.
  • Circuit breakers: Prevent cascading failures if downstream services (Feature Store, Model Serving) degrade.

Comprehensive observability ensures that performance regressions are caught before users notice.

Step 14: Security and Privacy​

  • Authentication & Authorization: Only authenticated users can receive personalized recommendations. Users can only query their own data.
  • Data Minimization: Collect only necessary behavioral data. Provide users with controls to delete history.
  • GDPR / Privacy Compliance: User data must be deletable on request; the training pipeline must be able to remove that data’s influence (e.g., via data deletion and model retraining or federated learning).
  • Secure Feature Storage: Sensitive features encrypted at rest; access controlled.
  • Bias and Fairness: Monitor recommendations for demographic bias. Implement fairness constraints in ranking (e.g., ensure diverse content creators are represented).
  • Abuse prevention: Rate limit recommendation requests; detect fake click patterns.

Real-World Example: TikTok-like Recommendation​

TikTok’s “For You” feed is a prime example of a real-time, personalized recommendation system.

  1. The user opens the app; the Recommendation Service fetches recent user features.
  2. Candidate Generation uses a two-tower neural embedding model: the user tower runs online to produce a vector; the item tower vectors are pre-computed and indexed in an ANN service (like FAISS). The service quickly retrieves the nearest item vectors, plus trending and social graph candidates.
  3. Ranking scores each candidate with a deep model, applying business logic (e.g., prefer new uploads, ensure diversity), and returns the final list.
  4. User actions flow back via Kafka, updating the feature store in real-time (e.g., “user watched 5 videos in category X”). These features immediately influence the next request.
  5. Offline, embeddings are retrained with the latest data to capture shifts in interest.

Trade-offs​

  • Recommendation quality vs latency: More complex models improve relevance but increase inference time. Staged ranking and model compression balance this.
  • Personalization vs privacy: Highly personalized systems need detailed user data, which conflicts with privacy. On-device inference and federated learning are emerging solutions but add complexity.
  • Freshness vs cache efficiency: Real-time personalization prevents heavy caching of results; each user’s feed is unique. Caching of intermediate results (embeddings, candidate sets) is vital.
  • Complexity vs maintainability: Sophisticated multi-stage pipelines deliver better recommendations but are harder to debug and operate. Simpler models can be a pragmatic starting point.
  • Online inference vs pre-computation: Pre-computing recommendations (e.g., daily recommendation lists) reduces serving cost but lacks real-time responsiveness. Modern systems favor online inference for personalization, with pre-computation for non-personalized scenarios.

Common Mistakes​

  • Treating ranking as the only step: Skipping candidate generation and trying to score the entire catalog in real-time is impossible at scale. Generation and ranking are distinct.
  • Ignoring feature freshness: Stale features lead to poor recommendations (e.g., recommending a video the user just finished watching).
  • No feedback loop: Without capturing user actions and retraining, the model becomes stale.
  • No cold-start strategy: New users receive empty or irrelevant feeds, causing churn.
  • Large synchronous inference pipeline: Tightly coupling all steps in the request path increases latency. Use asynchronous feature updates and pre-computation where possible.
  • Tight coupling between ML and application logic: The recommendation system should be a separate service with clear APIs. Don’t embed model loading in the web server.
  • Poor monitoring: Deploying an ML system without monitoring recommendation quality metrics (CTR, conversion) is flying blind.

Interview Perspective​

Designing a recommendation system is a common advanced-level system design question. Interviewers want to see that you can think beyond simple databases and APIs into the world of ML systems and data pipelines. Expect questions like:

  • Design YouTube / TikTok / Amazon recommendations.
  • Explain candidate generation vs ranking.
  • How do you reduce latency in a recommendation system?
  • What is a feature store and why do you need it?
  • How do you handle cold start?
  • How do you scale model serving?

Demonstrate an understanding of the full data lifecycle—from user event to model prediction—and the architectural components needed to serve millions of personalized feeds.

Summary​

A production recommendation system is a sophisticated marriage of large-scale data engineering and machine learning. It ingests billions of user events, trains models offline, and serves personalized results in milliseconds using a multi-stage pipeline of candidate generation, feature retrieval, and ranking. Core infrastructure includes a real-time event stream, a feature store, a model serving platform, and robust fallback mechanisms. Balancing personalization, freshness, latency, and privacy is the hallmark of a well-designed system. By mastering these components and trade-offs, engineers can architect recommendation engines that power the next generation of intelligent, user-centric applications.

Further Reading​