Design a Distributed Cache
Modern internet applications serve millions of users with sub‑second response times. A single relational database cannot sustain this load, no matter how well‑tuned. Distributed caching is the primary technique for reducing database load, cutting latency, and enabling horizontal scalability. Systems like Redis, Redis Cluster, Memcached, Amazon ElastiCache, and Azure Cache for Redis are the workhorses behind product catalogs, session stores, leaderboards, shopping carts, and authentication tokens. In this article, we design a production‑grade distributed cache from first principles, covering partitioning, replication, consistency, eviction, and the trade‑offs that keep caches fast and reliable.
Step 1: Requirements Clarification
Functional Requirements
- Read cached objects – Retrieve a value by key with minimal latency.
- Write cached objects – Store a key‑value pair, optionally with a TTL.
- Delete cache entries – Explicitly remove a key.
- Automatic expiration (TTL) – Keys are evicted after a specified time‑to‑live.
- Cache replication – Data is copied across multiple nodes for fault tolerance.
- Cache partitioning – Data is spread across many nodes to scale memory and throughput.
- Cache statistics – Expose hit ratio, memory usage, evictions, and latency metrics.
- Multi‑key operations – Efficiently fetch or update multiple keys in one operation.
- Pub/Sub (optional) – Notify clients about key changes in real time.
Non-Functional Requirements
- Low latency – Reads and writes must complete in under 1 ms for in‑memory operations.
- High throughput – Support millions of operations per second across the cluster.
- Horizontal scalability – Add nodes to increase capacity without downtime.
- High availability – The cache remains operational even if individual nodes fail.
- Fault tolerance – Data loss is minimized through replication and persistence.
- High cache hit ratio – Most requests should be served from cache, not the backing store.
- Fast recovery – A failed node should be replaced and repopulated quickly.
- Operational simplicity – Operations teams can manage the cluster with standard tooling.
Step 2: Capacity Estimation
Assume a large e‑commerce platform:
- Total cacheable objects (product pages, user sessions, cart data): 10 billion keys.
- Average object size: 1 KB.
- Total cache size (raw): 10 TB. With replication factor 2, ~20 TB.
- Requests per second (steady): 2 million.
- Peak QPS (flash sale): 10 million.
- Read/write ratio: 10:1 (reads dominate).
- Network bandwidth per node: 10 Gbps.
Cache workloads are read‑heavy because the primary purpose is to absorb read traffic. The write path often flows to the database first, then the cache is either updated or invalidated.
Step 3: API Design
A cache provides a simple key‑value interface, often as a network protocol (Redis RESP) or REST/gRPC for managed services.
- GET
key– Returns the value or null if not present. - SET
key value [EX seconds] [NX|XX]– Stores a value with optional TTL and conditional flags. - DELETE
key– Removes the key immediately. - EXPIRE
key seconds– Sets or updates the TTL. - INCR
key– Atomically increments an integer value. - MGET
key1 key2 …– Retrieves multiple keys in one round trip. - MSET
key1 value1 key2 value2 …– Sets multiple keys.
TTL semantics are critical. After expiration, the key is lazily evicted or removed during periodic scans. Idempotency is natural for SET (overwrites), and conditional flags (NX – set only if not exists) prevent race conditions.
Step 4: High-Level Architecture
A distributed cache cluster consists of multiple nodes, with client libraries that handle routing and failover.
- Cache Client SDK embeds the logic to connect, partition requests, and handle failures.
- Client‑side Router uses consistent hashing to map keys to nodes. On topology changes, it updates the mapping.
- Cache Nodes store data in RAM, optionally persisting to disk (RDB/AOF in Redis).
- Replicas provide read scalability and failover.
- Metadata Service (or gossip protocol) distributes cluster topology: which nodes are alive, who owns which hash slots.
- Primary Database is the source of truth; the cache is a read‑through/write‑behind layer.
Step 5: Cache Data Partitioning
To scale beyond a single node’s memory and throughput, data must be partitioned (sharded).
| Strategy | Description | Pros | Cons |
|---|---|---|---|
| Hash Partitioning | hash(key) % N | Simple, even distribution. | Massive redistribution when N changes. |
| Consistent Hashing | Nodes placed on a hash ring; each key assigned to the next node clockwise. | Minimal movement when adding/removing nodes. | With few nodes, load may be uneven. |
| Virtual Nodes | Each physical node owns several virtual nodes on the ring. | Improves load balance with consistent hashing. | Slightly more metadata. |
| Range Partitioning | Keys sorted and divided into ranges (e.g., by user ID prefix). | Supports range queries. | Hotspots if one range is accessed more. |
Consistent hashing with virtual nodes is the foundation of Redis Cluster and many distributed caches. When a node joins or leaves, only a fraction of keys need to be moved. The client library recalculates the owner for each key and migrates data.
Hot keys – a small number of keys that receive a disproportionate share of traffic (e.g., a viral product) – can overload a single shard. Mitigations include local caching within the application, key replication, or using a dedicated hot‑key cache.
Step 6: Cache Consistency
The cache holds a copy of data that exists in the primary database. Maintaining consistency between them is a classic challenge.
| Pattern | How it works | Pros | Cons |
|---|---|---|---|
| Cache Aside | Application checks cache first; on miss, loads from DB and writes to cache. | Simple, application‑controlled. | Stale data possible if DB is updated without invalidating cache. |
| Read Through | Cache itself loads from DB on miss (transparent to app). | App code simpler. | Cache must know about DB schema. |
| Write Through | Application writes to cache, which synchronously writes to DB. | Data is consistent. | Higher write latency. |
| Write Behind | Application writes to cache, which asynchronously writes to DB. | Low write latency. | Risk of data loss if cache fails before DB write. |
| Refresh Ahead | Cache proactively refreshes hot keys before they expire. | Reduces latency for popular items. | More complex, may load unnecessary data. |
Cache Aside is the most common pattern. The application writes to the database and then invalidates the corresponding cache key. On the next read, a cache miss triggers a load from the database. This pattern tolerates temporary inconsistency but requires careful handling of race conditions (e.g., using a short TTL or distributed locks during updates).
Step 7: Cache Invalidation
Cache invalidation is notoriously difficult. Stale cache data leads to confusing user experiences and business‑logic errors.
Invalidation strategies:
- TTL expiration – Keys expire automatically after a set time. Simple, but may serve stale data until expiration.
- Explicit invalidation – When the source of truth changes, the application explicitly deletes or updates the cached key.
- Event‑driven invalidation – Database change events (CDC from MySQL binlog, PostgreSQL logical replication) are published to a message queue. A cache invalidation worker consumes these events and purges affected keys.
- Version‑based invalidation – Each cache entry includes a version number. When the underlying data changes, the version is incremented. The cache serves only if the client’s requested version matches.
- Lazy expiration – When a key is accessed after its TTL, it is deleted and reloaded. Redis uses a combination of lazy expiration and periodic eviction.
This event‑driven approach decouples the application from cache management and ensures near‑real‑time invalidation.
Step 8: Eviction Policies
When memory is full, the cache must evict existing keys to make room for new ones. The policy directly affects the hit ratio.
| Policy | Behavior | Best for |
|---|---|---|
| LRU (Least Recently Used) | Evicts keys that haven’t been accessed for the longest time. | General‑purpose, good hit ratio. |
| LFU (Least Frequently Used) | Evicts keys with the fewest accesses. | Scenarios with clear hot/cold data. |
| FIFO (First In, First Out) | Evicts the oldest inserted keys. | Simple, predictable, but lower hit ratio. |
| Random | Randomly chooses keys to evict. | Surprisingly effective for uniform access; low CPU overhead. |
| TTL Eviction | Evicts keys whose TTL has expired (often combined with other policies). | All caches. |
Redis, for example, offers volatile‑lru, allkeys‑lru, volatile‑lfu, and more. The right policy depends on the access pattern. A product catalog cache benefits from LRU or LFU; a rate‑limiting counter cache might use TTL alone.
Step 9: Replication and High Availability
To survive node failures, the cache replicates data. Common topologies:
- Primary‑Replica – One primary accepts writes and replicates to one or more read‑only replicas. If the primary fails, a replica is promoted.
- Active‑Active – Multiple nodes accept reads and writes; conflict resolution is required (e.g., last‑write‑wins, CRDTs). Rare in traditional caches.
- Automatic failover – Redis Sentinel or Redis Cluster’s built‑in failover detect primary failure and promote a replica.
- Leader election – Uses consensus (Raft) or a sentinel quorum to decide the new primary.
- Split‑brain – When a network partition occurs, two nodes might both believe they are primary. Redis Sentinel requires a majority to elect a new primary, preventing split‑brain with a proper quorum configuration.
Data recovery after a failure may involve replaying an append‑only file (AOF) or snapshot (RDB). Replication lag can cause replicas to serve slightly stale data, which is acceptable for most caching use cases.
Step 10: Scalability Strategies
- Horizontal scaling – Add more shards. Consistent hashing ensures only a fraction of keys are remapped. Redis Cluster supports up to 16384 hash slots distributed across nodes.
- Cluster expansion – New nodes join the cluster; hash slots are migrated incrementally with no downtime.
- Auto‑scaling – In cloud environments, metrics like memory usage and CPU trigger adding or removing nodes.
- Multi‑region deployment – Active‑active geo‑replication (e.g., Redis Enterprise) serves users from the nearest region, with asynchronous replication to other regions.
- Edge caching – For static or rarely changing data, CDN edge nodes or local app‑instance caches (Caffeine, Ehcache) absorb even more read traffic.
To reach millions of requests per second, the cluster is partitioned into tens or hundreds of shards, each handling a subset of keys. Client libraries use connection pooling and pipelining to maximize throughput.
Step 11: Reliability and Fault Tolerance
Caches sit in the critical path. When they fail, the system must degrade gracefully.
- Retry – Clients retry on transient network errors with exponential backoff.
- Timeout – Set a fast timeout (e.g., 50 ms) on cache operations so a slow node doesn’t block application threads.
- Circuit Breaker – If a cache node continuously fails, the client opens the circuit and stops sending requests for a cooling period, falling back to the database.
- Cache stampede – When a hot key expires, many concurrent requests simultaneously hit the database. Prevented by probabilistic early expiration or mutual exclusion (only one request rebuilds the cache, others wait).
- Cache avalanche – Massive number of keys expire at the same time, causing a load spike. Mitigated by adding a random jitter to TTLs.
- Cache penetration – Requests for non‑existent keys (e.g., negative queries) always bypass the cache and hit the database. Solved by caching a null value or using a Bloom filter.
- Hot key mitigation – Replicate the hot key across multiple nodes or cache it locally.
This pattern prevents hundreds of identical DB queries and keeps latency low.
Step 12: Monitoring and Observability
To operate a distributed cache effectively, you need deep visibility.
- Cache hit ratio – Percentage of requests served from cache. A drop indicates a problem (e.g., invalidation storm, eviction).
- Miss ratio – Complement of hit ratio. Correlated with DB load.
- Latency – P50, P95, P99 of cache operations. Spikes may indicate network issues or slow commands.
- Memory usage – Per‑node and total. High usage triggers evictions; memory‑exhausted nodes may reject writes.
- Evictions – Rate of keys evicted. A sudden increase means the cache is undersized or TTLs are too long.
- Slow commands – Operations that take longer than a threshold (e.g.,
KEYS *, largeMGET). - Replication lag – Delay between primary write and replica visibility. High lag means stale reads.
- Node health – Up/down status, CPU, network throughput.
Dashboards (Grafana) and alerts are essential. For example, alert if cache hit ratio drops below 90% or memory usage exceeds 80%.
Step 13: Security Best Practices
- Authentication – Redis supports
AUTHwith a strong password. Cloud services offer IAM‑based auth. - Authorization – Many managed caches implement RBAC to restrict commands per user (e.g., read‑only users for replicas).
- TLS encryption – Encrypt client‑server and inter‑node communication.
- Network isolation – Deploy cache in private subnets; use security groups or firewall rules to restrict access.
- Access control – Only application servers should access the cache. Never expose it to the public internet.
- Secret management – Store passwords in a secrets manager (Vault, AWS Secrets Manager), not in configuration files.
- Encryption at rest – Enable disk encryption for persistence files.
- Audit logging – Log all administrative commands and access attempts.
Real-World Example: E‑Commerce Distributed Cache
Consider an e‑commerce platform with services: product catalog, user sessions, shopping cart, and inventory. The cache serves multiple roles.
- Product catalog – Read‑heavy. The application uses Cache Aside with a 1‑hour TTL. When a product is updated (e.g., price change), the inventory service publishes an event that triggers cache invalidation.
- User sessions – Stored entirely in the distributed cache with a TTL equal to session timeout. No database load for session reads.
- Shopping cart – Stored in the cache as a Redis Set (
cart:user:{id}). Persistent carts are periodically written to the database asynchronously (Write Behind). - Inventory – Cached with a short TTL (10 seconds) to reflect near‑real‑time stock levels. On a successful order, the inventory service explicitly updates or deletes the cache key.
During flash sales, the cache absorbs the massive read load for product pages. Write operations (orders) still hit the database, but they are relatively fewer. The cache hit ratio stays above 95%, protecting the database from overload.
Trade-offs
- Memory vs. cost – RAM is expensive. Storing everything in cache provides the best performance but at high cost. Selective caching (only hot data) reduces memory footprint.
- Cache size vs. hit ratio – Larger caches yield higher hit ratios, but the improvement diminishes. A 10% size increase might only gain 1% hit ratio.
- Consistency vs. performance – Strong consistency (Write Through) slows writes and adds complexity. Most systems accept eventual consistency between cache and DB.
- Replication vs. latency – Synchronous replication ensures no data loss on primary failure but adds write latency. Asynchronous replication is faster but may lose recent writes.
- Simplicity vs. flexibility – A simple key‑value model (Memcached) is easy to operate. Rich data structures (Redis) add flexibility but also operational complexity.
Common Mistakes
- Caching everything – Not all data benefits from caching. Caching rapidly changing data leads to frequent invalidation and low hit ratios.
- Ignoring cache invalidation – Leaving stale data in cache causes wrong prices, inventory levels, or user info. Always have an invalidation plan.
- Poor TTL selection – Too short a TTL increases database load; too long serves stale data. Align TTLs with data volatility.
- No replication – A single cache node is a single point of failure. Losing the cache can bring down the entire application.
- Ignoring hot keys – One viral product can saturate a single shard. Plan for hot key mitigation from the start.
- Cache stampede – High‑traffic applications must protect against simultaneous cache misses for the same key.
- Tight coupling with application logic – Cache logic (keys, TTLs, invalidation) scattered across many services becomes impossible to manage. Abstract caching behind a service layer.
Interview Perspective
Expect system design questions that test your understanding of distributed caching:
- Design a distributed cache like Redis Cluster.
- What is consistent hashing? How does it help scaling?
- Cache Aside vs. Write Through?
- How do you handle cache invalidation?
- How do you prevent cache stampede?
- How do you handle node failures?
- How would you scale Redis to 1 million QPS?
Show that you can reason about partitioning, replication, consistency trade‑offs, and failure modes. Mention real‑world techniques like consistent hashing, event‑driven invalidation, and stampede prevention.
Summary
A distributed cache is a foundational infrastructure service that enables modern applications to serve high traffic with low latency. By partitioning data across a cluster using consistent hashing, replicating for high availability, and employing well‑chosen eviction policies, the cache absorbs the bulk of read traffic. Careful attention to consistency patterns (Cache Aside, Write Behind) and invalidation strategies (TTL, event‑driven) keeps data fresh enough for business needs. Robust fault tolerance — circuit breakers, stampede prevention, and graceful fallback to the database — ensures the system remains available even when the cache falters. Mastery of distributed caching is essential for any engineer designing scalable, resilient systems.