Caching Part 6: When It Goes Wrong — Thundering Herds, Hot Keys, and Nightmares
Caching Deep Dive(9-part series)
- P4Caching Part 4: Eviction Policies — LRU, LFU, and Making Room
- P5Caching Part 5: Invalidation — The Hard Problem Nobody Solves Well
- P6Caching Part 6: When It Goes Wrong — Thundering Herds, Hot Keys, and Nightmares(here)
- P7Caching Part 7: In Production — Monitoring, Sizing, and Staying Alive
- P8Caching Part 8: System Design Interviews — Patterns, Trade-offs, and Model Answers
Caching works perfectly in development. Then you push to production, and at 10x traffic, things break in ways you never expected. This post covers the 5 most common production caching failures — and how to fix each one.
1. Thundering Herd (Cache Stampede)
What happens: A popular cache key expires. Hundreds of simultaneous requests all see a cache miss. They all hit the database at the same time.
The Fix — Distributed Locking:
Only one request rebuilds the cache. Everyone else waits (or gets a stale value).
class StampedeProtectedCache(
private val redisTemplate: RedisTemplate<String, Any>,
private val repository: ProductRepository
) {
fun getProduct(id: Long): Product {
val cacheKey = "product:$id"
val lockKey = "lock:$cacheKey"
// Try to get from cache
val cached = redisTemplate.opsForValue().get(cacheKey)
if (cached != null) return cached as Product
// Try to acquire lock (SETNX with expiry)
val locked = redisTemplate.opsForValue()
.setIfAbsent(lockKey, "1", Duration.ofSeconds(10))
if (locked == true) {
// I won the lock — I fetch and cache
try {
val product = repository.findById(id).orElseThrow()
redisTemplate.opsForValue().set(cacheKey, product, Duration.ofMinutes(10))
return product
} finally {
redisTemplate.delete(lockKey)
}
} else {
// Someone else is fetching — wait and retry
Thread.sleep(100)
return getProduct(id) // Retry
}
}
}Alternative: Use Caffeine's AsyncCache which handles this automatically — only one CompletableFuture runs per key.
2. Cache Penetration
What happens: Queries for data that doesn't exist bypass the cache every time. No cache entry, no DB result. Each request hits the database.
Example: Someone scans your API with random IDs: /api/products/99999, /api/products/99998, etc.
The Fix — Bloom Filter + Null Caching:
class PenetrationProtectedCache(
private val redisTemplate: RedisTemplate<String, Any>,
private val bloomFilter: BloomFilter<Long>
) {
fun getProduct(id: Long): Product? {
// Step 1: Check bloom filter — does this ID even exist?
if (!bloomFilter.mightContain(id)) {
return null // Definitely doesn't exist, skip DB entirely
}
val cacheKey = "product:$id"
// Step 2: Check cache (including null markers)
val cached = redisTemplate.opsForValue().get(cacheKey)
if (cached == NULL_MARKER) return null // Cached as "doesn't exist"
if (cached != null) return cached as Product
// Step 3: Fetch from DB
val product = repository.findById(id).orElse(null)
if (product == null) {
// Cache the absence — prevents repeated DB hits
redisTemplate.opsForValue().set(cacheKey, NULL_MARKER, Duration.ofMinutes(5))
return null
}
redisTemplate.opsForValue().set(cacheKey, product, Duration.ofMinutes(10))
return product
}
companion object {
private const val NULL_MARKER = "NULL"
}
}Bloom filters are probabilistic — they might say "yes, this might exist" when it doesn't. But they never say "no" when it does exist. This means zero false negatives, minimal false positives.
3. Hot Key Problem
What happens: One key gets disproportionate traffic (e.g., a trending product, a celebrity's profile). The Redis node holding that key becomes a bottleneck.
The Fix — Local Cache + Distributed Cache Layering:
@Service
class HotKeyProtectedService(
private val redisTemplate: RedisTemplate<String, Product>,
private val repository: ProductRepository
) {
// L1: Local in-process cache (handles hot keys without hitting Redis)
private val localCache: Cache<Long, Product> = Caffeine.newBuilder()
.maximumSize(1_000)
.expireAfterWrite(Duration.ofSeconds(30)) // Short TTL for freshness
.build()
fun getProduct(id: Long): Product {
// L1: Check local cache first
val local = localCache.getIfPresent(id)
if (local != null) return local
// L2: Check Redis
val cached = redisTemplate.opsForValue().get("product:$id")
if (cached != null) {
localCache.put(id, cached)
return cached
}
// L3: Fetch from DB, populate both layers
val product = repository.findById(id).orElseThrow()
localCache.put(id, product)
redisTemplate.opsForValue().set("product:$id", product, Duration.ofMinutes(10))
return product
}
}Caffeine L1 cache is 1000x faster than Redis for hot keys.
4. Big Key Problem
What happens: A single Redis key holds a very large value (e.g., a 10MB JSON list, a huge hash). Operations on it block Redis's single thread.
The Fix — Break up big keys:
// BAD: One big key
redisTemplate.opsForValue().set("feed:user:123", hugeListOf10000Items)
// GOOD: Paginated keys
redisTemplate.opsForValue().set("feed:user:123:page:1", first100Items)
redisTemplate.opsForValue().set("feed:user:123:page:2", next100Items)// BAD: One huge hash
redisTemplate.opsForHash<String, String>().putAll("sessions", allSessionsMap)
// GOOD: Individual keys with a prefix
allSessions.forEach { (key, value) ->
redisTemplate.opsForValue().set("session:$key", value)
}Rule of thumb: Keep individual Redis values under 1 MB. If you need more, split into multiple keys or use a different data structure.
5. Cache Outage — When Redis Goes Down
What happens: Redis becomes unreachable. Every request becomes a cache miss. Your database gets hammered.
The Fix — Circuit Breaker + Stale Fallback:
@Service
class ResilientCacheService(
private val redisTemplate: RedisTemplate<String, Product>,
private val repository: ProductRepository
) {
private val circuitBreaker = CircuitBreaker.ofDefaults("redis")
fun getProduct(id: Long): Product {
return Try.ofSupplier(
CircuitBreaker.decorateSupplier(circuitBreaker) {
val cached = redisTemplate.opsForValue().get("product:$id")
if (cached != null) cached
else {
val product = repository.findById(id).orElseThrow()
redisTemplate.opsForValue().set("product:$id", product, Duration.ofMinutes(10))
product
}
}
)
.recover { _ ->
// Circuit is open — go straight to DB
repository.findById(id).orElseThrow()
}
.get()
}
}With Resilience4j annotations:
@CircuitBreaker(name = "redis", fallbackMethod = "getProductFallback")
fun getProduct(id: Long): Product { /* ... */ }
fun getProductFallback(id: Long, e: Exception): Product {
log.warn("Redis unavailable, fetching from DB directly")
return repository.findById(id).orElseThrow()
}Quick Reference
| Problem | Symptom | Fix |
|---|---|---|
| Thundering Herd | DB spike after key expiry | Distributed lock (SETNX) |
| Cache Penetration | DB load from nonexistent keys | Bloom filter + null caching |
| Hot Key | One Redis node overwhelmed | L1 local cache (Caffeine) |
| Big Key | Redis blocks on large values | Split into smaller keys |
| Cache Outage | DB overwhelmed when Redis down | Circuit breaker + fallback |
Key Takeaways
- Thundering herds — use distributed locks so only one request rebuilds the cache
- Cache penetration — bloom filters for "does this exist?", null caching for "it doesn't"
- Hot keys — add a local L1 cache (Caffeine) in front of Redis
- Big keys — keep Redis values under 1MB, split large data across keys
- Cache outages — circuit breakers protect your database when cache is down
- All of these are production-only problems — you won't see them in dev or at low traffic
Tags
Related Posts
Caching Part 4: Eviction Policies — LRU, LFU, and Making Room
Your cache can't hold everything. Learn how LRU, LFU, and Redis' 8 eviction policies work — and how to pick the right one.
Caching Part 1: Fundamentals — Why, When, and When Not to Cache
The complete guide to caching fundamentals — hit ratios, eviction basics, and the real question: when is caching actually a bad idea?
Caching Part 2: The Landscape — Where to Put Your Cache
From browser to database — every layer of your stack has caching opportunities. Learn where each one fits and how they work together.