Caching Part 7: In Production — Monitoring, Sizing, and Staying Alive

Adding a cache is easy. Running it in production for months without incidents is the real challenge. This post covers what to monitor, how to size it, how to keep it available, and what to do when things break.

The 5 Metrics That Matter

You don't need 20 dashboards. Focus on these five:

Metric	Why It Matters	Healthy Range	Alert Threshold
Hit Ratio	Is the cache earning its keep?	85-95%	< 70%
Latency (p99)	How fast is the cache responding?	< 2 ms	> 10 ms
Eviction Rate	Are you evicting too much?	< 5% of ops	> 20%
Memory Usage	Are you close to the limit?	60-80%	> 90%
Connected Clients	Are connections leaking?	Stable	Spiking

Cache Hit Ratio

92%

Your most important metric. If this drops, users feel it immediately.

Monitoring with Micrometer

@Configuration
class CacheMetricsConfig {

    @Bean
    fun cacheMetricsRegistrar(
        cacheManager: CacheManager,
        meterRegistry: MeterRegistry
    ) {
        cacheManager.cacheNames.forEach { cacheName ->
            val cache = cacheManager.getCache(cacheName) ?: return@forEach
            val nativeCache = cache.nativeCache

            if (nativeCache is com.github.benmanes.caffeine.cache.Cache<*, *>) {
                val stats = nativeCache.stats()
                Gauge.builder("cache.hit.rate", stats) { stats.hitRate() }
                    .tag("cache", cacheName)
                    .register(meterRegistry)

                Gauge.builder("cache.eviction.count", stats) { stats.evictionCount() }
                    .tag("cache", cacheName)
                    .register(meterRegistry)

                Gauge.builder("cache.size", nativeCache) { nativeCache.estimatedSize() }
                    .tag("cache", cacheName)
                    .register(meterRegistry)
            }
        }
    }
}

Redis Monitoring

# Essential Redis stats
redis-cli INFO stats | grep -E "keyspace_hits|keyspace_misses|evicted_keys|used_memory"
redis-cli INFO memory | grep "used_memory_human"
redis-cli SLOWLOG GET 10   # Last 10 slow operations

Capacity Planning: How Much Memory Do You Need?

A practical formula for estimating cache size:

Cache Memory = (Avg Value Size × Number of Keys × Replication Factor × Overhead)

Example Calculation

Input	Value
Cached items	500,000
Avg value size (serialized JSON)	2 KB
Redis key + metadata overhead	~100 bytes per key
Replication factor (primary + 1 replica)	2x
Safety buffer	1.3x

Raw = 500,000 × (2 KB + 0.1 KB) = ~1 GB
With replication = 1 GB × 2 = 2 GB
With buffer = 2 GB × 1.3 = 2.6 GB → round up to 4 GB

Always allocate 30% more than your estimate. Redis performance degrades when memory is above 80% utilization. Plan for growth.

High Availability

Option 1: Redis Sentinel (Automatic Failover)

Sentinel monitors your Redis instances and automatically promotes a replica if the primary fails.

# sentinel.conf
sentinel monitor mymaster 127.0.0.1 6379 2
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 10000
sentinel parallel-syncs mymaster 1

// Spring Boot Sentinel configuration
@Configuration
class RedisSentinelConfig {

    @Bean
    fun redisConnectionFactory(): RedisConnectionFactory {
        val sentinelConfig = RedisSentinelConfiguration()
            .master("mymaster")
            .sentinel("sentinel1.example.com", 26379)
            .sentinel("sentinel2.example.com", 26379)
            .sentinel("sentinel3.example.com", 26379)

        return LettuceConnectionFactory(sentinelConfig)
    }
}

Option 2: Redis Cluster (Sharding + HA)

For datasets too large for a single node. Redis Cluster shards data across multiple nodes, each handling a subset of keys.

Setup	Best For	Max Dataset Size
Single Instance	Small apps, development	Server RAM
Sentinel	Medium apps needing HA	Single server RAM
Cluster	Large datasets, high throughput	Sum of all node RAMs

Start with Sentinel. Cluster adds complexity (cross-slot operations, migrations). You only need it when your data doesn't fit in one node.

Security Considerations

Key Namespacing

Use prefixes to avoid collisions between services:

// Each service gets its own namespace
const val CACHE_PREFIX = "user-service"
fun cacheKey(id: Long) = "$CACHE_PREFIX:user:$id"

Preventing Cache Poisoning

// Validate all user input before using it as a cache key
fun getUser(username: String): User {
    val sanitized = username.lowercase().trim()
        .replace(Regex("[^a-z0-9_-]"), "")

    return cache.get("user:$sanitized") ?: repository.findByUsername(sanitized)
}

Never use raw user input as a cache key. An attacker could craft a key that overwrites critical data or causes excessive memory usage.

Graceful Degradation

When the cache is down, your app should still work — just slower. Never let a cache outage take down your entire application.

@Service
class ResilientProductService(
    private val redisTemplate: RedisTemplate<String, Product>,
    private val repository: ProductRepository
) {
    fun getProduct(id: Long): Product {
        return try {
            val cached = redisTemplate.opsForValue().get("product:$id")
            cached ?: fetchAndCache(id)
        } catch (e: RedisConnectionFailureException) {
            log.warn("Redis unavailable, serving from database")
            repository.findById(id).orElseThrow()
        }
    }

    private fun fetchAndCache(id: Long): Product {
        val product = repository.findById(id).orElseThrow()
        try {
            redisTemplate.opsForValue().set("product:$id", product, Duration.ofMinutes(10))
        } catch (e: Exception) {
            log.warn("Failed to cache product, continuing without cache")
        }
        return product
    }
}

Migration Strategy: Changing Cache Topology Without Downtime

When you need to change your cache setup (new Redis version, different instance, changed key format):

Dual-write: Write to both old and new cache
Dual-read: Read from new cache first, fall back to old
Verify: Compare hit rates between old and new
Cut over: Stop writing to old cache
Clean up: Remove old cache

Key Takeaways

Monitor 5 metrics: hit ratio, latency, eviction rate, memory, connections
Size with a formula: raw data × replication × 1.3 buffer
Sentinel for HA in most cases — Cluster only when data exceeds one node
Namespace your keys and validate user input to prevent poisoning
Graceful degradation — catch cache exceptions, serve from DB, log warnings
Migrate with dual-write/dual-read to change cache topology without downtime

The 5 Metrics That Matter

You don't need 20 dashboards. Focus on these five:

Metric	Why It Matters	Healthy Range	Alert Threshold
Hit Ratio	Is the cache earning its keep?	85-95%	< 70%
Latency (p99)	How fast is the cache responding?	< 2 ms	> 10 ms
Eviction Rate	Are you evicting too much?	< 5% of ops	> 20%
Memory Usage	Are you close to the limit?	60-80%	> 90%
Connected Clients	Are connections leaking?	Stable	Spiking

Cache Hit Ratio

92%

Your most important metric. If this drops, users feel it immediately.

Monitoring with Micrometer

@Configuration
class CacheMetricsConfig {

    @Bean
    fun cacheMetricsRegistrar(
        cacheManager: CacheManager,
        meterRegistry: MeterRegistry
    ) {
        cacheManager.cacheNames.forEach { cacheName ->
            val cache = cacheManager.getCache(cacheName) ?: return@forEach
            val nativeCache = cache.nativeCache

            if (nativeCache is com.github.benmanes.caffeine.cache.Cache<*, *>) {
                val stats = nativeCache.stats()
                Gauge.builder("cache.hit.rate", stats) { stats.hitRate() }
                    .tag("cache", cacheName)
                    .register(meterRegistry)

                Gauge.builder("cache.eviction.count", stats) { stats.evictionCount() }
                    .tag("cache", cacheName)
                    .register(meterRegistry)

                Gauge.builder("cache.size", nativeCache) { nativeCache.estimatedSize() }
                    .tag("cache", cacheName)
                    .register(meterRegistry)
            }
        }
    }
}

Redis Monitoring

# Essential Redis stats
redis-cli INFO stats | grep -E "keyspace_hits|keyspace_misses|evicted_keys|used_memory"
redis-cli INFO memory | grep "used_memory_human"
redis-cli SLOWLOG GET 10   # Last 10 slow operations

Capacity Planning: How Much Memory Do You Need?

A practical formula for estimating cache size:

Cache Memory = (Avg Value Size × Number of Keys × Replication Factor × Overhead)

Example Calculation

Input	Value
Cached items	500,000
Avg value size (serialized JSON)	2 KB
Redis key + metadata overhead	~100 bytes per key
Replication factor (primary + 1 replica)	2x
Safety buffer	1.3x

Raw = 500,000 × (2 KB + 0.1 KB) = ~1 GB
With replication = 1 GB × 2 = 2 GB
With buffer = 2 GB × 1.3 = 2.6 GB → round up to 4 GB

Always allocate 30% more than your estimate. Redis performance degrades when memory is above 80% utilization. Plan for growth.

High Availability

Option 1: Redis Sentinel (Automatic Failover)

Sentinel monitors your Redis instances and automatically promotes a replica if the primary fails.

# sentinel.conf
sentinel monitor mymaster 127.0.0.1 6379 2
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 10000
sentinel parallel-syncs mymaster 1

// Spring Boot Sentinel configuration
@Configuration
class RedisSentinelConfig {

    @Bean
    fun redisConnectionFactory(): RedisConnectionFactory {
        val sentinelConfig = RedisSentinelConfiguration()
            .master("mymaster")
            .sentinel("sentinel1.example.com", 26379)
            .sentinel("sentinel2.example.com", 26379)
            .sentinel("sentinel3.example.com", 26379)

        return LettuceConnectionFactory(sentinelConfig)
    }
}

Option 2: Redis Cluster (Sharding + HA)

For datasets too large for a single node. Redis Cluster shards data across multiple nodes, each handling a subset of keys.

Setup	Best For	Max Dataset Size
Single Instance	Small apps, development	Server RAM
Sentinel	Medium apps needing HA	Single server RAM
Cluster	Large datasets, high throughput	Sum of all node RAMs

Start with Sentinel. Cluster adds complexity (cross-slot operations, migrations). You only need it when your data doesn't fit in one node.

Security Considerations

Key Namespacing

Use prefixes to avoid collisions between services:

// Each service gets its own namespace
const val CACHE_PREFIX = "user-service"
fun cacheKey(id: Long) = "$CACHE_PREFIX:user:$id"

Preventing Cache Poisoning

// Validate all user input before using it as a cache key
fun getUser(username: String): User {
    val sanitized = username.lowercase().trim()
        .replace(Regex("[^a-z0-9_-]"), "")

    return cache.get("user:$sanitized") ?: repository.findByUsername(sanitized)
}

Never use raw user input as a cache key. An attacker could craft a key that overwrites critical data or causes excessive memory usage.

Graceful Degradation

When the cache is down, your app should still work — just slower. Never let a cache outage take down your entire application.

@Service
class ResilientProductService(
    private val redisTemplate: RedisTemplate<String, Product>,
    private val repository: ProductRepository
) {
    fun getProduct(id: Long): Product {
        return try {
            val cached = redisTemplate.opsForValue().get("product:$id")
            cached ?: fetchAndCache(id)
        } catch (e: RedisConnectionFailureException) {
            log.warn("Redis unavailable, serving from database")
            repository.findById(id).orElseThrow()
        }
    }

    private fun fetchAndCache(id: Long): Product {
        val product = repository.findById(id).orElseThrow()
        try {
            redisTemplate.opsForValue().set("product:$id", product, Duration.ofMinutes(10))
        } catch (e: Exception) {
            log.warn("Failed to cache product, continuing without cache")
        }
        return product
    }
}

Migration Strategy: Changing Cache Topology Without Downtime

When you need to change your cache setup (new Redis version, different instance, changed key format):

Dual-write: Write to both old and new cache
Dual-read: Read from new cache first, fall back to old
Verify: Compare hit rates between old and new
Cut over: Stop writing to old cache
Clean up: Remove old cache

Key Takeaways

Monitor 5 metrics: hit ratio, latency, eviction rate, memory, connections
Size with a formula: raw data × replication × 1.3 buffer
Sentinel for HA in most cases — Cluster only when data exceeds one node
Namespace your keys and validate user input to prevent poisoning
Graceful degradation — catch cache exceptions, serve from DB, log warnings
Migrate with dual-write/dual-read to change cache topology without downtime

Caching Deep Dive(9-part series)

The 5 Metrics That Matter

Monitoring with Micrometer

Redis Monitoring

Capacity Planning: How Much Memory Do You Need?

Example Calculation

High Availability

Option 1: Redis Sentinel (Automatic Failover)

Option 2: Redis Cluster (Sharding + HA)

Security Considerations

Key Namespacing

Preventing Cache Poisoning

Graceful Degradation

Migration Strategy: Changing Cache Topology Without Downtime

Key Takeaways

Tags

Related Posts

Caching Part 3: Strategies — Read-Through, Write-Through, and Beyond

Caching Part 1: Fundamentals — Why, When, and When Not to Cache

Caching Part 9: Hands-On — Code Labs and Real Implementations

Caching Deep Dive(9-part series)

The 5 Metrics That Matter

Monitoring with Micrometer

Redis Monitoring

Capacity Planning: How Much Memory Do You Need?

Example Calculation

High Availability

Option 1: Redis Sentinel (Automatic Failover)

Option 2: Redis Cluster (Sharding + HA)

Security Considerations

Key Namespacing

Preventing Cache Poisoning

Graceful Degradation

Migration Strategy: Changing Cache Topology Without Downtime

Key Takeaways

Tags

Related Posts

Caching Part 3: Strategies — Read-Through, Write-Through, and Beyond

Caching Part 1: Fundamentals — Why, When, and When Not to Cache

Caching Part 9: Hands-On — Code Labs and Real Implementations