Redis is using a lot of memory. Is that this problem?

Not by itself. A key with no expiry makes Redis grow. A stampede is many rebuilds of the same key at the moment of a miss. Write the key pattern before you buy memory.

How is this different from a process that grows until the server kills it?

That process keeps memory after the work is done. A stampede spikes for the expiry window and then goes quiet. Buying RAM for the first does not collapse fifty rebuilds into one.

Our deploy clears the cache. Is that fine if the box is large?

A flush of the hot set stampedes any size you rent. Plan the herd, or stop deleting those keys, or make the rebuild single-flight. Hardware does not remove a flush.

Will a queue stop the herd?

Only if one job rebuilds the key. Fifty jobs that all rebuild the same key are still a herd. They have only moved into the worker. The lock still needs a short TTL so a dead worker cannot hold it forever.

Can we learn the ratio on the live site?

You can, and you will learn it while people are waiting. A copy with realistic volume is enough. An empty database makes the rebuild look cheap, so the herd looks harmless.

Redis cache stampede: what to check before you scale the box

A page can stay fast until one Redis key expires. Then many requests miss at once, and the database and CPU jump. A larger box delays the pain. It does not turn fifty rebuilds of one key into one rebuild.

You can usually name this incident before you open a hardware pricing page. Traffic is normal. Redis is available. A hot key reaches its TTL, or a deploy deletes a group of keys on purpose. For a short window, every request misses. Each miss runs the expensive path: a heavy query, a report, a render that calls three services. The database queue grows. App CPU rises. Someone says the server is too small. The next expiry repeats the same pattern on a larger machine, because the machine was never the real issue.

This is a cache stampede, also called a thundering herd. One miss is cheap if one request rebuilds the value and the others wait or keep serving the previous value. Fifty misses mean fifty rebuilds of the same key. More Redis RAM does not merge those fifty into one. More app CPU gives each rebuild a bit more room and still runs all of them. Both purchases may be valid later. Before you watch one expiry, they are still guesses.

This is not the case where one process keeps growing all day until the server kills it. That pattern is different, and it belongs in what to check before you buy RAM for a growing process. A stampede can look like a memory or CPU emergency for one minute and then stay quiet until the next expiry. If you only look at the peak, you will buy hardware for a timer.

1. Name the key, the TTL, and what a miss actually runs

Write the key pattern, not just “the cache.” A key per page, per account, per report, per user. Write the TTL in seconds, and write what sets it. A fixed TTL that makes many keys expire in the same minute is a herd you scheduled yourself. A deploy that flushes keys together is the same. A key with no TTL is a different bug: Redis grows because nothing expires. Do not mix that growth with a stampede. Growth without expiry is a retention problem. A stampede is duplicated rebuild work at the moment of a miss.

Then write down the work that runs on a miss. Open the function behind the miss and name the query or external call in one sentence. If you cannot name it, you are not ready to scale the box, because you do not know what the box would repeat. If that work contains a slow statement, read it as a query first. The method for grouping a slow log before you split a service is in reading the slow log before a refactor. A stampede multiplies a slow statement. It does not remove the need to inspect that statement. It also does not prove that your service boundary is wrong.

The key pattern, the TTL, and who deletes the key.
Whether many keys expire in the same minute, or a deploy flushes them together.
The query or call a single miss runs. One sentence.
A note that Redis memory growth without a TTL is a separate check.

2. Watch one expiry instead of the average chart

Averages hide this problem. The p50 can stay healthy because most minutes are cache hits. The spike lives in the minute when the key expires. Pick one hot key on a copy with realistic volume. Let it expire, or delete it once, while you record three numbers: how many requests missed, how many rebuilds ran, and how long one rebuild took. You need those three numbers for one expiry window, not a week of dashboard screenshots.

If one miss window produces one rebuild, you do not have a stampede yet. You have a slow rebuild, and the next question is the query or call inside it. If one miss window produces many rebuilds of the same key, you have the herd. Scaling the box before you see that ratio is how teams pay for duplicated work. The copy matters. An empty database makes the rebuild look cheap, so the herd looks harmless. The live site is a poor place to learn this ratio, because people wait while you learn it.

Also record whether the old value disappeared before the new one was ready. That gap is the dangerous window. If you can keep serving the previous value until the new one is stored, the window shrinks to the requests that truly have nothing to serve. Many stacks skip that and treat every expiry as a hard miss. A hard miss is a design choice. It is not a Redis law.

3. Make one rebuild win, and give the lock a short life

The usual fix is single-flight. The first request that sees the miss takes a lock and rebuilds. The others do not all run the same query. They wait briefly, or they serve the previous value, or they return a clear “try again” if waiting is worse for that page. When the rebuild finishes, it writes the key and releases the lock. A second rebuild of the same key in that window is a bug, not protection.

The lock needs its own expiry. If a worker dies while holding a lock with no TTL, the stampede turns into downtime: nobody rebuilds, and nobody else may try. Set a short TTL on the lock. It should be longer than a healthy rebuild and shorter than the time you are willing to serve stale data or fail the request. If the same job arrives twice, it still must write the value once. That is the same idempotent habit you already want in a queue: a retry must not duplicate the side effect. When the rebuild is heavy enough to leave the request path, the boundary is the one in when work should leave the request for a queue. A queue does not fix a stampede by itself. Fifty jobs that rebuild the same key are still a herd. They only moved into the worker.

Jitter belongs next to the lock. If a hundred keys share one TTL, they still expire together even when each key has a perfect lock. Add a random slice to the TTL so expiries spread out. Early refresh is the other half: rebuild before the key expires, while readers still get the old value. The read path stays a hit. The write path becomes one job. You do not need a new cache product for either step. You need the key, the lock, and a decision about stale reads.

4. When a larger box is the honest answer

Buy RAM or CPU after one rebuild is already the only rebuild, and the remaining cost comes from real traffic. If a single rebuild is cheap and the database still saturates because many different keys are legitimately cold, that can be real load. Measure it as distinct keys, not repeated work on one key. If Redis is evicting hot keys because the memory limit is real, and those keys have TTLs and a clear purpose, more memory can be the right purchase. Write down the eviction reason and the key pattern first. “Redis is full” without a pattern is how unbounded keys get a larger machine instead of a fix.

Do not buy the box to survive a flush. A deploy that deletes the hot set will trigger a herd on any size you rent until the flush is gone or the rebuild becomes single-flight. Do not buy the box instead of reading the statement that runs on a miss. Fifty fast queries are still fifty queries, and one slow query multiplied by fifty is the same pain repeated. The order is intentionally boring. Name the key. Count rebuilds for one expiry. Collapse them to one. Spread the TTLs. Then look at the machine after duplicated work is gone.

One expiry on a copy: misses, rebuilds, and the duration of one rebuild.
A lock with a TTL, and a rebuild that stays single if the job runs twice.
Jitter, or an early refresh that serves the previous value.
No deploy flush of the hot set unless you have planned the herd.
A hardware note only after distinct work remains, with the key pattern written down.

When to ask someone to read the miss path with you

You can name the key and delete it once on a copy this week if you already run Redis. Ask for help when every miss still runs a query nobody has timed, when a deploy flush is treated as “just how we release,” or when the lock and the queue disagree and a retry rebuilds the value again. The job is to make one expiry cheap. It is not to buy a new cache product, and it is not to rent a larger box as the first move.

Services cover that review: the key, the miss, and the query or job behind it. Selected work keeps each timing next to the operation, not as a slogan about scale. If you write, send the key pattern, the TTL, and what one forced expiry did on a copy. Leave production passwords out of the first message.