If you’ve ever watched used_memory sit there calm and reasonable while the OOM killer took the process anyway, and everyone in the channel immediately said leak, this is for you. The OOM killer, if you haven’t met it, is the Linux kernel’s out-of-memory killer, the thing that picks a process and terminates it when the machine (or the cgroup) runs out of memory. Most of the time it’s fragmentation rather than a leak, and the reason nobody believes that at first is that the number they’re all staring at genuinely does look fine.

The problem

A Redis process gets killed by the kernel for running out of memory while used_memory still reads comfortably under its limit, so the whole channel assumes a leak. It usually isn’t one. After a big delete used_memory drops but the memory the kernel actually counts stays high, maxmemory never reacts because it’s watching the wrong number, and the process walks into the OOM killer looking healthy the whole way. This is the reproduction and the three memory numbers that tell them apart.

I wanted to reproduce this the same way I reproduce anything I don’t fully trust myself to explain, small enough to run on a laptop and argue with the same allocator that’s doing it to you in production. The setup is a queue-shaped workload: load a pile of small keys, pretend to batch-process them, then delete millions of them at once, which is the part that matters. That mass delete is where used_memory and the actual resident footprint stop agreeing.

Every memory behavior and cgroup OOM rule reproduced perfectly on the first attempt. I left the shapes that didn’t move RSS in the repo too, because a version where the memory quietly comes back on its own would make for a neater story than the one that actually happened.

The Docker harness, deterministic workload, Redis config, and the raw INFO memory dumps are in the repo. These are laptop numbers from one container with a hard memory limit, the mechanism transfers, the absolute megabytes do not. Redis 7.4.0, jemalloc 5.3.0.

The mass delete, as the two numbers see it

used_memory (what Redis asked for) used_memory_rss (what the OS backs)

used_memory drops the instant the keys are gone. RSS doesn't move, because jemalloc is still holding the freed pages. The gap between the two lines is the fragmentation everyone mistakes for a leak. Numbers pending the benchmark run.

First, there are three numbers

The thing that makes this confusing is that “how much memory is Redis using” isn’t one question, it’s three, and the three don’t get reported to the same place.

• used_memory is what Redis asked jemalloc for. Keys, values, overhead. This is what your dashboard graphs and what people mean when they say we’re at 60% and we’re fine.
• used_memory_rss is the resident set size, RSS, which is the amount of physical memory the OS is actually keeping resident for the process. The OOM killer and your cgroup only ever look at this one, which is what makes it the one that pages you at 3am.
• mem_fragmentation_ratio is just rss / used. A bit over 1 is normal and healthy. Sitting at 19 right after you deleted a few hundred thousand keys is the whole post.

So the trouble is that pretty much everything you’d naturally watch is built on the first number, and the one that actually kills the process is the second.

The workload I used

Nothing exotic. Load 220,000 small keys, each value about 200 bytes, the shape of a queue that’s buffered a lot of little jobs. Let it settle. Snapshot the numbers. Then delete essentially all of them in one go, the way a batch job does when it finishes a run and clears its working set.

Here is Redis right before the delete, straight out of INFO memory:

used_memory_human:71.61M
used_memory_rss_human:74.82M
mem_fragmentation_ratio:1.04
maxmemory_human:95MB
evicted_keys:0

Sensible. used and rss are close, the ratio is near 1, we’re comfortably under maxmemory. Now the same block right after the mass delete:

used_memory_human:3.81M
used_memory_rss_human:74.91M
mem_fragmentation_ratio:19.68
maxmemory_human:95MB
evicted_keys:0

used_memory collapsed, exactly like you’d expect after deleting the keys. used_memory_rss barely moved. The fragmentation ratio jumped to 19.68. That gap is memory the process is still holding from the OS that Redis will happily tell you it isn’t using.

Why the memory didn’t come back

When you delete a key, Redis frees it back to jemalloc. That is not the same as jemalloc giving the page back to the kernel. jemalloc manages memory in runs and chunks, and a page only becomes returnable once everything living on it is freed. After a mass delete you get pages that are mostly empty but not entirely, one surviving allocation is enough to pin a whole page as resident.

On top of that, jemalloc doesn’t rush to hand pages back even when it can. It keeps freed-but-dirty pages around on purpose so it can reuse them fast instead of going back to the kernel for every allocation, and it only decays them back over time. So immediately after a big delete you’re in the worst spot: the keys are gone from used_memory, but the pages are still resident in used_memory_rss, and they’ll only trickle back slowly if nothing forces the issue.

This is why it reads as a leak when it isn’t one. Nothing is actually lost, the memory is still accounted for and still reusable, and jemalloc does hand it back eventually, just a lot slower than the OOM killer is willing to wait.

Why maxmemory watched it happen

This is the part that actually pages someone. maxmemory is checked against used_memory, never against RSS. Eviction fires when the number Redis asked for crosses the limit, and nothing else moves it.

So through all of this used_memory is low, sometimes very low right after a delete. Redis looks at it, compares it to maxmemory, and correctly decides there’s nothing to evict. evicted_keys sat at 0 the whole time. Meanwhile the resident footprint is climbing toward the container’s 95MB limit, and the cgroup could not care less what used_memory says, it measures RSS. When RSS crossed 95MB the kernel OOM-killed the container, exit code 137, which is its way of not leaving a note.

So maxmemory, the thing you set to protect yourself, is checking used_memory, which looks fine the whole time, and nobody ever put a limit on RSS, which is the number that actually gets the process killed.

maxmemory was measuring the wrong number

What maxmemory checked

used_memory3.81M

maxmemory95MB

What the cgroup checked

RSS74.91M

cgroup limit95MB

Same instant, two measurements. Redis saw plenty of headroom under maxmemory and evicted nothing. The cgroup saw RSS against its own limit. Bar widths are plotted using measurements from benchmarks/redis-oom/results/.

The fix

A few things actually help here, and restarting the process isn’t one of them.

The direct answer to resident-but-empty pages is activedefrag. Turn it on and Redis walks the fragmented allocations, copies live data into denser pages, and lets the emptied ones go back to the kernel. In my run, RSS after the same delete came down to 19.80M and the ratio settled to 5.2 instead of staying pinned up near the peak. It burns some CPU while it runs. I’ll take that over a 137.

The second thing is to set maxmemory with real headroom under the cgroup limit, not right up against it. Since eviction watches used_memory and the kill watches RSS, you’re deliberately leaving room for the gap between them. If the hard limit is X, maxmemory has to sit far enough below X that even an ugly fragmentation ratio can’t shove RSS past the limit before eviction or defrag catches up.

And if you can, don’t delete everything in one shot. The spike is the bulk free, so when I deleted the same keys in batches with UNLINK instead, RSS only peaked at 20.29M and the ratio held around 5.33, because jemalloc got to reuse and decay pages as it went rather than being handed the whole cliff at once. UNLINK also frees on a background thread, so you’re not stalling command processing while it cleans up.

Stuff worth remembering

• When Redis looks like it’s leaking, check mem_fragmentation_ratio before you reach for a heap profiler. Calm used_memory sitting next to high rss is fragmentation, and it wants a completely different fix than a real leak.
• maxmemory only ever looks at used_memory, so it will happily let RSS climb after a mass delete without evicting a thing. Under a hard memory limit, minding that gap is on you.
• Deleting a lot of keys at once can cost you more resident memory for a while, not less. Batch it with UNLINK when you can.
• These are laptop-scale numbers, here to show the mechanism, not to size anything. What matters is which INFO field is telling the truth and which limit is actually load-bearing.

The takeaway

The reason this one is worth keeping in your head is that it’s a ten-minute fix once you know to look at RSS, and I’ve watched it turn into a two-day “we have a leak” hunt when nobody did. used_memory is the number Redis asked for and used_memory_rss is what the OS is actually holding, so when the two disagree by a lot, trust RSS, because that’s the number the kernel is about to act on.