The first memcached tier I ran reported plenty of free memory and evicted my hottest keys anyway. stats said limit_maxbytes was nowhere near full, the hit rate was sliding, and the keys falling out were the ones I most wanted to keep. I spent an afternoon chasing the client, the TTLs, the traffic pattern. The problem was none of those. The problem was that memcached had already decided how to cut up its memory before I stored a single byte, and my values didn’t fit the cuts.
The problem
memcached doesn’t hand out memory a byte at a time. It carves each 1 MB page into fixed-size chunks, and every item you store gets rounded up to the nearest chunk. The chunk sizes step up by a growth factor, 1.25 by default, so the ladder near a kilobyte looks like 944, 1184, 1480, 1856. Store an item whose real size is 945 bytes and it goes into a 1184-byte chunk. The other 239 bytes are gone. Not to another item, not to overhead you can point at, gone, held by the allocator, counted as used, invisible in stats unless you go digging in stats slabs.
That rounding is fine when your values happen to sit near a chunk size. It is a tax when they sit just past one. And “just past one” is not a rare accident. It is whatever your value size happens to be, which nobody picks with the slab ladder in mind. So I picked the worst case on purpose and measured what it costs.
What the slab ladder actually does
Probe it yourself. Store one item, dump stats slabs, and read the chunk_size for each class. On memcached 1.6.45 with the default factor, near a kilobyte:
class 11: 944 class 15: 2320
class 12: 1184 class 16: 2904
class 13: 1480 class 17: 3632
class 14: 1856 class 18: 4544
The total item size memcached stores is your value plus the key plus a header. For a 16-byte key that overhead measured 75 bytes flat. So a 870-byte value becomes 945 bytes on the wire, which is one byte over the 944 chunk, which means it rounds all the way up to 1184. A 1109-byte value becomes 1184 exactly and fits class 12 with nothing left over.
Same class, same 400,000 items, same amount of real data. The only difference is which side of 944 the item lands on.
Same 400,000 items. One value size wastes a fifth of the RAM.
value 870 B, one byte over 944
value 1109 B, fills the chunk
Twenty percent. The cache is holding 378 MB of data and paying for 473.6 MB of RAM, and every byte of that 95.6 MB gap is doing nothing but sitting between the end of one item and the edge of its chunk. stats will tell you the bytes are used. It won’t tell you they’re empty.
The knob you can actually turn
The growth factor is a startup flag, -f. Drop it and the chunks step up in smaller increments, so any given item rounds up less. I took the same worst-case 870-byte value and started memcached with -f 1.08 instead of the default.
Tighter growth factor, same data, less rounding.
The finer ladder cut the waste from 20.2% to 2.4%, from 95.6 MB down to 9.2 MB, without touching a single value. You can’t make the rounding disappear, only make each step smaller. And smaller steps mean more slab classes, which means memcached spreads its pages across more buckets. That’s fine until it isn’t, which brings up the second way slabs bite you.
When the pages get stuck
Here’s the part that actually cost me the afternoon. memcached assigns whole pages to a slab class, and by default it does not take them back. Once a page belongs to the class for 100-byte items, it stays there, even if you stop storing 100-byte items entirely and start storing 8 KB items that are starving for space. The RAM is right there, free, and the new items can’t have it. That’s slab calcification, and it’s why a cache with “free memory” evicts your hot keys.
I reproduced it in a 64 MB cache. First I filled it with small items until every page belonged to the small class. Then I switched the workload entirely (a working set of large 8 KB items, rewritten over and over) and watched what the large class could get. Then I ran the exact same sequence again with one flag flipped: -o slab_automove=2, which tells memcached to reassign pages from a class that’s stopped needing them to one that’s thrashing.
Same large working set. One allocator hoards, one rebalances.
One flag, and the same workload went from evicting itself 74,882 times to 1,976. The free pages were there the whole time. With movement off they stayed pinned to a workload that had already stopped, and only reassigned once I let memcached move them.
One honest caveat, because the raw number can lie. If you look at global evictions instead of large-class evictions, automove=2 looks worse, 236,799 versus 74,882. That’s because reassigning a page first evicts the stale small items still living on it. Those are one-time reclamations of data I’d already abandoned. The number that matters is the live workload’s own thrashing, and that’s the one that dropped 38x. It took some tuning to get a clean read. My first attempt sized the large set bigger than the cache could ever hold, so both modes evicted on capacity and the allocator’s effect vanished into the noise. That failed run is written up under results/attempts/ because the wrong-looking number is the whole point.
The takeaway
memcached decides how to cut up memory before you store anything, and it rounds every item up to fit. Two things follow. First, your value size interacts with a slab ladder you didn’t choose. Land one byte over a boundary and you can hand a fifth of your RAM to the allocator, invisible in every stat except stats slabs. Check where your real item sizes fall, and if they’re sitting just past a chunk, either reshape the value or tighten -f and eat the extra slab classes. Second, pages don’t move on their own unless you tell them to. If your workload’s item sizes shift over time, run with slab_automove on, or you’ll evict hot keys while free RAM sits locked to data nobody’s asking for.
Neither of these shows up as an error. The cache just quietly holds less than you paid for, and the only way to know is to measure the chunk your items actually land in.
The harness (digest-pinned memcached, the slab probe, and every number above) is on GitHub. These are laptop numbers meant to show the mechanism, not your cluster’s capacity. The percentages are the point, not the megabytes.
Comments