Blog
One Byte Over the Slab
memcached rounds every item up to the next slab class. Land one byte over a boundary and you hand a fifth of your RAM to the allocator for nothing. I measured it.
The Messages RabbitMQ Confirmed and Lost Anyway
A publisher confirm is supposed to mean the broker has your message. On a classic mirrored queue I confirmed 5,000 messages and a single node failure lost all 5,000. Here's the reproduction on RabbitMQ 3.13, and what quorum queues do differently on 4.0.
The Query That Asked Every Shard
One Postgres table outgrows one machine, so you split it across eight. Then you find out that a query carrying the shard key touches one shard in 2ms, and the same query without it has to ask all eight and comes back in 12. The shard key wasn't a detail. It was the whole design.
Sharding RabbitMQ Without Reshuffling Everything
One queue can't keep up, so you add more and hash messages across them. Then you add a queue and a naive hash reshuffles 89% of your keys mid-flight. RabbitMQ's consistent hash exchange moved 11% instead. Here's the reproduction.
Our Fix for the Thundering Herd Was a Lock
A hot cached price expires, the herd hits the database, so we wrapped the recompute in a SET NX lock. The p99 didn't budge, and the day a pod died mid-recompute the tail hit 3.5 seconds. Here's why the lock was the wrong tool, measured, and what actually fixed it.
The Query Plans That Only Ran Once
Run the same query five hundred times with a different literal each time and SQL Server caches five hundred plans, each used once, eating 66MB. Parameterize it and you get one plan, 136KB. Then parameter sniffing shows up and hands the common value a plan built for a rare one.
The Expired Keys Your Redis Replica Still Counts
A replica doesn't expire keys on its own clock, it waits for the primary to tell it. I went to reproduce the classic stale-lock-on-a-replica bug and found modern Redis had quietly fixed half of it, and left the more dangerous half in plain sight.
Your Redis Pub/Sub Node Hears Its Own Messages
Redis pub/sub delivers every message to every subscriber on the channel, including the node that published it. If a node both publishes and subscribes, it hears its own echo. Here's why, a two-line fix, and a demo you can run to watch it happen.
Redis Brings Its Own malloc
Redis doesn't use your system's malloc. It ships jemalloc in its own source tree and compiles it in by default. I built the same Redis two ways and found that the allocator changed the memory numbers, just not in the direction I expected.
The Memory Redis Wouldn't Give Back
used_memory sat there calm while RSS climbed to the cgroup limit and the kernel killed the process. Everyone said leak. It wasn't. I reproduced it locally and chased down why the memory number Redis shows you isn't the one that gets you killed.
Everything I Got Wrong About Cache-Aside
Cache-aside is four lines and everyone writes it the same way, myself included. Then it serves a stale row until its TTL, melts the database when one hot key expires, and fills up Redis at 3am. Here's the pattern, when I actually reach for it, and every sharp edge that has personally bitten me.
Everything I Got Wrong About Rate Limiting
A rate limiter is three lines and everyone writes the same fixed-window counter, myself included. Then a boundary burst walks through, Redis eviction hands out free quota, and a stale replica makes the decision disagree with the headers. Here's the algorithm menu, the atomic Lua the counter actually needs, and every sharp edge that has personally bitten me.
Want to get blog posts over email?
Enter your email address and get notified when there's a new post!