Blog
Sharding RabbitMQ Without Reshuffling Everything
One queue can't keep up, so you add more and hash messages across them. Then you add a queue and a naive hash reshuffles 89% of your keys mid-flight. RabbitMQ's consistent hash exchange moved 11% instead. Here's the reproduction.
Our Fix for the Thundering Herd Was a Lock
A hot cached price expires, the herd hits the database, so we wrapped the recompute in a SET NX lock. The p99 didn't budge, and the day a pod died mid-recompute the tail hit 3.5 seconds. Here's why the lock was the wrong tool, measured, and what actually fixed it.
The Query Plans That Only Ran Once
Run the same query five hundred times with a different literal each time and SQL Server caches five hundred plans, each used once, eating 66MB. Parameterize it and you get one plan, 136KB. Then parameter sniffing shows up and hands the common value a plan built for a rare one.
The Expired Keys Your Redis Replica Still Counts
A replica doesn't expire keys on its own clock, it waits for the primary to tell it. I went to reproduce the classic stale-lock-on-a-replica bug and found modern Redis had quietly fixed half of it, and left the more dangerous half in plain sight.
Your Redis Pub/Sub Node Hears Its Own Messages
Redis pub/sub delivers every message to every subscriber on the channel, including the node that published it. If a node both publishes and subscribes, it hears its own echo. Here's why, a two-line fix, and a demo you can run to watch it happen.
Redis Brings Its Own malloc
Redis doesn't use your system's malloc. It ships jemalloc in its own source tree and compiles it in by default. I built the same Redis two ways and found that the allocator changed the memory numbers, just not in the direction I expected.
The Memory Redis Wouldn't Give Back
used_memory sat there calm while RSS climbed to the cgroup limit and the kernel killed the process. Everyone said leak. It wasn't. I reproduced it locally and chased down why the memory number Redis shows you isn't the one that gets you killed.
Everything I Got Wrong About Cache-Aside
Cache-aside is four lines and everyone writes it the same way, myself included. Then it serves a stale row until its TTL, melts the database when one hot key expires, and fills up Redis at 3am. Here's the pattern, when I actually reach for it, and every sharp edge that has personally bitten me.
Everything I Got Wrong About Rate Limiting
A rate limiter is three lines and everyone writes the same fixed-window counter, myself included. Then a boundary burst walks through, Redis eviction hands out free quota, and a stale replica makes the decision disagree with the headers. Here's the algorithm menu, the atomic Lua the counter actually needs, and every sharp edge that has personally bitten me.
The Lie in pg_stats
Postgres had the right index and still walked 18 million rows for a LIMIT 1 query. I built the failure locally, followed the planner's estimate back into pg_stats, and fixed it with a higher per-column statistics target.
On Remote Work and Deep Thinking
Reflections on how remote work has changed the way we think, collaborate, and find meaning in our professional lives.
On Building Meaningful Software
A reflection on what it means to build software that matters, beyond just solving technical problems.
Want to get blog posts over email?
Enter your email address and get notified when there's a new post!