You start with one queue and one consumer, and for a while that’s plenty. Then the consumer stops keeping up, the queue depth climbs all day and only drains overnight, and one slow message holds up everything behind it. So you do the obvious thing and add more queues with more consumers to work in parallel. And that’s where it gets interesting, because now you have to decide which message goes to which queue, and every easy answer to that question is wrong in some specific way.
The problem
You have a stream of messages that need to spread across several queues so more than one consumer can chew through them, but two things have to stay true while you do it. Messages for the same key, say the same user or the same account, have to keep landing on the same queue, otherwise two consumers process one user’s events at the same time and the ordering you were relying on is gone. And you have to be able to add a queue later when traffic grows, without that addition scrambling where everything else lands.
The easy answers each miss one of those. A fanout exchange copies every message to every queue, which isn’t sharding, it’s duplication. Round-robin across the queues spreads the load evenly but sends one user’s events to whichever queue is next, so their order is shot. Hashing in the producer, queue = queues[hash(key) % N], actually gets you both even spread and same-key-same-queue, right up until the day you add a queue. The moment N goes from 8 to 9, hash(key) % 8 and hash(key) % 9 disagree for almost every key, so almost every key jumps to a different queue at once, and every ordering guarantee you had shatters during the migration.
That last one is the trap worth measuring, because it looks fine until you scale.
What the consistent hash exchange does
RabbitMQ ships a plugin exchange type, x-consistent-hash, that does the hashing for you on the broker side. You bind your queues to it, each with a weight, and you publish with the routing key set to your partition key. The exchange hashes the routing key onto a ring, and the queue that owns that point on the ring gets the message. Same key always hashes to the same point, so it always lands on the same queue. And because it’s a consistent-hash ring and not a modulo, adding a queue only steals the slice of the ring near the new queue’s points, so only the keys in that slice move.
In pika it’s barely any code. You declare the exchange, bind the queues with a weight as the routing key, and publish with the partition key as the routing key:
channel.exchange_declare("events", exchange_type="x-consistent-hash")
for q in queues:
channel.queue_declare(q)
channel.queue_bind(q, "events", routing_key="1") # "1" is the weight (ring points)
# the routing key IS the partition key; the broker hashes it
channel.basic_publish("events", routing_key=f"user-{user_id}", body=payload)
That’s the whole swap. The producer no longer knows or cares how many queues exist, it just stamps each message with the key it should be ordered by.
I ran it against a real RabbitMQ 4.0.9 with the plugin enabled, eight queues bound at equal weight, and a pika producer and consumer, to check the two things I actually cared about: does the hash spread evenly, and what does adding a queue cost.
It spreads evenly
First the boring-but-necessary one. I published 100,000 messages across 43,126 distinct routing keys into the eight queues and counted where they landed. Ideal is 12,500 per queue:
100,000 messages across 8 equal-weight queues
The busiest queue was 3.70% over ideal, the quietest a bit under, and everything else clustered in between. If a queue needs to carry more, you give its binding a bigger weight and it takes proportionally more of the ring. Good enough to move on.
What adding a queue costs
This is the one that matters. I took 10,000 distinct keys and recorded which queue each landed on with eight queues. Then I added a ninth queue and recorded where each key landed again, and counted how many keys had moved. Then I did the same accounting for the naive hash(key) % N approach, % 8 versus % 9, on the same keys:
Keys that moved queues when going from 8 to 9 (of 10,000)
The modulo version moved 8,873 of the 10,000 keys. Adding a single queue picked up almost every key and dropped it somewhere new, which during a live migration means almost every in-flight user briefly has messages in two queues at once, being drained by two consumers, out of order. The consistent hash exchange moved 1,090, which is 10.90%, near enough the 1/9 you’d predict. The other 89% of keys never noticed the ninth queue existed.
Same key, same queue
The reason the low remap number matters is that it protects ordering, so I checked ordering directly too. I published three copies of each of 10,000 keys through the eight-queue ring and then looked at how many queues each key’s messages ended up spread across. Every key landed in exactly one queue, zero keys showed up in more than one. So as long as each queue has a single consumer draining it, every key is processed in order, and adding a queue only disturbs order for the ~11% of keys that actually move, briefly, instead of all of them.
The takeaway
If you need to spread a stream across queues so several consumers can keep up, and you need messages for the same key to stay ordered, the consistent hash exchange gives you both, and it lets you add capacity later without reshuffling the keys that were fine where they were. Bind your queues with weights to size them, publish with your partition key as the routing key, and let the broker do the hashing. Two things to keep in mind: it’s a plugin, so it has to be enabled (rabbitmq_consistent_hash_exchange), and the per-key ordering only holds if each queue has one consumer, because competing consumers on a single queue will reorder its messages themselves. Get those right and adding a queue costs you 11% of your keys for a moment instead of 89%, the broker, the harness, and the three experiments are in the repo. Laptop numbers, RabbitMQ 4.0.9, but the consistent-hashing behaviour is the same wherever you run it.
Comments