A notification service starts life as one queue. OTPs go in it, password resets go in it, payment receipts go in it, and the 9 a.m. marketing blast goes in it too. This is fine for a long time, because the queue is usually near empty and everything drains the moment it lands. Then marketing schedules a campaign, twenty thousand “you left something in your cart” messages hit the queue in one burst, and somewhere in the middle of that a customer tries to log in and waits fifteen seconds for a code that should have taken twenty milliseconds. Support gets tickets. Marketing did nothing wrong. The queue did exactly what you built it to do.
The problem
One queue can’t tell the difference between traffic that can wait and traffic that can’t. A FIFO queue serves in arrival order, so once a bulk campaign is sitting in front of your OTP, the OTP waits for the entire campaign to drain first. It doesn’t matter that the OTP is the one message in there a human is actively staring at. The fix everyone reaches for is a priority queue, and the surprising part is that adding one, by itself, barely helped. The OTP still waited almost four seconds. The reason had nothing to do with priority and everything to do with a setting most people never touch.
I reproduced all of it on a single RabbitMQ 4.0.9 broker on my laptop. Twenty thousand bulk messages, then two hundred OTPs injected after the backlog was already enqueued and draining, and I measured how long each OTP sat from publish to the moment a consumer actually got it. Same load every time, only the queue config changed.
What FIFO does to an OTP
First the naive setup: one plain queue, no priorities. The consumer pulls messages in order and spends about half a millisecond on each, so it drains around 1,300 a second. The backlog is twenty thousand deep. Do the arithmetic and an OTP landing at the back has to wait for roughly twenty thousand messages to clear before its turn, which is exactly what happened.
OTP p99 latency, single FIFO queue
Fifteen seconds at p99. A login code that arrives fifteen seconds late has already failed, the user has hit resend twice and filed a ticket. So you do the obvious thing.
The obvious fix, and why it didn’t work
RabbitMQ has priority queues built in. You declare the queue with a max priority and publish urgent messages higher:
channel.queue_declare(queue="notify", durable=True,
arguments={"x-max-priority": 10})
# bulk campaign
channel.basic_publish(exchange="", routing_key="notify", body=payload,
properties=pika.BasicProperties(priority=0))
# OTP
channel.basic_publish(exchange="", routing_key="notify", body=payload,
properties=pika.BasicProperties(priority=9))
Same load as before, OTPs now at priority 9, bulk at 0. I expected this to be the whole story. It wasn’t.
OTP p99 latency, priority queue with prefetch 5000
Priority helped, in the sense that fifteen seconds became four. But four seconds is not what “priority” is supposed to buy you, and the number itself is the clue. The consumer drains about 1,300 a second and its prefetch was set to 5,000. Divide 5,000 by 1,300 and you get about 3.8 seconds. The OTP wasn’t waiting behind the twenty-thousand-message backlog anymore. It was waiting behind the five thousand messages the consumer had already pulled out of the queue.
Prefetch is a second queue you forgot about
Priority only sorts the messages that are still sitting in the queue in the ready state. The moment a consumer with a prefetch of 5,000 connects, the broker hands it up to 5,000 messages at once and marks them unacknowledged, in flight, gone from the ready set. Those messages now live in a buffer on the client side, in plain arrival order, and priority has no say over them anymore. Your urgent OTP arrives at the broker, gets sorted to the front of the ready queue correctly, and then waits for the consumer to chew through the five thousand bulk messages it already grabbed before it ever asks for the next one.
You built a priority queue and then quietly put a five-thousand-deep FIFO in front of it. The prefetch buffer is a queue too, it just doesn’t look like one because it lives in your consumer and nobody named it.
The real fix
Drop the prefetch. With a prefetch of 1, the consumer holds exactly one message at a time and asks the broker for the next one only after it acks. Every time it asks, the broker re-checks the ready queue and priority actually gets to do its job, so a fresh OTP jumps the entire backlog.
channel.basic_qos(prefetch_count=1)
OTP p99 latency, priority queue with prefetch 1
Three and a half milliseconds at p99, behind a twenty-thousand-message backlog. That’s the line-cutting you thought you were buying when you declared the priority queue. The only thing that changed between the four-second version and this one is a single integer.
The cost is real, and it’s throughput
Prefetch 1 isn’t free, and if it were, RabbitMQ would default to it. Every message now pays a full round trip: deliver, process, ack, ask for the next. On my laptop that dropped bulk drain from about 1,300 a second to 823.
Bulk drain throughput
So prefetch is a straight trade. High prefetch pipelines the consumer and gives you throughput, at the cost of a deep client-side buffer that priority can’t reorder. Prefetch 1 gives priority full control and instant OTPs, at the cost of a third of your bulk throughput. In practice you don’t want either extreme, you want a small prefetch, something like 10 or 20, low enough that the client buffer is a few messages instead of thousands, high enough that the consumer isn’t idling on every round trip.
The other answer, and the one I’d actually reach for in production, is to stop making OTPs share a queue with a marketing campaign at all. Give critical traffic its own queue and its own consumers, and the bulk backlog can’t get in front of it no matter how deep it is. Priority queues are the single-queue tool for when you can’t or don’t want to split them. Either way, the lesson is the same.
The takeaway
A priority queue only reorders messages the broker still holds. Anything your consumer has already prefetched is a private FIFO that priority can’t touch, so a high prefetch quietly rebuilds the head-of-line blocking you added the priority queue to fix. If urgent messages share a queue with a bulk backlog, keep the prefetch small, small enough that the client-side buffer can’t hide a campaign in it, and measure the p99 of the urgent traffic under a real backlog rather than trusting that “priority” means what it sounds like. And if you can afford it, give the urgent traffic its own queue and skip the whole problem.
The harness that produced these numbers is on GitHub. These are laptop numbers meant to show the shape of the effect, not a capacity statement for your broker, run it against your own setup before you trust the magnitudes.
Comments