A publisher confirm is a promise. You send a message, the broker sends back an ack, and that ack is supposed to mean the message is safe, it’s written down, it survived, you can forget about it and move on with your day. I built a system once that leaned on that promise for years and it was fine. Then one afternoon a broker node died, another took over exactly like it was supposed to, and a batch of messages that had been confirmed, acked, promised-safe, were simply gone. No error, no warning. The publisher had done everything right and the broker had told it so.
That was the classic RabbitMQ mirrored-queue trap, and it bit enough people over enough years that RabbitMQ eventually pulled mirrored queues out of the product entirely. So the question I actually wanted answered in 2026: does it still happen? I put three RabbitMQ nodes in Docker, reproduced the loss on the old model, then ran the exact same murder against the new one. Here’s what came back.
The problem
A publisher confirm on a classic mirrored queue tells you the master node has your message. That’s it. It does not tell you a single mirror also has it. RabbitMQ’s default when the master dies is to promote a mirror even if that mirror never caught up, because an empty queue that’s online beats a queue that’s gone. So the failover can hand the master’s job to a mirror that was sitting there empty, and every message that only ever lived on the dead master goes down with it. Your confirm was real. It only ever covered the master.
What a confirm actually promises
On a mirrored queue the confirm comes back once the message is on the master and on all synchronised mirrors. The whole thing turns on that one word, synchronised. A mirror that joined the cluster late, or one whose sync you never finished, isn’t synchronised. It’s just there, empty, waiting to be told to catch up. And the default policy setting, ha-promote-on-failure=always, says promote a mirror when the master dies whether or not it ever caught up. Put those two together and you get the failure: you publish to the master, you get your confirms, the master dies before an empty mirror has synced, and RabbitMQ promotes the empty mirror because that’s what you told it to do.
I set ha-sync-mode=manual to keep the mirrors from auto-catching-up, which sounds like I’m rigging it until you’ve watched a multi-gigabyte queue that never manages to finish syncing, or met the operator who turned auto-sync off on purpose because the sync itself stalls the queue while it runs. It’s a real configuration people run in production, and it makes the loss deterministic instead of a race.
I confirmed 5,000 messages and lost all 5,000
Three nodes, RabbitMQ 3.13.7, a durable classic queue mirrored across all three. The policy:
rabbitmqctl set_policy ha-all "^ha\." \
'{"ha-mode":"all","ha-sync-mode":"manual","ha-promote-on-failure":"always"}'
Then the sequence, in order:
- Kill the two mirror nodes, so the queue’s master is the only copy.
- Publish 5,000 persistent messages to the master with publisher confirms on, and wait for every ack.
- Start the two mirrors back up. They rejoin the cluster, but unsynchronised,
synchronised_slave_pidsis an empty list. - Kill the master.
The publish side is the ordinary pika confirm loop, nothing clever:
channel.confirm_delivery()
for i in range(5000):
channel.basic_publish(
exchange="", routing_key="ha.q",
body=payload,
properties=pika.BasicProperties(delivery_mode=2), # persistent
)
# with confirms on, basic_publish raises if the broker doesn't ack the message
The broker positively confirmed all 5,000. Then I killed the master, RabbitMQ promoted rmq2, one of the mirrors that had rejoined empty, and I reconnected to the survivor and drained the queue. It handed me nothing. confirmed = 5000, recovered = 0. Every message the broker had promised me was gone, and at no point did anything report an error. That’s the part that makes it dangerous, the loss is completely silent.
Same murder, quorum queue
Quorum queues are the replacement, and they’re the reason RabbitMQ could remove mirrored queues at all. A quorum queue is a Raft group. A message isn’t confirmed until a majority of the nodes have it written down, so the confirm means the message survived to more than one place before you were told it was safe, not just to whichever node happened to answer you.
Same three nodes, RabbitMQ 4.0.9, a quorum queue this time:
channel.queue_declare("qq", durable=True, arguments={"x-queue-type": "quorum"})
Same sequence: publish 5,000 with confirms (5,000 of 5,000 acked), find the current leader through the management API, kill it. A new leader was elected, I reconnected and drained. confirmed = 5000, recovered = 5000. The kill that erased everything on the mirrored queue did nothing here, because by the time each message was confirmed it already lived on a majority of the nodes, and killing one node out of three leaves that majority standing.
Confirmed messages recovered after one node failure (of 5,000)
The two runs diverge at exactly one moment, the failover, and it’s worth seeing where each queue’s messages go through it:
Queue message count through the same failover
The catch
That safety isn’t free, and the cost shows up the moment you lose the majority. I took the same quorum queue and removed nodes one at a time, publishing 50 confirmed messages at each step:
Publisher confirms that succeeded as nodes are removed (of 50)
With two of three nodes up, every confirm still went through, because two is a majority of three. Drop to one node and the confirms just stopped, all 50 of them blocked. There’s no majority left to agree a write happened, so the queue refuses the write rather than accept one it can’t stand behind. Your publisher blocks and waits instead of getting an ack it can’t trust. That’s the opposite of what the mirrored queue did, and it’s the trade you’re actually making: a minority failure costs you nothing, and a majority failure stops your writes instead of losing them. You have to build the publisher side expecting to occasionally be told to wait.
The takeaway
If you’re still on RabbitMQ 3.x with classic mirrored queues, this failure is live, and publisher confirms will not save you from it, a confirm on a mirrored queue only ever meant the master had the message. Go look at what ha-promote-on-failure is set to before you do anything else. On 4.x the decision is made for you, mirrored queues are gone and quorum queues are the default, and that removal is the whole point, the old thing could lose data it swore it had. Just know the shape of what you signed up for: your messages survive losing a minority of the cluster, and in exchange, losing a majority stops writes rather than dropping them, so set your publisher timeouts and back-pressure with that in mind.
Three nodes, two RabbitMQ versions, the full failover sequence and the raw numbers are in the repo. Laptop numbers on RabbitMQ 3.13.7 and 4.0.9, but the behaviour, confirmed-then-lost on an unsynced promotion, nothing lost on a quorum majority, is the same wherever you run it. Both builds confirmed all 5,000 messages. Only the 4.0 one still had them a minute later.
Comments