Here we look at some options for redelivering messages from a RabbitMQ queue to an exchange. The queue can for example be a dead letter queue.
The background for this is that an organization I’m working with at the moment wants to automatically generate web pages based on information entered into an internal system, let’s call it system A. We decided on an architecture where system A publishes domain events using RabbitMQ, and a separate system, system B, listens to a particular kind of event and generates a web page based on information in the event.
The two systems are developed by different suppliers, and they have different release cycles. In this case, the new version of system A was released about a week before the new version of system B, and so domain events were published before anyone was ready to consume them.
The queue that system B listens to is configured with a time-to-live and a dead letter exchange. The reason for this is to avoid the problem of poison messages, i.e., messages that for some reason cause the receiving system to fail, returning the message to the queue to be redelivered, causing the system to fail again, and so on. The old book J2EE AntiPatterns called this the hot potato antipattern.
The result is that a number of messages ended up in the dead letter queue because their time-to-live was reached. When system B was ready to start consuming messages, we wanted to redeliver those messages to give system B a chance to handle them.
Options for Redelivering Messages in RabbitMQ
There are several different ways to redeliver messages in RabbitMQ:
- Manually, using the admin GUI
- Using the Shovel plugin
- Using custom code
If there are just a few messages to redeliver, you can do it manually using the RabbitMQ admin GUI. In the dead letter queue, use “Get Message(s)” to get all messages, optionally setting “Requeue” to false. For each message, copy the payload and routing key and use the information to fill out the “Publish message” section for the exchange you want to redeliver the message to.
If you want to redeliver messages on a more permanent basis, e.g., to synchronize between different RabbitMQ hosts, then the Shovel plugin is probably the right way to go.
In this particular case, there were around 2,000 messages to redeliver, and we only needed to redeliver the messages once, so I decided to write some Java code to do it:
public void moveAllMessagesToExchange(String fromQueue, String toExchange) throws IOException {
Connection connection = connectionFactory().newConnection();
Channel channel = connection.createChannel();
while (true) {
GetResponse response = channel.basicGet(fromQueue, false);
if (response == null) {
return;
}
Envelope envelope = response.getEnvelope();
String routingKey = envelope.getRoutingKey();
channel.basicPublish(toExchange, routingKey, response.getProps(), response.getBody());
channel.basicAck(envelope.getDeliveryTag(), false);
}
}
To easily run the code, I added a dummy JUnit test and ran it using Eclipse:
@Test
public void foo() throws Exception {
MoveMessages moveMessages = new MoveMessages("rabbitmq.reallifedeveloper.com", "admin", "tops3cret", "/");
moveMessages.moveAllMessagesToExchange("foo.bar.domain.dlq", "foo.bar.domain");
}
Notes
- The code above redelivers messages from a queue to an exchange. This means that the messages will be routed to all appropriate queues that are bound to that exchange. It is easy to change the code to deliver the messages directly to a specific queue, just change the call to
basicPublishto use the empty string as exchange name, and the name of the queue as routing key. - When you redeliver the messages from a dead letter queue, temporarily disable the consuming systems so that they do not consume messages that are being redelivered, if at all possible. If you do not, you run the risk of the consuming system nacking the messages, which cause them to be put in the dead letter queue again, where the redelivery code tries to redeliver them again. If this happens fast enough, you may have an endless loop on your hands.
- The code to redeliver messages can be made smarter if necessary, for example by looking at the contents of the messages and only redeliver messages with certain properties, or deliver them to different exchanges based on the contents.
Conclusion
Redelivering messages from a queue to an exchange in RabbitMQ can be done in several ways. If you need to do this only once and for a lot of messages, the simple code shown above can be used.