Redelivering Dead-Lettered Messages in RabbitMQ

Here we look at some options for redelivering messages from a RabbitMQ queue to an exchange. The queue can for example be a dead letter queue.

The background for this is that an organization I’m working with at the moment wants to automatically generate web pages based on information entered into an internal system, let’s call it system A. We decided on an architecture where system A publishes domain events using RabbitMQ, and a separate system, system B, listens to a particular kind of event and generates a web page based on information in the event.

The two systems are developed by different suppliers, and they have different release cycles. In this case, the new version of system A was released about a week before the new version of system B, and so domain events were published before anyone was ready to consume them.

The queue that system B listens to is configured with a time-to-live and a dead letter exchange. The reason for this is to avoid the problem of poison messages, i.e., messages that for some reason cause the receiving system to fail, returning the message to the queue to be redelivered, causing the system to fail again, and so on. The old book J2EE AntiPatterns called this the hot potato antipattern.

The result is that a number of messages ended up in the dead letter queue because their time-to-live was reached. When system B was ready to start consuming messages, we wanted to redeliver those messages to give system B a chance to handle them.

Options for Redelivering Messages in RabbitMQ

There are several different ways to redeliver messages in RabbitMQ:

  • Manually, using the admin GUI
  • Using the Shovel plugin
  • Using custom code

If there are just a few messages to redeliver, you can do it manually using the RabbitMQ admin GUI. In the dead letter queue, use “Get Message(s)” to get all messages, optionally setting “Requeue” to false. For each message, copy the payload and routing key and use the information to fill out the “Publish message” section for the exchange you want to redeliver the message to.

If you want to redeliver messages on a more permanent basis, e.g., to synchronize between different RabbitMQ hosts, then the Shovel plugin is probably the right way to go.

In this particular case, there were around 2,000 messages to redeliver, and we only needed to redeliver the messages once, so I decided to write some Java code to do it:

    public void moveAllMessagesToExchange(String fromQueue, String toExchange) throws IOException {
        Connection connection = connectionFactory().newConnection();
        Channel channel = connection.createChannel();
        while (true) {
            GetResponse response = channel.basicGet(fromQueue, false);
            if (response == null) {
                return;
            }
            Envelope envelope = response.getEnvelope();
            String routingKey = envelope.getRoutingKey();
            channel.basicPublish(toExchange, routingKey, response.getProps(), response.getBody());
            channel.basicAck(envelope.getDeliveryTag(), false);
        }
    }

To easily run the code, I added a dummy JUnit test and ran it using Eclipse:

    @Test
    public void foo() throws Exception {
        MoveMessages moveMessages = new MoveMessages("rabbitmq.reallifedeveloper.com", "admin", "tops3cret", "/");
        moveMessages.moveAllMessagesToExchange("foo.bar.domain.dlq", "foo.bar.domain");
    }

Notes

  • The code above redelivers messages from a queue to an exchange. This means that the messages will be routed to all appropriate queues that are bound to that exchange. It is easy to change the code to deliver the messages directly to a specific queue, just change the call to basicPublish to use the empty string as exchange name, and the name of the queue as routing key.
  • When you redeliver the messages from a dead letter queue, temporarily disable the consuming systems so that they do not consume messages that are being redelivered, if at all possible. If you do not, you run the risk of the consuming system nacking the messages, which cause them to be put in the dead letter queue again, where the redelivery code tries to redeliver them again. If this happens fast enough, you may have an endless loop on your hands.
  • The code to redeliver messages can be made smarter if necessary, for example by looking at the contents of the messages and only redeliver messages with certain properties, or deliver them to different exchanges based on the contents.

Conclusion

Redelivering messages from a queue to an exchange in RabbitMQ can be done in several ways. If you need to do this only once and for a lot of messages, the simple code shown above can be used.

Why You Should Publish Domain Events

A domain event is a concept from domain-driven design that signals that the state of a system has changed in a way that may be interesting to others. For example, one type of domain event might show that a user has been added to the database, while another type of domain event could signal that an invoice has been approved, reduced or rejected.

In the highly recommended book Implementing Domain-Driven Design, the author Vaughn Vernon devotes a whole chapter to domain events and how to implement them. This post presents some ideas from the book. Please take the time to read the book for a more in-depth discussion of the topic.

Benefits of Domain Events

So why should you start publishing domain events? The simple answer is that it is an easy way to make your systems more loosely coupled. Domain events can be used to facilitate eventual consistency, which can eliminate the need for distributed transactions, and can improve scalability and performance.

Domain events also make it possible to create integrations that you originally did not plan for. The events provide a way to plug into the processing of a system, and to add new “modules” in the form of completely separate systems.

If you think of the times when you have had to poll a system for changes using complex queries in order to perform some processing, you can surely see how much simpler the task would have been if you had been informed of the changes you were interested in.

Taken together, the benefits of using domain events are great, and the investment in developer time very small. Once the infrastructure for working with events is in place, actually publishing the events is trivial.

Basics of Domain Events

Here are a few things to keep in mind when working with domain events:

  • A domain event shows that something already has happened. Domain events should therefore be named using past tense, for example, UserAdded, InvoiceApproved, InvoiceReduced or InvoiceRejected.
  • Since domain events represent something that has already happened, they are immutable.
  • The domain events should make sense to domain experts, and the events should become a part of the ubiquitous language used by both domain experts and developers.
  • The producer of a domain event should not have to care about which consumers, if any, there are. A domain event is simply published, and may be picked up by zero or more consumers that can be local, i.e., code in the same system, or external, i.e., separate systems.
  • An existing system can be incrementally updated to publish domain events to solve a new integration requirement.

Implementing Domain Events

A domain event should keep track of when it occurred. It may also be useful to keep a version number, e.g., starting at 1, in case we want to examine old events after the implementation has evolved.

Other than that, the domain event should have properties that make it possible to understand what happened. It may be helpful to think of what information would be necessary to trigger the event again, and to include all such information in the event.

Since the event should be useful to external consumers, it should contain only simple data types such as strings, numbers and dates. What we are saying is basically that the event should be serializable, even though we probably will use our own serialization format, such as JSON, and not use the serialization mechanism built into the development language.

Publishing Domain Events

A domain event signals that something has occurred. This means that the event must be transactionally coupled to the domain model. If the transaction is rolled back, the effect must be the same as if the event was never published.

If you use domain events internally, for example to let a domain entity publish an event that should be acted upon by other code in the same bounded context, this is easy. Just create a thread local list of all internal subscribers, and call the event handling method of the subscribers when an event is published.

Normally, you also want to publish domain events externally, to other systems. A message queue is the perfect way to reach remote subscribers. Here we have to be careful not to send the message if the transaction in which the event was created is rolled back. This can either be accomplished by using distributed transactions, or simpler using an event store.

An event store uses the same data store as the domain model, so it is automatically included in the same transaction that generates the domain event. A background thread is then used to regularly send any new events to the message queue.

The event store has the added benefit of being a complete history of all events produced by a system. This can provide a description of the state of the system, which can be used as an alternative to storing the current state of objects, a technique called event sourcing. Even if you do not use event sourcing, having a history of events is often useful when debugging problems in a live system.

When you publish events from a service, you can normally use dependency injection to access the publisher. When you publish events from entities or value objects, you will probably have to provide a way for the producer to statically access the event publisher.

An example implementation based on the ideas from the book Implementing Domain-Driven Design can be found on GitHub.

Conclusion

  • Practically any system that has a state that can change can benefit from domain events.
  • Publishing domain events requires some infrastructure, such as support code and a message queue. This is a one-time investment.
  • The publishing of domain events can be incrementally added to an existing system.