What is RabbitMQ? Find out in our introduction to the powerful Message Queue tech.
Why Rabbit? What is MQ? How can it improve our applications? Why would I want to learn more about it? — These are questions that I asked when I was first introduced to RabbitMQ. I’m Gabor, and I’m now a RabbitMQ engineer and consultant. In my time working with RabbitMQ, I’ve learned that even experienced customers ask these questions.
What problem does RabbitMQ solve?
Before we delve into what RabbitMQ is and how to use it, it is worth learning more about the problem domain itself. Communication between different services (a.k.a computers) is an age-old problem.
On one hand, there are the different protocols defining the means of transportation and the properties of the communication. Some examples of such protocols include SMTP, FTP, HTTP or WebSockets (to name a few), which are all based on TCP/UDP. They deal with the formatting, reliability and finding the correct recipient of a message.
On the other hand, we can explore the communication from the perspective of the message. It exists in one system, then it is transported to another, gets transformed, that is, it has a lifecycle. As it travels from one system to another, we should be aware of where the message is, and who owns it at any given point of time.
The communication protocols mentioned above can make sure that the ownership (and the “physical” location) of the message is transferred from one system to the other (although it may take some time to execute this transaction). We can consider the transfer to be a transaction between the two parties while both are present. Most of the time, this active exchange is desirable, e.g. asking for the status of the service and expecting a timely and accurate answer. An example from the physical world would be calling somebody over the phone: 1) we start the call, 2) wait for the other party to answer, 3) have a nice discussion, 4) hang up the phone.
But there are other times when we don’t need the answer, we just need the receiver to take ownership of the message and do its job. In this case, we need an intermediary agent, another system to take ownership of the message (temporarily) and make sure that the message reaches its destination. To push the phone example further, the other party is not available at the moment, so we leave a voice message. The voicemail service will notify the intended receiver.
This asynchronous (delayed) message delivery is what RabbitMQ provides. Obviously, it can do more than a simple answering machine, so let’s explore some of the options it provides below
(If you are interested in learning more about the history of RabbitMQ, I recommend the first chapter of “RabbitMQ in Action” by Alvaro Videla and Jason Williams. It will reveal the answer to why it is named after Rabbits).
What is RabbitMQ?
RabbitMQ is a free, open-source and extensible message queuing solution. It is a message broker that understands AMQP (Advanced Message Queuing Protocol), but is also able to be used with other popular messaging solutions like MQTT. It is highly available, fault tolerant and scalable. It is implemented in Erlang OTP, a technology tailored for building stabe, reliable, fault tolerant and highly scalable systems which possess native capabilities of handling very large numbers of concurrent operations, such as is the case with RabbitMQ and other systems like WhatsApp, MongooseIM, to mention a few.
At a very high level, it is a middleware layer that enables different services in your application to communicate with each other without worrying about message loss while providing different quality of service (QoS) requirements. It also enables fine-grained and efficient message routing enabling extensive decoupling of applications.
RabbitMQ Use Cases
To show off the versatility of RabbitMQ, we are going to use three case studies that demonstrate how RabbitMQ is well suited as a black-box managed service approach, as one that integrates tightly with the application enabling a well-functioning micro-service architecture, or as a gateway to other legacy projects.
RabbitMQ as a general message-bus
When a monolith system is broken down to separate subsystems, one of the biggest problems that needs solving is which communication technology to use. A solution like Mulesoft, or MassTransit can “wire” services by declaring HTTP listeners and senders. This kind of solution treats RabbitMQ as a black box, but is still able to leverage the capabilities of RabbitMQ. As an example of direct communication, let’s use HTTP to “connect” the individual services. While it is a well-supported and solid choice, it has some drawbacks:
1) The discovery of services is not solved. A possible solution is to use DNS. As the system scales and grows, so too does the complexity of finding and balancing this load. RabbitMQ can mitigate the increased complexity of the solution.
2) The communication is ephemeral. Messages are prone to being dropped or duplicated on the network layer. If a service is unavailable temporarily, the delivery fails.
RabbitMQ can help in both cases by utilising message queues as a means of transport. Services can publish and consume messages, which decouples the end-to-end message delivery from the availability of the destination service. If a consuming service is temporarily unavailable, unlike HTTP, the message is safely buffered and retained in RabbitMQ, and eventually delivered when the service comes back online.
Discoverability is simplified too. All we need to know is where RabbitMQ is and what the queue name is. Although it seems like this just reinvents the problem, this is scalable. The queue name acts as the address of the service. Consuming messages from the queues by the individual services offer a means for scalability, i.e. each queue can serve multiple consumers and balance the load. There is no need to change the queue configuration already built into the services.
This moderately static queue configuration pushes RabbitMQ to a middleware layer where a solid design can guarantee a stable service quality in the long term.
RabbitMQ as an advanced routing layer for micro-services
On the other end of the spectrum is an architecture which is more fluid and adapts to the ever-changing needs of many micro-services. What makes RabbitMQ shine in this environment is the very powerful routing capabilities it provides.
The routing logic is implemented in different (so-called) exchange types that can be dynamically created by the application when needed. The destination services create the queues that they wish to consume from, then bind them to exchanges by specifying a pattern for the keys the publishers can use when publishing the message. (Think about these keys as metadata that the exchanges can use to route and deliver the messages to one or more queues.)
RabbitMQ comes with four useful exchange types that cover most of the use-cases for messaging:
1) Direct exchange. This will deliver the incoming message to any queue whose binding key exactly matches the routing key of the message. If you bind the queues with the queue name as routing keys, then you can think about it as a one-to-one message delivery. It is simple to deliver the same message to multiple queues by using the binding keys for multiple queues.
2) Topic exchange. This will deliver the incoming message to any queue whose wild-card binding key matches the routing key of the published message. Binding keys can contain wild-card matching criteria for a compound routing key. (e.g. the binding key logs.*.error will match the routing keys logs.accounting.error and logs.ui.error). This enables us to write simple services where the logic is well contained, and the message will arrive to the correct services through the “magic” of RabbitMQ.
3) Fanout exchange. Some messages need to be delivered to all queues, this is where a fanout exchange can be used instead of writing an elaborate multicast logic in the application. With a RabbitMQ fanout exchange, each service binds the appropriate queue to the exchange without need to specify a binding key, and it all happens automatically. If a binding key is specified, the fanout exchange will simply ignore it and still route/broadcast messages to all queues bound to it.
4) Headers exchange. This exchange leverages the structure of AMQP messages and is capable of complex routing based on the headers (including custom ones) of the AMQP message. Headers are metadata attached to each message sent via AMQP.
In addition to exchanges, there are other useful features in RabbitMQ which enable the implementation of very complex messaging logic. Some of the most important features include:
1) Custom plug-ins. RabbitMQ is extensible by allowing its users to add plug-ins. Almost every aspect of RabbitMQ is customisable, including the management, authentication and authorisation, back-up solutions, and clustering. 2) Clustering. When a single RabbitMQ server is not enough, multiple RabbitMQ brokers can be connected to work together and scale the system. It can enable RabbitMQ to process more messages or increase resilience to errors. 3) Quality of Service tuning. Time-sensitive message delivery can be helped by attaching a TTL (Time-to-Live) value to either the message or the queue. Timed out messages can be automatically delivered to a Dead-letter queue. Combining ordinary routing logic and these extra features can lead to highly advanced routing logics. Another useful feature is using priority queues where the publisher can assign a priority level to each message. It is also possible to limit the number of unacknowledged messages, which allows for the performance tuning of the consuming services, in this case, RabbitMQ applies a back-pressure mechanism.
RabbitMQ integrated into legacy systems
In the previous use-case, I mentioned the possibility of using plug-ins to extend the functionality of RabbitMQ. This powerful feature allows RabbitMQ to act as a mediation layer between your RabbitMQ native (AMQP capable) services and other legacy applications. Some notable examples include:
1) Using RabbitMQ as an MQTT broker by simply enabling a plug-in. This opens up the landscape to many IoT technologies.
2) RabbitMQ’s JMS (Java Message Service) plug-in, which allows RabbitMQ to communicate with any JMS capable messaging solution.
3) If your application is using a proprietary protocol for communicating, it is possible to develop a custom plugin to connect to any such services.
RabbitMQ integrated into legacy systems
In the previous use-case, I mentioned the possibility of using plug-ins to extend the functionality of RabbitMQ. This powerful feature allows RabbitMQ to act as a mediation layer between your RabbitMQ native (AMQP capable) services and other legacy applications. Some notable examples include:
1) Using RabbitMQ as an MQTT broker by simply enabling a plug-in. This opens up the landscape to many IoT technologies.
2) RabbitMQ’s JMS (Java Message Service) plug-in, which allows RabbitMQ to communicate with any JMS capable messaging solution.
3) If your application is using a proprietary protocol for communicating, it is possible to develop a custom plugin to connect to any such services.
Conclusion
As the above examples demonstrate, there is hardly anything that RabbitMQ can’t communicate with. But as with anything in life, it has a price. Although configuring RabbitMQ is mostly straightforward, sometimes the mere number of features can be overwhelming. If you face any problems with designing, implementing or supporting your RabbitMQ brokers, reach out to our expert team here.
