RabbitMQ is an open-source message broker software that allows you to build distributed systems and implement message-based architectures. It’s a reliable and scalable messaging system that enables efficient communication between different parts of your application. However, managing RabbitMQ logs can be a challenging task, especially when it’s deployed on a large cluster. In this article, we’ll take you through some basic know-how when it comes to logging on RabbitMQ, including how to configure RabbitMQ logs to store them at a particular location, how to change the format of the logs, how to rotate RabbitMQ log files, how to monitor and troubleshoot RabbitMQ logs.
Understanding RabbitMQ Logs
RabbitMQ logs are a form of recorded information generated by the RabbitMQ message broker system. These log messages provide a lot of information on events taking place in the RabbitMQ cluster. For example, a log message is generated whenever a new message is received in a queue, when that message is replicated in another queue, if a node goes down or comes up, or if any error is encountered.
The log entries typically include timestamps, severity levels, and descriptive messages about the events that occurred. These logs can help administrators and developers diagnose problems, identify bottlenecks, analyse system performance, track message flows, and detect any anomalies or misconfigurations within the RabbitMQ infrastructure.
Before managing your RabbitMQ logs, you should first understand the different types of logs that RabbitMQ produces. There are several types of RabbitMQ logs:
- Crash Logs: produced when RabbitMQ crashes or encounters a fatal error.
- Error Logs: contain information about errors that occur during RabbitMQ’s runtime.
- Warning Logs: contain warnings about issues that may affect RabbitMQ’s performance.
- Info Logs: contain general information about RabbitMQ’s runtime.
- Debug Logs: contain detailed information about the RabbitMQ server’s internal processes and operations, such as message routing decisions, channel operations, and internal state changes.
Not having these log messages can make debugging issues very difficult, as they help us understand what RabbitMQ is doing at any given point in time. So enabling RabbitMQ logging and storing these log messages for a certain period is crucial for troubleshooting and fixing problems fast.
RabbitMQ logging location
Use RabbitMQ management UI or
rabbitmq-diagnostics -q log_location to find when a node stores its log file(s).
By default, the logging directory is:
The default value for the log folder location is saved in an environment variable named
RABBITMQ_LOGS. Changing this environment variable will change the RabbitMQ log folder.
You can check the info of environment variables here.
Or we can use the configuration file to change the logging folder. By default, this parameter is commented because the environment variable is set automatically while installing or deploying RabbitMQ. The default value of the log file location configuration in the rabbitmq.conf file is as follows:
log.dir = /var/log/rabbitmq
Configuring RabbitMQ Logging
RabbitMQ offers various log levels, each having a different severity number. This number is important, as it decides, along with other factors, if a message has to be logged or not.
By default, RabbitMQ logs at the
info level. However, you can adjust the log levels based on your needs. The
none level means no logging.
To make the default category log only errors or higher severity messages, use
log.default.level = error
Each output can use its own log level. If a message has lower severity than the output level, the message will not be logged.
For example, if no outputs are configured to log debug messages, even if the category level is set to
debug, the debug messages will not be logged.
On the other hand, if an output is configured to log debug messages, it will get them from all categories, unless a category is configured with a less verbose level.
Changing RabbitMQ log level
RabbitMQ provides two methods of changing the log level: via the configuration files or running a command using CLI tools.
Changing the log level in the configuration file requires a node restart for the change to take effect. But this change is permanent since it’s made directly in the configuration file. The configuration parameter for changing the log level is as follows:
log.default.level = info
You can change this value to any supported log level and restart the node, and the change will be reflected.
The second method, using CLI tools, is transient. This change will be effective only until a node restarts. After a restart, the log level will be read from the configuration file again. This method comes in handy when you want to temporarily enable the RabbitMQ debug log level to perform some quick debugging:
rabbitmqctl -n rabbit@target-host set_log_level debug
The last option passed to this command is the desired log level. You can pass any of the supported log levels. Changing the log level this way does not require a node restart, and the changes will take effect immediately
Log Messages Category
Along with log levels, RabbitMQ offers log categories to better fine-tune logging and debugging. Log categories are various categories of messages, which can be visualised as logs coming from various modules in RabbitMQ.
This is the list of log categories available in RabbitMQ:
- Connection: Connection lifecycle event logs for certain messaging protocols
- Channel: Errors and warning logs for certain messaging protocol channels
- Queue: Mostly debug logs from message queues
- Mirroring: Logs related to queue mirroring
- Federation: Federation plugin logs
- Upgrade: Verbose upgrade logs
- Default: Generic log category; no ability to override file location for this category
You can use these to override the default log levels configured at the global level. For example, suppose you’re trying to debug an issue and have configured the global log level as debug. But you want to avoid debug log messages for the upgrade category, or all log messages for the upgrade category.
For this, configure the upgrade log category to not send any log messages:
log.<category>.level = none
Similarly, you can configure all messages from a certain category to log to a separate file, using the following configuration:
This way, RabbitMQ makes it easy to segregate log messages based on both the log category and the log level.
RabbitMQ logs can quickly fill up your disk space, so it’s essential to rotate the logs regularly. By rotating the logs, you can ensure that you have enough disk space for your RabbitMQ logs.
Based on Date
You can use a configuration parameter to automatically rotate log files based on the date or a given hour of the day:
log.file.rotation.date = $D0
The $D0 option will rotate a log file every 24 hours at midnight. But if you want to instead rotate the log file every day at 11 p.m., use the following configuration:
log.file.rotation.date = $D23
Based on File Size
RabbitMQ also allows you to rotate log files based on their size. So, whenever the log file hits the configured file size, a new file is created:
log.file.rotation.size = 10485760
The configuration above rotates log files every time the file size hits 10 MB.
Number of Archived Log Files to Retain
When log file rotation is enabled, it’s also important to make sure you configure how many log files to retain. Otherwise, all older log files will be retained on disk, consuming unnecessary storage. RabbitMQ has a built-in configuration for this as well, and you can control the number of files to retain using the following configuration:
log.file.rotation.count = 5
In this example, five log files will be retained on the storage disk.
Rotation Using Logrotate
On Linux, BSD and other UNIX-like systems, logrotate is an alternative way of log file rotation and compression.RabbitMQ Debian and RPM packages will set up logrotate to run weekly on files located in default
/var/log/rabbitmq directory. Rotation configuration can be found in
How to Format RabbitMQ Logs
Properly formatting logs is crucial for effective log management in RabbitMQ. It helps improve readability, enables efficient searching, and allows for easier analysis. In most production environments, you would want to use a third-party tool to analyse all of your logs. This becomes easy when the logs are formatted in a way that’s easy to understand by an outside tool. In most cases, this is the JSON format.
You can use a parameter in the configuration file to change the format of the RabbitMQ log messages to JSON. This takes effect irrespective of the log level you have configured or the category of logs you’re logging.
To change the log format to JSON, you just have to add or uncomment the following configuration in the configuration file:
log.file.formatter = json
This will convert all log messages written to a file to JSON format. Or, change the format of the logs written to the console or syslog to JSON using the following configurations, respectively:
log.console.formatter = json
log.syslog.formatter = json
If you want to revert these changes, that is, if you don’t want the log messages to be in JSON format, then simply comment out these configuration parameters.
RabbitMQ Logging Configuration
RabbitMQ offers three target options for logging messages: to a file, the console, or syslog. By default, RabbiMQ directs all log messages to files. You can use the RabbitMQ configuration file to control this.
Logging to File
The following configuration will enable or disable RabbitMQ logging to a file:
log.file = true
There are numerous other configuration parameters for tuning RabbitMQ logs to files. For example, you can control the log level to files, create separate files for each log category, use the built-in log rotation feature, or even integrate with a third-party log rotation package, as discussed above. When you want to stop logging messages this way, all you have to do is change the configuration value to false.
Logging to Console
Logging to the console is similar to logging to a file. You have a configuration in the RabbitMQ configuration file to either enable or disable logging to the console. The configuration is as follows:
log.console = true
As with logging to a file, you can change the value to false to stop the logging. Also, there are numerous options for formatting log messages sent to the console.
Logging to Syslog
Logging to syslog takes a bit more configuration than logging to a file or the console. Once you enable logging to syslog, you also have to provide other related configurations, like the transport mechanism (TCP or UDP), transport protocol (RFC 3164 or RFC 5424), syslog service IP address or hostname, port, TLS configuration if any, and more.
To enable logging to syslog, use the following configuration:
log.syslog = true
By default, RabbitMQ uses the UDP transport mechanism on port 514 and the RFC 3164 protocol. To change the transport and the protocol, use the following two configurations:
log.syslog.transport = tcp
log.syslog.protocol = rfc5424
If you want to use TLS, do so using the following configuration:
log.syslog.transport = tls
log.syslog.protocol = rfc5424
log.syslog.ssl_options.cacertfile = /path/to/ca_certificate.pem
log.syslog.ssl_options.certfile = /path/to/client_certificate.pem
log.syslog.ssl_options.keyfile = /path/to/client_key.pem
Next, you need to configure the syslog IP address or hostname and the port using the following configuration:
log.syslog.ip = 10.10.10.10
log.syslog.host = my.syslog-server.local
log.syslog.port = 1514
Along with these, you can also change the format of the log messages to JSON as we discussed earlier.
Sample Logging Configuration
## See https://rabbitmq.com/logging.html for details. ## Log directory, taken from the RABBITMQ_LOG_BASE env variable by default. # log.dir = /var/log/rabbitmq ## Logging to file. Can be false or a filename. ## Default: # log.file = rabbit.log ## To disable logging to a file # log.file = false ## Log level for file logging ## See https://www.rabbitmq.com/logging.html#log-levels for details ## # log.file.level = info # log.file.formatter = json (Configure JSON format)
## File rotation config. No rotation by default. ## See https://www.rabbitmq.com/logging.html#log-rotation for details ## DO NOT SET rotation date to ''. Leave the value unset if "" is the desired value # log.file.rotation.date = $D0 # log.file.rotation.size = 0 ## Logging to console (can be true or false) # log.console = false ## Log level for console logging # log.console.level = info ## Logging to the amq.rabbitmq.log exchange (can be true or false) ## See https://www.rabbitmq.com/logging.html#log-exchange for details # log.exchange = false ## Log level to use when logging to the amq.rabbitmq.log exchange # log.exchange.level = info
Monitoring RabbitMQ Logs
Monitoring your RabbitMQ logs in real-time can help you identify issues before they become critical. There are several monitoring tools available, such as AppDynamics, DataDog, Prometheus, Sematext,….
Monitoring RabbitMQ with Sematext
Troubleshooting RabbitMQ Logs
Managing RabbitMQ logs is an essential part of maintaining a reliable and efficient RabbitMQ setup. However, even with the best logging practices in place, issues can still arise.
I usually work with the RabbitMQ logs follow these steps:
- Check the Log Files Regularly: Checking the logs on a regular basis can help you identify any issues as they arise, allowing you to address them promptly, also help you to track trends and identify patterns of behavior that may indicate potential issues.
- Look for Error Messages: When reviewing your RabbitMQ logs, pay close attention to any error messages that appear. Error messages can provide valuable clues about potential issues with your RabbitMQ setup.
- Check RabbitMQ Status: Using the RabbitMQ management console or command-line tools, check the status of your RabbitMQ instance by typing:”rabbitmqctl status”
Look for any warnings or errors that may indicate potential issues. Address any warnings or errors promptly to prevent them from becoming larger issues down the road.
- Check Network Connections: Check network connections and firewall settings to ensure that they are properly configured. Network issues can cause a wide range of issues with your RabbitMQ setup, so it’s essential to make sure that your network is configured correctly.
- Monitor Resource Usage: Monitor resource usage on your RabbitMQ server, including CPU, memory, and disk space. If any of these resources are running low, it can cause issues with your RabbitMQ instance.
Remember to set appropriate log levels, use a monitoring tool to keep track of your logs, and be proactive in identifying and resolving issues. With these practices in place, you can make sure that your RabbitMQ system is reliable and performing at its best.
If you have further questions, our team of expert consultants are on hand and ready to chat. Just head to our contact page.