MongooseIM 3.3.0: Supporting happy relations
by Piotr Nosek, Mateusz Bartkowiak
Have you ever tried to use Mnesia with datasets larger than your transient memory? Are you confused with Erlang data types stored within Mnesia? We know some of you were. That is why we answered these problems by introducing something that is familiar to mainstream developers and also efficient with larger datasets at the same time - i.e. RDBMS backends for our PubSub plugin. Now, lets bundle that up with full support for XEP-0178, as well as, a RabbitMQ backend for our Event Pusher for a more complete package. Ladies and gentlemen welcome MongooseIM version 3.3.0.
Relations meet PubSub
The Publish-Subscribe plugin was one of our main focal points for recent weeks thanks to our friends from Safaricom Alpha, who have sponsored this work. The rationale behind implementing it is simple, and yet not obvious, so let’s dig in.
The XMPP protocol features an extension called Personal Eventing Protocol which is a special case of PubSub, where any entity (e.g. a user) may become a PubSub node. In turn, this can be used for many purposes. For instance, a client may publish information to its contacts about the music currently being played. Alternatively, it may announce microblog entries (e.g. personal Twitter). Nevertheless, end to end encryption is the thing of importance to us and many of MongooseIM users in regard to PEP.
End to end encryption is often a selling point for many pieces of modern IM software. For example, the popular TLS encryption ensures that nobody will be able to eavesdrop on your communication with a bank website. Of course, there are many more properties but the crucial fact remains: it secures the data you exchange with the server. Therefore, when you connect to MongooseIM with TLS enabled your transmission is safe. You have to bear in mind that everything you write is still readable on the server side and if you would like to improve your privacy even further - you need to encrypt your message content as well.
There are several protocols you can use to achieve it, but in most cases, you will need a way to announce public data (e.g. public key) that others may use to establish a secure session with your device. This is where PEP comes in. It provides a facility for storage and distribution that can hold, for instance, OMEMO keys and metadata to your contacts.
Since this data is retrieved and updated fairly often, we have realised that a classic Mnesia backend is no longer sufficient for this purpose and we have put a lot of work into developing an efficient RDBMS backend for PubSub. Besides performance in high volume scenarios, it allows developers to use databases other than Mnesia, ones they are more familiar with.
During the development, we have been also able to pinpoint bottlenecks in the core PubSub code. That is why we have the parallelised distribution of PubSub messages.The pre-3.3 extension used only a single Erlang process to handle all requests, as well as, the broadcasts triggered by them. Currently, the notification distribution is done by a new, short-lived process for every request. Requests themselves are still processed in a single queue - since parallel execution led to transaction deadlocks in Mnesia - but the extension may be configured to process them in several queues. That fixes the issue we have observed on several occasions, where the PubSub process was simply overwhelmed with messages with no reasonable overflow control and back pressure what greatly impaired user experience.
Password-based authentication is still a basic method in many places. Why? Try remembering your RSA 4096-bit public key, not to mention the whole public+private key pair. (yes, we know about Keepass; but you most probably have it configured to use a master password, right?)
The PKI authentication is the current industry standard though. It is more secure than a private-public key pair and, if implemented properly, much more convenient for the average user to use. Support for this method debuted in MongooseIM 2.2.x and received a batch of improvements in every subsequent release. This one adds one of the last important pieces i.e. compliance with XEP-0178, which is an official PKI authentication method specification for XMPP. It describes how the SASL EXTERNAL mechanism should behave. In other words, it describes which certificate fields should be used in the process and how.
Pre-3.3 implementation verified only the Common Name field, while now it verifies xmppAddr fields (there may be more than one such field) with CN optionally used as a fallback. What is more, a full JID is verified instead of just the username.
Integration with RabbitMQ
The Event Pusher extension emerged in MongooseIM 2.1.x as a unification of several channels that our server used to deliver data to external endpoints - e.g. delivering user messages to a push notifications service. It has been extended over time and in MIM 3.3 it receives yet another backend: RabbitMQ.
It is especially beneficial for developers who need to digest IM events in an asynchronous manner. Since AMQP is a popular, powerful and pretty easy to learn protocol, it may be used to build a spam detection component. Events published via RabbitMQ may also be consumed for big data analysis (finding patterns in user preferences, behaviour etc.). These are only 2 examples. We imagine that every application may find innovative uses for a stream of events coming in from MongooseIM. What is more, using another Erlang-based piece of software ensures the reliability and performance of this tandem.
Consider the demo of the spam detection mechanism below to be an inspiration for you.
Please feel free to read the detailed changelog. Here, you can find a full list of source code changes and useful links. Contributors Special thanks to our contributors: Test our work on MongooseIM 3.3.0 and share your feedback
- Help us improve the MongooseIM platform:
- Star our repo: esl/MongooseIM
- Report issues: esl/MongooseIM/issues
- Share your thoughts via Twitter
- Download Docker image with new release
- Sign up to our dedicated mailing list to stay up to date about MongooseIM, messaging innovations and industry news.
- Check out our MongooseIM product page for more information on the MongooseIM platform.