Erlang

MongooseIM 2.1.0beta2 - Expanded Platform, first class quality

by Piotr Nosek

MongooseIM is a platform for businesses to build messaging and social apps, or add messaging and social features to existing apps. When you need to adapt to growing user demand, we’re here to guarantee that your service scales, grows with new functionalities and remains reliable for all.

The fullstack MongooseIM platform offers various components, both for the backend (Erlang, Elixir) and the frontend (iOS, Android, Web) for you to pick up and assemble like pieces of a puzzle.

Latest beta release of MongooseIM 2.1.0 is here!

This time, we focused on improving the source code and documentation quality. Everything to make your MongooseIM deployment easier to maintain and more stable.

Our team wants to ensure that the next version will be as mature as possible, with accurate and easy-to-read documentation, and friendly source code for developers; free of known bugs and issues.

Beta2 is also time to welcome our new ICE/STUN/TURN component, which has almost reached its stable version by now.

And by the way… Hi! I’m Piotr and I’m going to be your guide in the tour of beta2!

Read The Fine Manual

It’s easy to forget about documentation when you’re chasing after new features, improvements and world-changing pull requests. When that does happen, you run the risk of users not appreciating the new shiny module you prepared for them - mainly because they do not know what it is or how to configure it.

We understand it is a challenge to maintain up-to-date, high quality documentation of a project of MongooseIM’s size. It is one of our primary tasks for 2.1.0 release to pay off this debt and ensure that no important information piece is missing. It involves revising docs about every module and configuration description correctness. We are verifying the validity of high level concepts and guides.

Every GitHub issue, that narrows down to a configuration problem, is a clear indication what we should improve. It’s an ultimate goal to ensure that every MongooseIM user will be able to configure the server without any doubts on which option to use and how! It is our priority to ensure that our users always find the information they seek, no matter if these are abstract concepts, design decisions and development guidelines.

In beta2 we continue to work towards a high quality documentation with 50% of docs being reviewed and already improved. We aim at 100% by the 2.1.0 release freeze!

MongooseICE

Nowadays it’s not so easy to have each user occupy public IP addresses. In most cases both users are behind NATs, which effectively prevents them from establishing a peer-to-peer connection. It means no file transfers (except for e.g. inefficient In-Band Bytestreams), no video or voice calls. What can an application provider (or just two users connecting to public XMPP clusters) do under these conditions? ICE/STUN/TURN enables two clients to establish a peer-to-peer connection and exchange data, in a manner that is cheap and efficient.

You can find more information about our new backend service in the MongooseIM family esl/fennec. Yes, the link is correct. Fennec is simply the codename until its first stable release.

Bloated-stanza-proof

Beta2 features an important security update, which moves the stanza size check from Erlang code to the XML parser. It means it’s no longer possible to fill the server’s memory with an enormous stanza. Before this update, an oversized XML was discarded of course, but only after parsing, so the memory was already allocated and the damage was already done.

Because of this vulnerability, a malicious user could trigger out of memory exception and crash the server. Beta2 prevents it.

Goodbye, deprecation!

Erlang/OTP evolves over time, so does its API. Some functions become deprecated for good reasons. In beta2 we’ve got rid of erlang:now/0 and crypto:rand_bytes/1 calls.

The highlights of this change are:

  • Better performance (less Erlang VM locks)
  • More secure tokens and authentication salts

OK, now it gets technical. :)

The former function has been in use for a very long time but, as server hardware provided more and more CPU cores, this BIF’s weaknesses had to be repaired. First of all, now() call guarantees returning strictly monotonic values on a single machine, so it requires synchronisation between all CPU cores. Obviously it means a performance penalty. What is more, in extreme cases simultaneous calls to this function from multiple schedulers could shift the timer more and more into the future. We’ve replaced it with new time API, choosing appropriate BIFs for every use case (now() used to be “one size fits all”).

It is a bit tricky in case of the Message Archive Management though. Message IDs are heavily based on timestamps and we couldn’t risk ID duplication, so the microsecond part is replaced with an arbitrary integer (in the 0-999 range, of course) to reduce a chance of collision. It limits message ordering precision to a millisecond, so we are still exploring other possible timestamp sources. We’d appreciate all feedback from MAM users on this new method. For those who would like to learn technical details of these solutions: feel free to check the link to a pull request in the changelog. Look for the mod_mam_utils.erl file.

The latter change, crypto:strong_rand_bytes/1 is all about security. Instead of an internal pseudo-random number generator, it calls a RAND_bytes method from OpenSSL.

One deprecation remains: random module. It will be replaced with rand in MongooseIM source code once we stop supporting OTP 17.5 in favour of the latest version from the 20.x line.

True beauty lies within

A stanza’s tale

Amongst many internal technical changes, mongoose_acc is one of the most significant ones. With beta2, stanza’s whole life’s history is recorded inside a common structure. Although it was introduced quite some time ago, at last it is not unpacked in the middle of processing. It is created when a stanza arrives and lives until the end of processing.

If you haven’t heard about it yet: what if I told you that a stanza is not just some internal XML representation passed around by processes? What if the hook execution results were not lost? These can be privacy & AMP check results, response to an IQ or just some info extracted for convenience (e.g. from, to attributes). Handy, right? That’s what mongoose_acc is!

For now it provides more information in logs and easier access to some parameters, but in the future it will, for example, allow us to perform advanced stanza tracing on demand.

The cost of passing a bigger structure around is balanced by the fact that some operations are faster (e.g. no need to query the XML structure to get some commonly accessed attribute).

Rock ‘n’ Roll

Some of you, especially contributors, may have already met Elvis - our King of Code Style. Obviously we’ve always paid attention to our coding style but we’re humans - it’s not difficult to nest expressions too much or abuse macros. Now we have an automatic checker that ensures we don’t omit anything. It verifies every pull request and mercilessly points out every mistake.

We not only care for the new code, but for the existing one as well. Thus, we’ve refactored a large part of the existing codebase to meet these standards as well. It obviously brings value to us, the developers. It includes not only the MongooseIM team, but all other contributors as well. By eliminating ugly or overly complicated parts we increase the pace of future work and make all these modules and functions easier to read and understand. This has a direct benefit for our whole community, as all the new features will be developed faster and bugs will appear less and less frequently!

It’s the little things…

Deeper into REST world

MongooseIM’s client REST API grows. With beta2 the roster management is now possible via this interface. This improvement is yet another result of our efforts to help the XMPP world cross the boundaries between protocols. You can find more information in our beautiful Swagger documentation (see “Contacts” group).

Silent push notifications

MongooseIM is able now to send “silent” push notifications. They enable client applications to receive data via this channel without any information shown to the human user. It is fully up to the application to handle this data and decide on the action.

This feature requires MongoosePush 0.9.0 component or newer.

Remaining technical improvements

There are other new features worth checking out. Some of them are beneficial to MongooseIM users, others can make MongooseIM developers’ and contributors’ lives easier.

  1. mod_vcard supports RSM in queries now. It means that vCard search results can be paginated now.
  2. You can choose whether Message Archive Management should archive groupchat messages in users’ private archives or not. A default behaviour so far was to store them anyway.
  3. One of our Travis CI jobs executes unit & integration tests on OTP 19.3, effectively ensuring MongooseIM compatibility with latest stable OTP version.
  4. New command available in mongooseimctl: import_users. It takes a CSV file (with user,domain,password columns) as input and I think it’s pretty obvious what it does then. :)
  5. Requesting a roster entry from mod_roster backend may now return a does_not_exist atom.
  6. gen_mod will print a warning message when a module’s start/2 callback links to the caller. It is a discouraged practice, because it can cause problems when starting modules via RPC or in console.

Load tests

Our standard Tide tests results in 1-1 messaging show no difference in terms of latency between beta1 and beta2.

Changelog

Please feel free to read the detailed changelog, where you can find a full list of source code changes and useful links.

Contributors

Special thanks to our contributors: @astro @strugee @msantos @daniel-e @deadjdona!

What’s next?

Final release is coming!

It’s the last stretch for bug fixing or improving performance, documentation and style. We anticipate the feature set to remain stable until our final release, but surprises do happen so stay tuned for new tweets and new PRs!

We strongly encourage everyone to join our wonderful circle of contributors. Every PR is welcome, no matter whether it’s a fix in Erlang code, Makefile fix or new documentation section.

Spoilers!

  • ODBC backend for mod_muc.
  • SSL support for MySQL connection.
  • Better JWT authentication support.

But ssssssh! You haven’t heard it from me, allright? :)

Test our work on MongooseIM 2.1.0beta2 and share your feedback

Help us improve the MongooseIM platform:

Sign up to our dedicated mailing list to stay up to date about MongooseIM, messaging innovations and industry news.

Check out our MongooseIM product page for more information on the MongooseIM platform.

Go back to the blog

×

Request more information:

* Denotes required
×

Thank you for your message

We sent you a confirmation email to let you know we received it. One of our colleagues will get in touch shortly.
Have a nice day!