10x reduction in the time it takes to update the site
The system easily handles over 200 million push notifications per day
150 servers were reduced to just 8
Bleacher Report is the leader in real-time, social-first, mobile-first sports content provider, reaching over 250M people each month. In the sports news cycle, activity can spike at any time and a platform that can support all kinds of traffic bursts and works at scale is essential. The creation of its mobile app, Team Stream, was a turning point for the company. Originally written in Ruby, the app started presenting technical challenges at scale.
The development team was dealing with ever increasing complexity in their caching strategy to get the app to serve the huge amount of real-time content to the rapidly growing user base. In addition to the huge numbers of users, the content in Team Stream is both real-time and personalised, which makes for heavy merging and sorting on demand. This does not lend itself to an aggressive caching strategy, so tests are needed for any technical implementation. This is especially true at times of high traffic, which are frequent for a popular sports app.
The team investigated a number of different programming languages, but it was the combination of the Erlang Virtual Machine (VM) and Ruby syntax that made Elixir a compelling choice. The Erlang VM gave them the scalability, concurrency and reliability needed to handle traffic spikes without breaking, whilst the Ruby syntax reduced the barrier to entry and the ability to get up to speed quickly. But while Elixir and Ruby look similar, the underlying characteristics are different. In effect, you can take good Ruby, translate it into Elixir and it can become bad code. Bleacher Report first engaged Erlang Solutions after they completed their first production system. They wanted to make sure their code was sound and they hadn’t done anything that might cause pain down the road.
- Our team spent time on site to review Bleacher Report’s systems and helped to educate the development team on best practices.
- One of the first things we discovered was that Bleacher Report’s use of Docker containers was proving inefficient. With our help, they were able to significantly reduce the build and deployment times.
- We went on to review the code on a regular basis and provided a much deeper view on how the VM works.
- We helped Bleacher Report achieve a different way of thinking about concurrency and parallelism that helped them simplify their code.
- We also helped clear bottlenecks that were appearing downstream through a process of chunking the data.
I spoke with two CTOs I know who utilize Erlang and they both recommended Erlang Solutions. They were able to come in with their expertise, help us establish best practice and give us confidence that going forward our systems would be efficient and reliable.
The Bleacher Report dev team isn’t as concerned about releases, even on the biggest nights of the year such as the NFL Draft. Changes can be made with a few lines of code and it doesn’t need to split its expertise across multiple technologies or depend on third party clients or integrations.
Thanks to the Elixir shift, Bleacher Report now enjoy:
- A significant reduction in code complexity
- A significant decrease in time taken to develop the code
- A significant cost savings
- Time and resource freed up to address the technical debt
- Excellent availability of coding expertise having converted their Ruby team into Elixir experts.
Ability to handle 8x the traffic without autoscaling
95th percentile latency hovers around 100ms and is not noticeably affected by traffic spikes due to breaking news or other events
Adding content to all streams went from 30-40 seconds to 3-4 seconds
Resource intensive features, previously requiring aggressive horizontal scaling, now run on about 1/10th of the servers with low CPU utilisation