Page 1
1
Optimising
Optimising TCP/IP connectivity
TCP/IP connectivity
An exploratory study in
An exploratory study in socket
socket-intensive
intensive Erlang systems
Erlang systems
ACM
ACM Sigplan
Sigplan Erlang Workshop
Erlang Workshop
Freiburg,
Freiburg, Germany
Germany, October
October 05, 2007
05, 2007
Oscar Hellström
oscar@erlang-consulting.com
Page 1
Contents
Contents
• Erlang
Erlang based
based TCP/IP intensive
TCP/IP intensive applications
applications
• Stress testing
Stress testing Ejabberd
Ejabberd on
on inexpensive
inexpensive machines
machines
• Benchmark
Benchmark results
results
• Conclusions
Conclusions
Page 2
2
Page 2
Introduction
Introduction
• Erlang TCP/IP intensive applications are common
Erlang TCP/IP intensive applications are common
– HTTP (YAWS, ETC Web Platform)
– Instant Messaging (ejabberd)
– Networking (RabbitMQ)
• Scalability of Operating Systems
Scalability of Operating Systems
– Old connectivity benchmarks floating around [1]
– Rapidly changing TCP/IP stacks and OS kernels
– Still not really what the paper is about
Page 3
Introduction
Introduction
• Scalability of Erlang
Scalability of Erlang
– Has been investigated in a number of papers [2]
– Network scalability has not been addressed
– How to combine Erlang applications with a suitable OS?
– How to optimise runtime settings of TCP/IP intensive
applications?
Page 3
3
Page 4
Research approach
Research approach
• Benchmarking Ejabberd
Benchmarking Ejabberd
– Tsung, a multi-protocol load testing tool
– User scenarios
• 20% idle users
• 60% semi-active users
• 20% active users
• Major Operating Systems
Major Operating Systems
– Linux (SuSE 9.3)
– FreeBSD 6.2
– NetBSD 4.1
– Solaris 10
– Windows port of Erlang does not support kernel poll
Page 5
Research approach
Research approach
• Low end benchmark machines
Low end benchmark machines
– Intel Celeron
– 2.4Ghz
– 512 MB RAM
• 2 Tsung machines
2 Tsung machines
• 1 Ejabberd machine
1 Ejabberd machine
• 100 switched Mbit LAN
100 switched Mbit LAN
Page 4
4
Page 6
Research approach
Research approach
• Erlang Run
Erlang Run-Time system settings
Time system settings
– Basic installation
– Asynchronous threads
– Kernel-polling
• OS TCP stack optimisations
OS TCP stack optimisations
– Window Size
– Selective Acknowledgments (SACK)
Page 7
Results
Results
Page 5
5
Page 8
Results
Results – Concurrently connected users
Concurrently connected users
17300
17150
16994
Solaris 10
19971
17058
16581
NetBSD 3.1
23898
17687
17855
SuSE 9.3
Kernel Poll
Async threads (50)
Default settings
17241
18248
16506
Solaris 10
19262
20308
20974
NetBSD 3.1
23034
23528
22593
SuSE 9.3
K + A (75)
K + A (50)
K + A (25)
Page 9
Results
Results
• Resource limits in the OS
Resource limits in the OS
– File descriptors per process
– Sockets per process
– Maximum heap size
• Resource limits in the Erlang virtual machine
Resource limits in the Erlang virtual machine
– Maximum default number of allowed processes
– Maximum number of ETS tables and ports
• Hardware constraints
Hardware constraints
– CPU
– Memory
How did we reach the limits ?
How did we reach the limits ?
Page 6
6
Page 10
Conclusions
Conclusions
• Operating Systems
Operating Systems
– Use SUSE LINUX for small scale systems
• Kernel
Kernel-poll is the must have optimisation
poll is the must have optimisation
– From a CPU bound system to a memory bound system
• Threadpool
Threadpool size is a could have optimisation
size is a could have optimisation
– When you need that last bit of performance
– Dependent on the application
• OS TCP stack optimisations
OS TCP stack optimisations
– Dependent on the network environment
– Dependent on traffic profile
• No impact on reliable IP networks
Page 11
Contents
Contents
• Erlang
Erlang based
based TCP/IP intensive
TCP/IP intensive applications
applications
• Stress testing
Stress testing Ejabberd
Ejabberd on
on inexpensive
inexpensive machines
machines
• Benchmark
Benchmark results
results
• Conclusions
Conclusions
Page 7
7
Page 12
Questions
Questions
Page 13
References
References
1.1. Felix von
Felix von Leitner
Leitner, Benchmarking BSD and Linux,
, Benchmarking BSD and Linux,
http://
http://bulk.fefe.de
bulk.fefe.de/scalability/
/scalability/
2.2. Nystr
Nyström J.H.
J.H. Trinder
Trinder P.W. King D.J.
P.W. King D.J.
High
High-level Distribution for the Rapid Production of
level Distribution for the Rapid Production of
Robust Telecoms Software: comparing C++ and
Robust Telecoms Software: comparing C++ and
Erlang
Erlang
Concurrency and Computation: Practice &
Concurrency and Computation: Practice &
Experience.
Experience.
Preprint:
Preprint:
http://www.macs.hw.ac.uk/trinder/papers/CPE2006.pdf
http://www.macs.hw.ac.uk/trinder/papers/CPE2006.pdf