Abstract

The most powerful Erlang based web server around is Yaws[9]. It was released as an open source in 2002 and implements most of the features met in today’s HTTP servers. What makes it really powerful, however, is its ability to execute dynamic pages with Erlang code.

In this paper, we discuss the lessons learned when internally developing HTTP based Erlang services on the application layer at Erlang Training and Consulting. The results of these development and prototyping activities resulted in a generic web platform, tools, and components also described in the paper. This generic development had to be developed on top of an existing infrastructure to make Erlang a suitable language to develop the glue and logic and back-ends of web based applications, moving the language a little closer to the Early Majority group of users.

Until recently, Erlang Training and Consulting (ETC) did not have any in-house systems development organisation. This started to change gradually in 2004 through its summer internship program, where second-year students were put to work on prototypes and tools needed internally within the company. On a very busy schedule, the senior staff did not have the time to review their code and give them the necessary feedback on a system which had an Erlang web based interface. The result was a system with no clear separation of the data model and dynamic generation of HTML. Experienced developers will know that data, code and HTML generation have to be separated to avoid inflexible, unsupportable code. Indeed, the prototype worked well, but any modification or extension in the data model or change in the HTML format, no matter how small, required a tedious rewrite of the code.

From one extreme to another, the second prototyping attempt resulted in the adoption of XSLT. XSLT is an XML based language which describes XML transformations, in our case, from XML to XHTML. Initially, XSLT with its generic way of transforming and formatting XML files seemed to ideally fit our requirements. Several pages presenting dynamic data were developed, but with time, the transformation turned out to be too formal and too strict. Merging it with other parts of the web page was inconvenient, and further development became an uneasy task. The largest problem resulted in very complex data transformations with multiple data sources. This, in turn, resulted in slow execution speed and the need of strict HTML, something not even the most popular HTML authoring tools produced. Our graphics designer was developing HTML that had to be passed through a linter and manually corrected. And to further convince us that this was not the way to go, the libxslt driver which we were using had memory leaks and no internal debug features. We quickly realised we had reached a dead end.

This is when we started developing our own web platform which would fit our requirements for a scalable architecture with dynamic content. The framework was started off as a prototype and tested internally on the erlang-consulting.com home page. Only after the successful migration of the site and extensive stress testing was the platform refactored and included in other commercial projects. Even though the commercial solutions had major time constraints and critical time to market requirements, Erlang, with its rapid development capability, allowed us to refactor the code and achieve the functionality and stability required by commercial systems.

During the development of ETC’s home page four requirements were identified. These are applicable to any web application.

The first version of the ETC home page was not an Erlang solution. Unfortunately, web hosts do not provide Erlang installations. The first version was implemented using the “traditional” Apache web server, Perl and PHP scripting languages. Data used to generate dynamic pages was stored in a MySQL database. Every time a new feature had to be added, the whole application became increasingly more complex and the data flow became harder to understand. As the complexity of the home page grew, reconstruction of the system was necessary. At that time the Web Platform concept was formulated and a new version of the home page was designed. Using the Web Platform and following the requirements specified above resulted in a more consistent architecture than the original PHP solution, and easier to maintain.

The Web Platform deals with the complexity by dividing the web site into static and dynamic content. Static content can be updated and changed during runtime but does not need to be processed in any specific way by a server. This includes all PDF documents, images, some CSS and HTML files. Such content is sent to a client using the transport layer of the Yaws server. Dynamic content is generated as a result of executing a function over data kept in a Mnesia table. The function takes GET and POST data, which together with wpart element attributes determine what kind of transformation should be applied.

Sets of functions generating dynamic content are developed as components. A component delivers functions to transform data of the same kind. As an example we can describe Erlang events that are short announcements about speeches, meetings or conferences potentially interesting for the Erlang community (see Figure 2).

The event component consists of:

In the table, data like date, location and description is stored. The modules implement transformation functions and a call-back function which is used to hook transformations to a web page. To make a decision about which transformation to execute, URL of the client request or a GET/POST variable can be used. The XHTML file is split into parts. Every part is used to present a different aspect of Erlang events or to present the same aspect in different ways. For instance, we can list all events or only a short summary about future events. A form, possibly password protected, allows a user to create or edit events.

To ensure a consistent appearance in a web site, reused elements are stored as generic templates (see Figure 3). This avoids the maintenance issues raised by cut and paste programming. Templates are implemented as a set of dynamic pages that can be included by a dedicated wpart element. Any data that should be formatted in a template is passed on with a request dictionary.

One of the signs of maturity within any engineering discipline is the standardization of tools and components used during construction and the availability of third party components, both open source and commercial. Two of the most well known web development languages today are PHP and Perl. One of the advantages they have over Erlang is the community behind them and the availability of reusable components.

Taking existing Perl and PHP based components off the shelf always results in compromises over functionality versus time to market. A component written in other programming languages might not have all of the desirable features listed in the requirement specification, and adding them has often resulted in a steep learning curve when having to understand the code. Owning the components will ensure that they do exactly what we required them to do, with the ability to adapt them to our needs and wants or the application.

Many of the components currently used were developed alongside the Web Platform, as it was constantly evolving with non backwards compatible interface changes. The components had to be made generic enough to cater for these changes, and were thus developed in two layers. One layer addresses the platform specific issues involved with the task that the components was designed for, and one layer which handles the generic features that could be reused throughout the development period. This ensured the components did not have to be rebuilt from scratch each time a new platform was released, future proofing our system for two-tier security architecture. The two software layer approach created reusable components, which if not requiring different format of the output, could be reused as they are. Two of the components created were the bulletin and the content manger. These components are currently being used in three websites, requiring very few, if any changes.  

The first component developed was the bulletin. It was originally only used for news announcements, but has since been expanded to be used as both an event and a job bulletin. While the code implementing the first layer of the component has changed to cater for its different uses, the core remained the same.

As problems being solved required more complex data models, the bulletin component was not powerful enough. This resulted in a general content management component, which through indexing allows data to be accessed and sorted using different keys. The complexity of the component, however, comes at a price. At the time of writing, it is still cumbersome to work with it, as there is no general way of accessing its data. As the Web Platform matures, the ways in which data can be retrieved from this component will be addressed.

A lack of dynamic records caused flexibility issues when front-end features were added or removed. The whole code had to be recompiled every time the record definition changed. Even if this is not a web-specific issue, dynamic record is a construct which would be much appreciated in Erlang, and welcomed by the community.

Other problems included the lack of strings as data types. The existing list implementation used to denote strings is highly inefficient and consumes too much memory, as it does not take advantage of the knowledge that it is storing an array of ASCII characters. Workarounds converting HTML to binaries for efficiency reasons, both memory and message passing wise, had to be made in the platform.

Lastly, the regular expression parser in Erlang is highly inefficient. If many regular expressions are used, an external parser has to be interfaced to the Erlang application, either through a port or a linked in driver. Having an efficient Erlang implementation would simplify the architecture and remove many of the issues associated with interfacing external programs.

In reality, time needed by a web browser to contact the server and download a page will be different for every user based on network latency, browsers and CPU power, so presented figures should be viewed with caution. The fact that the application on which we were running the stress tests was running on the same machine will furthermore have affected the results. What is interesting and important in these results is the comparison between dynamic pages and static pages, and the comparison between dynamic pages that use different components. The latter allows us to evaluate the complexity and efficiency of developed modules. For instance, the index.html page is generated using three components, while the jobs.html page uses only one. The number of components used in a dynamic page does not affect the performance, but the way the components are built and used has. In the jobs page, for example, sorting and filtering of the jobs results in the higher request time than the news page, where a similar amount of data is just displayed in chronological order. Also of interest is the fact that for simple dynamic pages, the results are very close to the static ones.

The Erlang web servers and databases are mature, as is the use of Erlang for glue and logic. What is lacking while developing interfaces for operation and maintenance of network systems built on Erlang/OTP, is a general framework for developing web based components, and the components themselves. It means a solid web development framework is needed which would reduce time and efforts spent on implementing components that follow similar design patterns.

At ETC the Web Platform framework was developed. The Web Platform delivers a generic concept of wparts used for implementing reusable web components. The next step is to identify generic web components and implement them using the Web Platform framework.

The learning curve and entry costs to develop the tools and components have been high. Now that we have reached this level of maturity within our organisation, we have the competitive advantage Erlang/OTP provides over the classic web application languages in the early and late majority adopters. The same can however be argued over Telecom based Erlang projects in 1993, when many of the existing predecessors of the OTP applications were being implemented in parallel, or did not exist at all.

The Web Platform is still in its early days, and still has a long way to go. Planned future work includes creating more generic components such as billing, blog, user registration, shopping carts and credit card/paypal components.

The infrastructure to allow a two-tier security architecture, where the front-end machine runs the web servers, parses the data and generates the HTML dynamic code, while the back-end machines handle the transactions and host the databases is in place, but still has not been implemented. The web components developed so far are ready for this migration thanks to the two-tier module approach. An RPC based API, which uses sockets to communicate with other nodes has already been implemented, and will be used as soon as there is a requirement for such a level of security.    

Finally, a Master’s Thesis is being planned to compare our web platform with Ruby On Rails. It will give us a better understanding of similarities and differences between the two solutions, and give us guidelines on what was done right, what could have been done better, and what has been missed. The results of the thesis will hopefully give assurances that Erlang, development of web-based applications, is slowly moving from the group of Early Adopters into the group of Early Majority users.