In the previous blog article, I focused on designing performance test suites for Ruby on Rails applications. I made an attempt to capture the differences between performance, load and stress testing. I also discussed some of the issues which frequently need to be addressed during the testing process. I would like to devote this post to benchmarks and tools – the topic which deserves a more in-depth treatment.
A variety of available solutions
Regarding performance and benchmarking tools, a large number of solutions available makes the choice hard indeed. The value of the data collected during performance testing depends, among other things, on the quality of the toolset used for testing. There is no straightforward answer as regards the selection of testing tools if only for the fact that each tool comes with specific characteristics which make it more / less suitable for specific scenarios. There are benchmarks which measure your application performance and which can be launched by inputting just one command in the console. They usually hit your website with series / number of requests. Thus the benchmark resembles a DOS attack rather than a performance test. Since that is rarely the purpose of performance testing, I am inclined to use more sophisticated solutions which allow you to write “user stories” which reflect the business flow of a given web application. It all boils down to your priorities – what is more important:
- to have confidence that you Ruby on Rails application can sustain a given load and properly perform the tasks it was specifically built for (i.e., the tasks reflecting the business logic) OR
- to find out how your Ruby on Rails application scores against some standard industry benchmarks (e.g. what is your application throughput while serving the homepage)?
On the whole, I attach more importance to the former. Still, it is sometimes the case that one needs to tune up specific pages which apparently are a bottleneck in the application. Alternatively, the SLA / performance requirements are tied to some standard industry benchmark tools and thus one needs to monitor the divergence from such an agreed benchmark. Last but not least, a project stakeholder may just expect you to deliver the data gathered by means of a particular tool. Knowing the tools available well provides you with more options with which to address the problem. Various performance / load benchmarks offer different sets of features and exhibit specific strengths and weaknesses. It is desirable to simulate the user traffic in as a realistic manner as possible. While selecting the solutions for comparison, I gave preference to the benchmarks that are popular, easy to use, reliable or innovative in their approach to user traffic simulation. The following solutions found their way into the list:
- Apache Bench,
- tsung,
- httpserf,
- httpsload,
- Trample
- JMeter,
I also added a follow-up section describing performance monitoring tools for Ruby on Rails applications, like New Relic.
Apache Bench
I think the reason behind the popularity of Apache Bench is the fact that it is packaged with a very popular server: the Apache web server. I cannot see many other reasons why Apache Bench has been so widely adopted; the solution does not offer a way to simulate a collection of user sessions performing various actions throughout the system. All the requests produced by Apache Bench are identical. It is not possible to write down a user story, for example: “A user visits home page, clicks on a featured product, adds it to the cart and proceeds to the checkout”. This is a major limitation of AB, but at least benchmarking Ruby on Rails applications with Apache Bench is a breeze. If you are on a Linux machine, all you need to do is install apache2-utils and feed AB with something to “chew”. For the sake of simplicity, I prepared a sandbox Spree 1.1 running on:
- Ruby on rails 3.2.3,
- Ruby 1.9.3-p194,
- the Thin web server,
to facilitate the demonstrations that follow. The landing page of the aforementioned application can be found here. Let’s see what happens if we type in the following command in the console: $ ab -n 250 -c 25 https://localhost:3000/ I have instructed Apache Bench to send 250 requests in total and to perform 25 requests at a time.
I find the httpserf output a little bit more informative than that delivered by, for instance, Apache Bench. Httperf also seems to be richer in terms of features. It is definitely worth a trial. By rule of thumb you should not run load / stress testing benchmarks on the same machine you run a Web Server on. Removing the network factor can lead to false expectations as to what the end-user experience is like. Besides, frequent benchmark runs are resource-intensive. You can skew your results if there is competition for system resources between the benchmark tool and your Ruby on Rails application. Still, if for a reason the network infrastructure is of no concern for you, you may arrive at solid results by using a benchmark tool that is generally considered ‘lightweight’; lightweight enough to run on the same machine with your application. For this reason, I picked https_load – a multiprocessing https test client; it runs in a single process and, therefore, it does not bog down the client machine.
I have not removed anything out of the output. As you can see, there is little data to examine here. When it comes to testing philosophy and features, httpsload is very close to Apache Bench: it does only throughput tests, which, in my opinion, are suitable for some purposes. However, Apache Bench output provides more useful information. Httpload is more like DOS attack; it generates a specified number of requests every second without waiting for the previous requests to complete. The process can generate a lot of noise in your server log files as the web server may fail to answer non-existent requests.
Trample
Being a Ruby programmer, I could not escape the feeling that some other fellow Rubyist must have tackled the problem “in a Ruby way” before. “Wouldn’t it be great to be able to write performance and load testing scenarios in the same manner as we write our test suites with Rspec, Capybara and the like?” My gut feeling led me to discover Trample. Find an up-to-date fork of the project here. Trample is a load simulator whose main advantage is the randomization of the requested urls. The characteristic allows mimicking the real-world traffic more closely. I had some trouble installing Trample as the gem gets installed without dependencies. The moment I sorted it out, I started writing my first load test in Trample.
Running a performance test is as simple as invoking trample start filename.rb in the console. Trample looks promising, but it lacks a few things:
- a way to present the test results in a sensible manner; the current output from the console is far from informative;
- the ability to group requests in transactions; something similar to JMeter transaction controller, for instance. Trample requests are fired up in random order.
- cookie and session management (present in JMeter).
Still, the idea to write performance / load and stress tests in a way one writes unit and integration tests is powerful. If anyone knows about similar solutions for Ruby, share the knowledge in the comments section below the post, please. A follow-up to the benchmarking tools comparison: web applications performance management tools. If you are doing performance tuning or you just want to keep track of your application performance over time, the ability to display and compare the results obtained is a significant factor to consider when selecting a benchmark tool for a Ruby on Rails application. Choose benchmarks which facilitate creating baselines. It is also a good idea to include web application performance monitoring and management tools in the performance test suite for a Ruby on Rails application. The tools like New Relic operate on real production data and provide real time insights into:
- response time and throughput,
- errors and availability,
- user satisfaction,
- server resource utilization.
Although the tools like New Relic cannot provide the information gathered during performance / load and stress testing, you can use New Relic to validate and further improve your performance test suite. With such tools at hand, you obviously gain the regular benefits of 24/7 monitoring of your application performance in the production environment.
Summary from Ruby on Rails developer’s perspective
I hope this blog post together with the previous article on designing performance / load test suites shed some light on the common problems and solutions that Ruby on Rails development teams face while designing and running performance test suites for their applications. There are different tools for different jobs. If you need to conduct simple throughput testing and/or measure the responsiveness of particular pages in the system, the tools like Apache Bench or httpserf will do the trick. If, however, your performance test suite is to reflect the business logic of the application, you should look for tools like JMeter, i.e., those which allow you to combine requests into transactions creating “user stories”. The choice of benchmarking tools is often a compromise between a number of factors such as: ease of use, features, and time needed to prepare and maintain a test suite. You may find out that, for example, a tool delivers outputs which are very useful and easy to read, but, unfortunately, it is unable to saturate your web application in the desired way, or has other limitations which do not satisfy your requirements. While experimenting with benchmarks, you should limit yourself to using a limited number of them. The greatest value of benchmarking comes from the ability to compare the application performance parameters over time. Stick to the key issue and you may be able to use the right number of tools. Creating and maintaining tests suites for different benchmarking tools increases the costs of performance testing, the costs which may not necessarily be offset by the associated benefits.