Radosław Jędryszczak, Technology


In the previous blog article I focused on designing performance test suites for Ruby on Rails applications. I made an attempt to capture the differences between performance, load and stress testing. I also discussed some of the issues which frequently need to be addressed during the testing process. I would like to devote this post to benchmarks and tools – the topic which deserves a more in-depth treatment. A VARIETY OF AVAILABLE SOLUTIONS As regards performance and benchmarking tools, a large number of solutions available makes the choice hard indeed. The value of the data collected during performance testing depends, among other things, on the quality of the tool set used for testing. There is no straightforward answer as regards the selection of testing tools if only for the fact that each tool comes with specific characteristics which make it more / less suitable for specific scenarios. There are benchmarks which measure your application performance and which can be launched by inputting just one command in the console. They usually hit your website with a series / number of requests. Thus the benchmark resembles a DOS attack rather than a performance test. Since that is rarely the purpose of performance testing, I am inclined to use more sophisticated solutions which allow you to write “user stories” which reflect the business flow of a given web application. It all boils down to your priorities – what is more important:

  • to have confidence that you Ruby on Rails application can sustain a given load and properly perform the tasks it was specifically built for (i.e., the tasks reflecting the business logic) OR
  • to find out how your Ruby on Rails application scores against some standard industry benchmarks (e.g. what is your application throughput while serving the homepage)?
On the whole, I attach more importance to the former. Still it is sometimes the case that one needs to tune up specific pages which apparently are a bottleneck in the application. Alternatively, the SLA / performance requirements are tied to some standard industry benchmark tools and thus one needs to monitor the divergence from such an agreed benchmark. Last but not least, a project stakeholder may just expect you to deliver the data gathered by means of a particular tool. Knowing the tools available well provides you with more options with which to address the problem. Various performance / load benchmarks offer different sets of features and exhibit specific strengths and weaknesses. It is desirable to simulate the user traffic in as realistic manner as possible. While selecting the solutions for comparison, I gave preference to the benchmarks that are popular, easy to use, reliable or innovative in their approach to user traffic simulation. The following solutions found their way into the list:
  • Apache Bench,
  • tsung,
  • httperf,
  • httpload,
  • Trample
  • JMeter,
I also added a follow-up section describing performance monitoring tools for Ruby on Rails applications, like New Relic.


I think the reason behind the popularity of Apache Bench is the fact that it is packaged with a very popular server:  the Apache web server. I cannot see many other reasons why Apache Bench has been so widely adopted; the solution does not offer a way to simulate a collection of user sessions performing various actions throughout the system. All the requests produced by Apache Bench are identical. It is not possible to write down a user story, for example: “A user visits home page, clicks on a featured product, adds it to the cart and proceeds to the checkout”. This is a major limitation of AB, but at least benchmarking Ruby on Rails applications with Apache Bench is a breeze. If you are on a Linux machine, all you need to do is install apache2-utils and feed AB with something to “chew”. For the sake of simplicity, I prepared a sandbox Spree 1.1 running on:
  • Ruby on rails 3.2.3,
  • Ruby 1.9.3-p194,
  • the Thin web server,
to facilitate the demonstrations that follow. The landing page of the aforementioned application can be found here. Let’s see what happens if we type in the following command in the console: $ ab -n 250 -c 25 http://localhost:3000/ I have instructed Apache Bench to send 250 requests in total and to perform 25 requests at a time. Let’s carry on:
This is ApacheBench, Version 2.3 <$Revision: 655654 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking localhost (be patient) Completed 100 requests Completed 200 requests Finished 250 requests Server Software:        thin Server Hostname:      localhost Server Port:                3000 Document Path:         / Document Length:      12722 bytes Concurrency Level:  25 Time taken for tests:   60.947 seconds Complete requests:    250 Failed requests:         0 Write errors:               0 Total transferred:        3332250 bytes HTML transferred:      3180500 bytes Requests per second:           4.10 (mean) Time per request:       243.788 received Connection Times (ms)             min  mean: min 11100.2 avg 16389.6 max 26983.4 median 14637.5 stddev 4902.7 Connection time : 2.242 Request rate: 2.8 req/s (360.9 ms/req) Request size : min 0.0 avg 1.2 max 2.8 stddev 1.0 (12 samples) Reply time : header 612.0 content 12722.0 footer 0.0 (total 13334.0) Reply status: 1xx=0 2xx=74 3xx=0 4xx=0 5xx=0 CPU time : min 0.00 avg 0.08 max 0.20 stddev 0.06 (5/100) Session: avg 1.00 connections/session Session lifetime : 7.9 Session length histogram: 67 13 9 6 0 5
I find the httperf output a little bit more informative than that delivered by, for instance, Apache Bench. Httperf also seems to be richer in terms of features. It is definitely worth a trial. httpload By rule of thumb you should not run load / stress testing benchmarks on the same machine you run a Web Server on. Removing the network factor can lead to false expectations as to what the end-user experience is like. Besides, frequent benchmark runs are resource intensive. Your can skew your results if there is competition for system resources between the benchmark tool and your Ruby on Rails application. Still if for a reason the network infrastructure is of no concern for you, you may arrive at solid results by using a benchmark tool that is generally considered ‘lightweight’; lightweight enough to run on the same machine with your application. For this reason  I picked http_load – a multiprocessing http test client; it runs in a single process and, therefore, it does not bog down the client machine. Httpload has not been actively developed for some time now, but I found a fork of the project with a very good readme. Here is some sample output:
socjopata@socjopata-pc:~$ httpload -rate 10 -seconds 60 url.txt 205    0    53258    3000259    31429676    200 206    0    53259    3000326    31336053    200 232    0    53285    3000391    31371277    200 221 fetches, 379 max parallel, 2.81156e+06 bytes, in 60.0003 seconds 12722 mean bytes/connection 3.68332 fetches/sec, 46859.2 bytes/sec msecs/connect: 108.609 mean, 3000.39 max, 0.121 min msecs/first-response: 18502.2 mean, 38271.5 max, 258.467 min HTTP response codes:  code 200 — 221
I have not removed anything out of the output. As you can see, there is little data to examine here. When it comes to testing philosophy and features, httpload is very close to Apache Bench: it does only throughput tests, which, in my opinion, are suitable for some purposes. However, Apache Bench output provides more useful information. Httpload is more like DOS attack; it generates a specified number of requests every second without waiting for the previous requests to complete. The process can generate a lot of noise in your server log files as the web server may fail to answer non-existent requests. Trample Being a Ruby programmer, I could not escape the feeling that some other fellow Rubyist must have tackled the problem “in a Ruby way” before. “Wouldn’t it be great to be able to write performance and load testing scenarios in the same manner as we write our test suites with Rspec, Capybara and the like?” My gut feeling led me to discover Trample. Find an up-to-date fork of the project here. Trample is a load simulator whose main advantage is the randomization of the requested urls. The characteristic allows to mimic real world traffic more closely. I had some trouble installing Trample as the gem gets installed without dependencies. The moment I sorted it out, I started writing my first load test in Trample:
Trample.configure do  concurrency 5  iterations 10  get “″  get “”  login do             post “” do             {:user_email => “socjopata@gmail.com”, :user_password => “secret”}             end  end end
Running a performance test is as simple as invoking trample start filename.rb in the console. Trample looks promising, but it lacks a few things:
  • a way to present the test results in a sensible manner; the current output from the console is far from informative;
  • the ability to group requests in a transactions; something similar to JMeter transaction controller, for instance. Trample requests are fired up in random order.
  • cookie and session management (present in JMeter).
Still the idea to write  performance / load and stress tests in a way one writes unit and integration tests is powerful. If anyone knows about similar solutions for Ruby, share the knowledge in the comments section below the post, please. A follow-up to the benchmarking tools comparison: web applications performance management tools. If you are doing performance tuning or you just want to keep track of  your application performance over time, the ability to display and compare the results obtained is a significant factor to consider when selecting a benchmark tool for a Ruby on Rails application. Choose benchmarks which facilitate creating baselines. It is also a good idea to include web application performance monitoring and management tools in the performance test suite for a Ruby on Rails application. The tools like New Relic operate on real production data and provide real time insights into:
  • response time and throughput,
  • errors and availability,
  • user satisfaction,
  • server resource utilization.
Although the tools like New Relic cannot provide the information gathered during performance / load and stress testing, you can use New Relic to validate and further improve your performance test suite. With such tools at hand you obviously gain the regular benefits of 24/7 monitoring of your application performance in the production environment.
Radek supports us with his Ruby on Rails/database programming skills as well as his invaluable insights on projects embedded in rich social contexts.