Friday, November 9, 2007

Tutorial: Adding a Markup field to a Rails App

Recently we added a Textile-based text markup field to an internal webpage. We used the RedCloth library. It was very easy:

1. Install RedCloth
This is dead simple using gem:
# gem install RedCloth
Bulk updating Gem source index for:
Successfully installed RedCloth-3.0.4

2. Add a Database Column to Hold the Text
We'll make a Rails migration so that this change can be tracked:
$ cat db/migrate/053_add_constraints_to_styles.rb
class AddConstraintsToStyles < ActiveRecord::Migration
def self.up
add_column :styles, :constraints, :string

def self.down
remove_column :styles, :constraints

3. Add Test Data
This part is so that we can see if our view will work before adding the editing capability (which I'll show next time):
# ./script/console
Loading development environment.
>> p = Style.find("04600")
>> p.constraints
=> nil
>> p.constraints = "Test: _this_ string +is+ *textile*"
=> "Test: _this_ string +is+ *textile*"
=> true
>> exit

4. Modify the View
In the view I first add just
<%= @style.constraints %>
to make sure the model is pulling the right data. Looks good, so now go for the gold!
<%= %>

Improving Selenium Performance in IE6

If you have played around with a Selenium Suite of any size you have probably noticed that running it in IE6 is anywhere from 5 to 50 times slower than running it in Firefox. Part of this comes from the fact that IE6's Javascript implementation is really, really slow compared to the Firefox (or even IE7) implementation. The other part comes from the fact that most versions of Selenium Core, especially 0.8.3 that CustomInk currently uses, leak memory in IE6. This means that every test that runs in a suite runs slower than the previous test. There are two simple methods that CustomInk has adopted to deal with these problems.

Never, Ever, use Xpath Selectors
The easiest way to get big speed ups in IE6 for your Selenium Test Suite is to not use XPath selectors, ever. At CustomInk our initial test suite used XPath selectors almost exclusively. By removing these and replacing them with CSS and ID selectors we achieved a 10x speed up. Doing XPath in IE6 has to be completely implemented in Javascript, which is really slow in IE6. By removing it and replacing the XPath selectors with primarily CSS selectors you can achieve a very large speed up at a relatively low cost.

Never, Ever, have a Test Suite Larger than 20 tests
The other big speed up that we achieved at CustomInk was to take one huge Test Suite and break it up into many small Test Suites. This change helped across all browsers. Chunking up the tests and resetting the browser before starting on the next suite allows all of the memory that was being used/leaked to be cleared out. At CustomInk we fooled around with a lot of different suite sizes and found that 20 was about the right size.

This is the only way that we have found to really get around the IE6 memory leaking problem. We tried to dig into the Selenium Javascript and were not able to make much headway. Even if the Selenium Core didn't leak at all, resetting the browser every 20 tests or so will always be good for performance.

With these two simple rules CustomInk was able to get a 15x increase in performance. Even with these changes our IE6 tests still take 5x as long as our Firefox tests. If anyone has any other performance tips/tricks please post them and share with everyone.

Friday, October 19, 2007

Distributing Selenium Tests

One of the nice things about using Selenium is that it runs in the browser, which allows you to test multiple browsers using the same test suite. The problem with doing this is that it is time consuming for any test suite of significant size. The best way to solve this problem is to run the test suite on multiple browsers in parallel. This greatly cuts down on the amount of time that it takes to run the tests while still giving you the same coverage.

Generally speaking there are two ways to parallelize the execution of Selenium test suites, one is to use the Selenium Remote Control and the other is to use the resultsUrl post back. The way that the Selenium Remote Control works is to allow the execution of Selenium commands on a remote server one command at a time. This involves installing a small server-type Java program on all of the machines that you want to remotely run Selenium commands on. Then authoring a program (or using one of the pre-baked ones) to connect to one of the remote machines and execute a Selenium test one command at a time, so the execution loop looks like this:
  1. Connect to remote server
  2. Load Selenium test file
  3. Pop command off of the top of the Selenium test file stack
  4. Send command to remote server
  5. Wait for response
  6. If fail, then fail
  7. else goto 2
The Selenium Remote Control server has nice hooks build into it that allow you to control almost any browser on most platforms, so it is straight forward to have one test be run on multiple machines using multiple browsers at once.

At CustomInk this was the first approach that was tried, executing one command at a time through custom Selenium test files. It is highly customizable because you are, more or less, developing everything from scratch. The Selenium Remote Control gives you a basic framework on which to develop a broader system but the actual execution and coordination of the tests is up to the client program.

After playing around with this approach it was not used at CustomInk for two main reasons. First, the fact that the Selenium Remote Control sends off one command at a time is a very fragile way to do things. If there are any timing problems on either the client or the server, or the client just gets bogged down for whatever reason, the tests will fail spectacularly. Second, developers needed to be able to use individual tests or sets of tests to test changes that they were making, much like unit tests are used. At CustomInk Selenium is not only used as part of QA but also as part of the development process to keep code stable and robust. The fact that running things locally and running things in the distributed environment end up working differently, due to both different test formats (one is html tables and the other is something else) and the fact that they execute so differently, one locally in browser and the other through the Remote Control, turned out to be a problem. Both because it was necessary to maintain different file formats and because things began to get hard to reproduce in both environments so the ability to use Selenium as a debugging tool was decreased.

The other approach is to use the resultsUrl mechanism to build a distributed method for executing Selenium tests on multiple machines over multiple browsers. The resultsUrl method is a technique by which a whole test suite is run to completion on a remote server and then the results of all of those tests are posted back to a central location. The advantage of doing this is that it is both more stable, since entire suites are running remotely rather than one command at a time so the communication coordination is much less between client and server and the fact that the same tests that developers are running locally in their browsers are being executed remotely in a browser. This leads to only a single test format and the ability to reproduce problems much more easily.

This technique is still built on the Selenium Remote Control, so there is a server running remotely and a client program that sends Selenium commands remotely to the server. The difference is the only command that the client sends to each server is to open and execute a whole test suite, using a specific browser, deployed on a staging server passing in a parameter called results resultsUrl. The resultsUrl parameter tells Selenium Core to post the results of the test suite execution back to that URL. An example of the URL that the Selenium Remote Control client would tell the server to open, in a specific browser, would be: http://staging/selenium/TestRunner.html?test=tests&auto=true&resultsUrl=http://seleniumhub?execution_number=53. The auto parameter tells Selenium Core to simply execute the test suite automatically and the resultsUrl tells Selenium Core to post the results back to that URL. So the execution loop would look like this:
  1. Connect to remote server
  2. Send command to open the URL http://staging/selenium/TestRunner.html?test=tests&auto=true&resultsUrl=http://seleniumhub?execution_number=53 in IE6
  3. Client code finishes
  4. On the server, IE6 opens and runs the test suite to the end
  5. The results of the tests are posted back to the URL specified through resultsUrl
This approach involves much less communication between the client and server portions of the Selenium Remote Control, meaning that there are less timing and network issues, but it also involves having the tests deployed on some centralized server as well as having some URL that can collect and display the results of individual tests. Here is a how the all of the pieces work together:

Since all of the code that Selenium is testing, including the test suite itself, is hosted on the Staging Box, reproducing problems is straight forward. Anyone can point their browser at the Staging Box and run the exact same test as the distributed test suite just ran.

The piece of code that resultsUrl posts back to can be very simple. Here at CustomInk we have a simple Rails project that just accepts results (along with the execution_number) and stores all the results with the same execution number to disk. It then allows people to go in and browse the results from the whole distributed test suite by execution number. CustomInk calls this the SeleniumHub. The downside of this is that it is code that you have to develop yourself, the upside is that it isn't very complex code.

CustomInk also integrated the execution of these tests with Cruise Control RB. This is straight forward to do but a script has to be written to kick of the tests and then return passed or failed based on the results that come back. The biggest problem with doing this is that a complete integration test suite built in Selenium normally takes to long to execute to be running in a continuous way. To get around this problem it is straight forward to create "Smoke Test" test suite that runs a subset of the whole suite and can run in a reasonable amount of time. Again, this is not difficult and allows our Continuous Integration Server to not only run our unit tests but also to run a

Overall this approach has proven to be robust and repeatable. The largest problems occur when individual tests having timing issues built into them, especially if they are testing AJAX applications, and tests fail intermittently. This is not a problem with this approach however, but rather a problem with the your tests.

Tags: , ,,
, , , , ,

Friday, October 5, 2007

Selenium on Rails for Java

In our previous blog posts we have gone over using Selenium on Rails to write custom DSLs to make in browser web based testing much easier. The largest complaint that we hear about this technique is that it uses Selenium on Rails and most people out there are not in an environment where they can use Ruby and/or Rails. The solution to this problem, at least for Java development houses, is to use JRuby and gold spike to make a Selenium on Rails installation that packages itself up as a war. This allows you to develop DSLs using the convenience of Ruby and Selenium on Rails but never having to actually install Ruby. It also allows you to deploy the test suite to your application server of choice. By doing this you also solve the cross-domain scripting problem that often plagues selenium test suites (the need for the tests and the application to run on the same domain).

In this post we attempt to go over how to get Selenium on Rails up and running in an Application Server in such a way that you can easily write your own custom DSLs. For the purposes of this example we are going to use a Red Hat Linux box and Apache Tomcat 6. More detail on troubleshooting a JRuby installation can be found here.

Getting Started

1. Install Jruby:
The easiest way to do this is with a direct download:

Untar this wherever you want to install jruby, I am installing it in /home/apache/jruby so my examples will be using that.
2. Set your JRUBY_HOME and PATH variables to make jruby accessible everywhere:

export JRUBY_HOME=/home/apache/jruby/
3. Test your jruby installation
  • Create a file test_jruby.rb with the following contents: puts "I t-shirts"
  • Run your simple ruby program: jruby test_jruby.rb -> This should output I t-shirts.
4. Create a testing Ruby on Rails project
  • Install the gem for rails (this will take a few minutes):
    $JRUBY_HOME/bin/gem install rails -y --no-rdoc --no-ri
  • Now create a selenium-tests rails project:
    jruby $JRUBY_HOME/bin/rails selenium-tests 
    This should result in a directory and a whole bunch of files called selenium-tests being created
  • Next we want to make our rails project self contained, not dependent on having the rails gem installed. To do this do the following:
    cd selenium-tests
    rake rails:freeze:gems
    This will leave you with the rails gem installed in your rails project.
  • Finally install something called golden spike. This is what allows a rails project to be compiled into a war file. This is done by running the following command from your selenium-tests directory:

    jruby script/plugin install
    This will install the golden spike plugin into Rails.
Selenium on Rails

Now that Jruby in install and Ruby on Rails is running (without ruby actually be installed) we can get on with installing Selenium on Rails and then testing everything from within a Tomcat instance.

1. Installing Selenium on Rails
  • Selenium on Rails is nothing more than a Rails plugin, like golden spike. To install it we use the script/plugin command. To be specific, from your selenium-tests directory run the following:
    jruby script/plugin install
    When this command completes selenium-on-rails will be present in your vendors/plugin directory.
  • Finally generate a basic selenium test that you can use for double checking that everything is working. To do this use the selenium generator. From in your selenium-tests directory type the following:

    jruby script/generate selenium test
  • To test to make sure everything is working up to this point you can start the internal Ruby on Rails web server and see if you can connect to the default selenium suite. To do this type tho following in your selenium-tests directory:
    jruby script/server -e test
    Now the ruby web server is running on port 3000 so point your web browser at: http://localhost:3000/selenium. You should see a selenium test suite with your single test in it.
2. Running everything under Tomcat
  • The way that the golden spike plugin works is that will compile your entire project into a war. This war can then be put under an Apache Tomcat instance and have the tests run in the same domain (and same environment) as your normal web based java application. The first step is to install the Activerecord-jdbc gem. To do this type the following:

    $JRUBY_HOME/bin/gem install activerecord-jdbc --no-ri --no-rdoc
    This should cause the gem service to start and it to say that Activerecord-jdbc was successfully installed.
  • One of the ways that selenium-on-rails operates is that it only exists when the rails environment is set to test. Unfortunately golden spike (the war creator) does not support RAILS_ENV very well right now. The easiest way to do this is to edit line 77 of selenium-tests/vendor/plugins/goldenspike/lib/war_config.rb and change the @rails_env variable from "production" to "test". Hopefully golden spike will respect the RAILS_ENV variable a bit better in the future and this step will not be necessary.
  • Now it is time to build the war! To do this issue the following command from your selenium-tests directory:
    $JRUBY_HOME/bin/rake war:standalone:create 
    This will take a moment to run as it downloads the appropriate jars and builds them into a single war. For more information on this process, and the wide variety of options that can be passed in to optimize the process, look at JRuby Wiki. You should also notice that we are doing all of this in test mode, this is how selenium-on-rails prefers to operate, though it can be tweaked to run in any mode. The result of this command should be a selenium-test.war file.
  • Now lets test this war. To do this just copy the selenium-tests.war to the webapp directory of a running instance of Apache Tomcat. Then point your web browser at http://localhost:8080/selenium-tests/selenium. You should now see the basic selenium test suite running in selenium-on-rails in a Java application server.
Once a basic installation of selenium-on-rails is running inside Tomcat it is then possible to follow our previous blog posts and create domain specific languages (DSL) for testing or do any of the other nice tricks that Ruby + Selenium give you.

Tags: , ,,
, , , , , , ,

Monday, September 17, 2007

Getting started writing DSLs using Selenium on Rails

In previous posts we have discussed the reasons for using Domain Specific Languages (DSLs) to make implementation of Selenium tests much simpler and more maintainable. This post is going to talk about how to go about actually starting to write Selenium DSLs using Selenium Core and Selenium on Rails. This technique relies on the fact that either you are testing a rails application or you have a method for ensuring that the domain that your application to be tested is on the same domain as your rails app that will host Selenium on Rails (the way that we have done this is by using Apache and mod_proxy).

For this example we will use one CustomInks internal Vendor Management system as the demonstration. Exactly what is being tested is not as important here, this same set of steps should work for anything - just how you lay out your DSL commands will differ.

Step 1: Install Selenium on Rails

The first thing to do is to install selenium on rails. This is easily done:
script plugin/install \
This will install both Selenium on Rails and Selenium-Core into your rails project. Next what we typically do (at least for the development stage of the DSL) is to make it so that Selenium will work in development mode as well as in testing mode. Normally Selenium on Rails will only function when rails is started in testing mode both to make sure that it was using the right database and to be sure that the tests were not accessible in production. This is accomplished by first copying the config.yml.example that comes with Selenium on Rails to just config.yml, so:
mv vendor/plugins/selenium-on-rails/config.yml.example \
Once this has been accomplished you then edit config.yml and uncomment the line under 'environments' that says development (line 9 in my version). So it would look like:
- test
- development
Now start webrick(or your rails server of choice) and point your browser to http://localhost:3000/selenium. You will see the basic Selenium startup page with no tests to write.

Step 2: Write a very simple test

Now lets write a simple test to make sure that things are working. We are going to author a RSel file, which is really nothing more than a ruby file with the Selenium API automatically loaded. Note: There are several additional ways to write tests in Selenium on Rails, using RSel is the way that we have found to be the most effective.

Selenium on Rails expects all of the RSel files to be in the directory: test/selenium. Selenium on Rails does not automatically create that directory so go ahead and create it:
mkdir test/selenium
Now create a new file in the test/selenium directory called yahoo.rsel. yahoo.rsel should contain one line:
open ''
Save this file and point your browser back to http://localhost:3000/selenium. Now you should see one test called 'Yahoo'. Hit the Play button the right side and you should see the Yahoo home page load.

Step 3: Install basic infrastructure for your DSL

The simple Yahoo test case shows basically how Selenium on Rails works. The basic flow is as follows:
  1. Write RSel test case
  2. Load Test in Browser
    1. Browser makes request to Selenium on Rails for Test Case
    2. Selenium on Rails reads RSel file and compiles it to Selenium-Core table based structure
  3. Execute Selenium Test in Browser
    1. Javascript parses table
    2. Evaluates Command
    3. Loads
  4. Tests finish
The important thing to realize here is that the RSel files are compiled into a static HTML table before the test starts to execute. This means that dynamic conditions in tests are not possible with this method (look into using Selenium Remote Control if you are interested in doing this).

A DSL really is just a series of custom commands that emulate your business language and model your domain. In this case we are writing a DSL for testing but the actual task of writing tests in your own DSL will happen in the RSel files. The way that CustomInk built our DSL was to expose language commands using a Ruby module, then automatically mix that module into all of the Rsel files. This allows your Rsel files to look like this:
bird_clipart = add_clipart_to_product( '26602' )
verify_location on product_of bird_clipart, 0, 0
click_on bird_clipart
click_button 'Center'
verify_centered bird_clipart
To do this we first need to create a module in our Rails project where we can put all of our commands. To do this first make a directory under your Rails lib folder called Selenium:
mkdir lib/selenium
Now create a dsl.rb in the lib/selenium directory. We normally start with a template for our DSLs that looks like this:

module Selenium
module DSL
module DSLCommands
# Insert your commands here
def method_missing( symbol, *args )
if( ( not @delegate) )
@delegate = self)
args_for_print = (args.size == 1) ? args.first() : args.join(', ')
header "#{symbol.to_s.titleize} #{args_for_print.to_s}"
to_return = @delegate.method( symbol ).call( *args )
header "End #{symbol.to_s.titleize}: #{args_for_print.to_s}"
# Add a comment into the table
def comment text
@xml.comment! text
# Add a pretty header
def header text
command 'echo', text
class Delegate
include Selenium::DSL::DSLCommands
def initialize( parent_delegate )
@parent_delegate = parent_delegate
def method_missing( symbol, *args )
@parent_delegate.method( symbol ).call( *args )
def open( uri )
def type( id, text )
This template does a couple of things. First it adds in two additional methods to the Selenium on Rails API, header and comment. These actually used to be methods in Selenium on Rails but aren't anymore (for whatever reason). Header inserts a header into the table (creating comments that can be seen) and comment inserts a HTML comment that can not bee seen. Both of these are used as debugging methods. This template also makes it so that every DSL command that you write will be generated with a header comment before and after all of the individual Selenium commands. This also allows for easier debugging.

Finally we need to modify Selenium or Rails to automatically include our DSL so that all of the tests we write have access to the new commands. To do this you need to monkey patch: vendor/plugins/selenium-on-rails/lib/selenium_on_rails/rselenese.rb. You need to make two edits. First, on the first line of rselenes.rb add:
require 'selenium/dsl'

Then around line 15, make it so that SeleniumOnRails::RSelenese includes Selenium::DSL. So the class declaration should now look like this:
class SeleniumOnRails::RSelenese
attr_accessor :view
include Selenium::DSL
Or approximately anyway. What this does is to make all of your DSL command top level Rsel commands that can be accessed as though they were built directly into Selenium.

Step 4: Write your first DSL commands

Now everything should be in place to write your first DSL command. A DSL command in this context is nothing more than a Ruby method that is designed to fit together with a set of other methods to make a coherent sub-language. At the end of the day your Rsel files are still Ruby (meaning that you get all of the built in Ruby language constructs) but you have developed your own set of commands (or keywords) that more closely model your business language. Remember that Selenium is all Javascript and in order to avoid cross-domain security problems with Javascript your Selenium suite and your application must be on the same domain. Hence the reason that the examples that follow do not test

Lets start with something to load our application, since we are testing CustomInks internal vendor manager the command will be: load_vendor_manager. More broadly lets imagine that we want to test the fact that vendor capacity (the amount of work that a given vendor can be given) is set correctly. Before we even start writing DSL commands we imagine what the test should look like and then write the test with no implementation. This is an important step. Always remember that the goal of your DSL is to make test writing easy, not to make DSL implementation easy. So here is what we want our test to look like:
select_group "East"
select_vendor "Digital"
verify_vendor_capacity "15"
The nice thing about this test is that it seems to be obvious about what it is doing, it works in the language of the business and introduces commands that should be useful in other tests. To start the actual implementation go back to lib/selenium/dsl.rb and find the comment that says: Insert your commands here (around line 4). This is where the Ruby code that implements the DSL belongs. Here is our implementation for load_vendor_manager:

def load_vendor_manager()
command 'setTimeout', '60000'
open '/backend/vendor/details'
wait_for_element_to_not_exist 'id=vendorLoader_c'
assert_element_not_present 'id=vendorLoader_c'
assert_element_present 'id=vendorGroups'

This is a simple command that makes first sets the timeout, opens the page, waits for the pre-loader to disappear and then verifies that the page loaded correctly. Notice how we directly refer to element id's in this command but not in the tests. This is important as it separates out the actual implementation of the application from the tests so that as the application changes the tests don't have to be modified, only the details of the DSL have to be modified. Now if we add another Rsel file under test/selenium with the command load_vendor_manager in it Selenium on Rails will execute the more finely grained Selenium commands provided in the implementation and load the vendor manager.

Notice that in this command we also verify that the vendor manager is in a steady state - we wait for the page to completely load and then assert that it is correct. This is important to do. Since the complexity is hidden behind by the DSL, the more asserts that are baked into the DSL the more confident you can be of the state of your application.

The actual code that we implemented for load_vendor_manager is not what is really important here though. What is important is that this provides a very simple framework that you can add an arbitrary amount of complexity to in order to make test writing ease. The resulting tests are straight forward and complexity is hidden in the DSL.

Hopefully this helps people get started writing DSLs and easy to maintain Selenium tests. Our next post will cover more techniques for the implementation of DSL commands in Selenium on Rails.

Tags: , ,,
, , , , , ,

Monday, September 10, 2007

Making Selenium Test Writing Easier

Testing software is painful. Testing AJAX based sites is really really painful. Here at CustomInk we have a very AJAXy site in the CustomInk design lab ( and early on in its development we knew that automated testing was going to be necessary. We didn't have a QA department or the technical staff necessary to do black box testing so we knew that we had to try to squeeze as much as we could out of automated testing. The tool we chose for automating our web based testing was Selenium, for the reasons mentioned in previous posts.

The first test that we tried to automate was one that added a piece of clipart to a shirt and then made sure that the clipart could not be drug off of the shirt. It took two developers, three days to write one test and this is what it looked like:

Don't bother trying to read that, the point is that it was super complicated and very painful to write. Here are some highlights of it's most awful parts:
  1. Doing one thing always took multiple commands - A simple statement like "Load the lab' would often turn into 5 - 10 Selenium commands
  2. AJAX testing is all based around the Selenium command waitForCondition which involves ugly inlined javascript
  3. Copy and Paste == Selenium code reuse
  4. Selenium commands require xpaths or css locators that tightly couple your tests to the underlying implementation. Basically this means that every time an implementation detail changes (like the id of a div) all of your tests will break. Search and replace will then become your friend
Our problem had turned from "How to implement automated testing" to "How to make test writing for Selenium easy." In thinking about this problem we realized that this is not an issue with Selenium. Really Selenium is great for what it does, provide a generic framework for testing a generic application. We weren't try to test a generic application though, we were trying to test our application and there was never going to be an open source testing framework for our application.

We concluded that using straight Selenium was not a good idea and that we had to try to build something on top of Selenium to make test writing easier. This was our focus the whole time, making test writing easier. We tried several approaches including using the Selenium IDE and adding our own javascript functions into user-extensions.js - none of these worked. Eventually we used Selenium on Rails to write our own Domain Specific Language (DSL) which allowed us to write our tests in the language of our business rather than in the language of Selenium. This approach was the winner because it allowed us to turn our adding clipart and trying to move it off the shirt from the previous Selenium code into this:

This approach had several advantages:
  1. The keywords were easy to use and made sense in the context of our application. The words used here were the words of the business.
  2. It provides a layer of abstraction on top of the implementation details. In no place do be reference elements by name or use xpaths or css selectors. This is all handled in the DSL itself.
  3. No more copy and pasting, instead commonly copied and pasted Selenium code is rolled up into a single command that actually makes sense when you read it.
  4. The ability to actually use Ruby code when necessary. If you want to do one operation 5000 times you can with a simple for loop around that keyword.
  5. No javascript! All of the javascript for the Selenium waitForCondition commands is hidden in the DSL.
The biggest downside to this is that the DSL implementation is a living piece of code that has to be maintained. You can not just write it and then expect to just walk away from it. You are going to want to add things gradually and hide all of the implementation details in the DSL implementation. This means that whenever the underlying technical implementation of your web site/application changes you need to change the DSL implementation as well (but not your tests).

The idea of a Domain Specific Language (DSL) is actually not a new one at all and in fact the term means different things to different people. In this case we are using it to mean a High Level Language, based on the language of your business, that is built on top of another high level language (Ruby in this case). Really it is a way for you to construct commands so that people can write programs that look and flow like a conversation about your business. This is why a DSL was perfect for our situation, we had a generic framework that we wanted to make test a specific application. That is what a DSL does, take a generic programming language and allow you to make one that is specific for your needs.

Our DSL is now around 80 commands and supports a couple of hundred tests. We run it on multiple operating systems on multiple browsers in parallel. Whole books have been written on DSL construction but here are our tips on getting started writing your own DSL for Selenium tests:
  1. The goal here is to make test writing easy, so start by writing up three or four tests that cover the basic core functionality of your application. Then read them out loud, see if they actually make sense. Only once you have these tests just the way you want them should you sit down and try to make them work.
  2. When you do actually start implementation, just worry about getting the basic functionality for your sample tests up and working. Once you have the initial commands working then add new commands as needed.
  3. Don't try to design all your commands up front - you will never be able to. The people who should be designing the new commands should be the ones who will be writing the tests - not a sub-committee.
  4. If you see any of the following in a test you need to write another DSL command: element ids, css selectors, xpath or javascript.
  5. Always error on the side of adding another command. Yes you could make due with a command called "verify_position" and then just do the math to figure out if it is centered or not but that makes tests harder to write. Just go ahead and add a "verify_centered" command and make it easy to do forever.
  6. Every command should contain at least one assert. You want to assert as much as possible. Since you are writing a DSL anyway why not have each command assert that it is working correctly? Then, any situation that you use it in will always assert that the application is in the correct state.
There is a lot more to say about testing DSLs but really you just need to try it in order to understand how well it is going to work in your situation. It is not a magic bullet. It is not an open source application that can be downloaded, have a one click installer and automatically test your application. It is a technique and like any technique it should only be applied where it makes sense.

Our next post is going to talk about the specifics of implementing a DSL using Selenium on Rails.

Tags: , ,,
, , , , , , ,

Tuesday, September 4, 2007

CustomInk and Selenium

Selenium is an in browser testing framework that attempts to simulate a user using your application from the browser. It is a form of end-to-end testing that we use not only to determine if the next release of our web based products are working but also to test that they are working in many different browsers. The CustomInk design lab ( is a lot of javascript and testing it all by hand would be impossible. We adopted Selenium about a year ago to allow us to do this automation. We chose it for a variety of reasons:
  1. It works in all of the browses that we care about (IE6, IE7, Safari 2, Safari 3, FireFox 1.5 and FireFox 2) - not all in-browser testing suites do. Specifically things like Watir and Web Driver only support a browser or two.
  2. It is open source, which means that it is both free and very adaptable to our needs.
  3. It has an API that allows it to be run on multiple machines (running different browsers) at once
  4. It is Rails friendly (thanks to Selenium on Rails )
  5. It is straight forward to build your own framework/DSL on top of
  6. It is being supported by an active community
Selenium is actually a catch all term for the the testing suite which has several moving parts:
  1. Selenium Core - This is the basis for writing tests in Selenium. The way that Selenium Core works is that you author HTML tables that contain three columns. The first column contains a selenium command ( and the next two columns contain parameters to pass into that command. When you want to execute that test Selenium Core runs Javascript that parses the table columns and executes the commands in the browser and displays their success or failure. This is a very simple model for writing tests that allows for a lot of flexibility and power (since the tests are running in the browser and executing as they would for a normal user). There are commands to do almost everything - a lot more info is available on (also there is a video at the end of this post with our tests running in it).
  2. Selenium IDE - A simple FireFox extension that gives you a tool to more easily develop the html tables that selenium core wants to parse. Supports things like element look up and command completion.
  3. Selenium RC - The selenium remote control is a way to execute selenium commands on remote machines. This is actually two pieces, first it is a small java program that runs on the machines that you want to connect to and have run selenium tests. Second it is a set of APIs (for most languages) that allow you to connect to these remote machines and instruct them to execute individual Selenium commands or whole Selenium test suites.
  4. Selenium on Rails - This is a plugin for Ruby on Rails that allows you to write selenium tests in ruby and get them compiled down to the table structure that the Selenium Core wants.
More information about Selenium can be found at

At CustomInk we use the Selenium Core, Selenium RC and Selenium on Rails. So really everything except the Selenium IDE. We found that writing Selenium tests as raw HTML tables is a very bad idea. Writing them directly has two main problems, first it is tedious to do because you are working at the wrong level of abstraction. Selenium commands do things like click link X and drag div Y, they are very granular. So it takes a lot of them to do anything, especially when you mix in validating the correctness of the results of an action. Second, by writing tests this way you couple the tests directly to the underlying implementation. When you write Selenium commands you have to tell it exactly what element to click (an xpath, or a css selector generally) and exactly what to do with that element that it selected (drag left 3 px). This means that every time you change your application a lot of tests are going to break. Not because the functionality is broken but because some technical detail has changed. Even if you can deal with the tediousness and the verbosity of the tests long enough to write an initial suite this suite will quickly decay as the application changes under it and every test has to be changed to accommodate that test.

This is the reason that we chose not to use the Selenium IDE, and really why we suggest that no one uses the Selenium IDE, because you still have these two problems. At the end of the day the Selenium IDE is just an easier way to write HTML tables at a very granular level. When you start to think about working with Selenium you need to also think about how you are going to handle test writing and maintenance first and how to make that work in Selenium second. We strongly recommend developing your own Selenium Framework or your own Domain Specific Language for writing Selenium tests. Again, do not plan on writing all of your tests with Selenium commands by hand, it will eventually fail. Selenium is wonderful at providing lots of in browser technical functionality for testing but you need to develop your own way of describing the tests that you want it to run.

At CustomInk what we have done is to use Selenium on Rails to build a Domain Specific Language to allow us to describe our tests in the language of the business. Rather than say click link X we say Add Text "Blah" or Add Clipart "bird" or Checkout With Information a,b,c. The implementation of those keywords in Ruby then hides the underlying implementation as to what happens and generates the HTML table that Selenium Core can run. This is straight forward to do in Selenium on Rails (which we will cover another time).

When we run our Selenium Suite we do so on a series of VMWare images running various versions of Windows and with different browsers installed. Each VM is running the Selenium remote control. Using the Selenium RC Ruby API we tell each browser to run the test suite at location X and post the results of all those tests to URL Y. We then have a collector at that URL that puts the results into a database and a web page to display the results. This is straight forward to do and lets us add more configurations to test very easily.

We plan to post next on using Selenium on Rails and how we built our own DSL. Here is a video of our test suite in action:

Tags: , ,,
, , , , , , , ,