- It works in all of the browses that we care about (IE6, IE7, Safari 2, Safari 3, FireFox 1.5 and FireFox 2) - not all in-browser testing suites do. Specifically things like Watir and Web Driver only support a browser or two.
- It is open source, which means that it is both free and very adaptable to our needs.
- It has an API that allows it to be run on multiple machines (running different browsers) at once
- It is Rails friendly (thanks to Selenium on Rails )
- It is straight forward to build your own framework/DSL on top of
- It is being supported by an active community
- Selenium Core - This is the basis for writing tests in Selenium. The way that Selenium Core works is that you author HTML tables that contain three columns. The first column contains a selenium command (http://release.openqa.org/selenium-core/0.8.0/reference.html) and the next two columns contain parameters to pass into that command. When you want to execute that test Selenium Core runs Javascript that parses the table columns and executes the commands in the browser and displays their success or failure. This is a very simple model for writing tests that allows for a lot of flexibility and power (since the tests are running in the browser and executing as they would for a normal user). There are commands to do almost everything - a lot more info is available on openqa.org (also there is a video at the end of this post with our tests running in it).
- Selenium IDE - A simple FireFox extension that gives you a tool to more easily develop the html tables that selenium core wants to parse. Supports things like element look up and command completion.
- Selenium RC - The selenium remote control is a way to execute selenium commands on remote machines. This is actually two pieces, first it is a small java program that runs on the machines that you want to connect to and have run selenium tests. Second it is a set of APIs (for most languages) that allow you to connect to these remote machines and instruct them to execute individual Selenium commands or whole Selenium test suites.
- Selenium on Rails - This is a plugin for Ruby on Rails that allows you to write selenium tests in ruby and get them compiled down to the table structure that the Selenium Core wants.
At CustomInk we use the Selenium Core, Selenium RC and Selenium on Rails. So really everything except the Selenium IDE. We found that writing Selenium tests as raw HTML tables is a very bad idea. Writing them directly has two main problems, first it is tedious to do because you are working at the wrong level of abstraction. Selenium commands do things like click link X and drag div Y, they are very granular. So it takes a lot of them to do anything, especially when you mix in validating the correctness of the results of an action. Second, by writing tests this way you couple the tests directly to the underlying implementation. When you write Selenium commands you have to tell it exactly what element to click (an xpath, or a css selector generally) and exactly what to do with that element that it selected (drag left 3 px). This means that every time you change your application a lot of tests are going to break. Not because the functionality is broken but because some technical detail has changed. Even if you can deal with the tediousness and the verbosity of the tests long enough to write an initial suite this suite will quickly decay as the application changes under it and every test has to be changed to accommodate that test.
This is the reason that we chose not to use the Selenium IDE, and really why we suggest that no one uses the Selenium IDE, because you still have these two problems. At the end of the day the Selenium IDE is just an easier way to write HTML tables at a very granular level. When you start to think about working with Selenium you need to also think about how you are going to handle test writing and maintenance first and how to make that work in Selenium second. We strongly recommend developing your own Selenium Framework or your own Domain Specific Language for writing Selenium tests. Again, do not plan on writing all of your tests with Selenium commands by hand, it will eventually fail. Selenium is wonderful at providing lots of in browser technical functionality for testing but you need to develop your own way of describing the tests that you want it to run.
At CustomInk what we have done is to use Selenium on Rails to build a Domain Specific Language to allow us to describe our tests in the language of the business. Rather than say click link X we say Add Text "Blah" or Add Clipart "bird" or Checkout With Information a,b,c. The implementation of those keywords in Ruby then hides the underlying implementation as to what happens and generates the HTML table that Selenium Core can run. This is straight forward to do in Selenium on Rails (which we will cover another time).
When we run our Selenium Suite we do so on a series of VMWare images running various versions of Windows and with different browsers installed. Each VM is running the Selenium remote control. Using the Selenium RC Ruby API we tell each browser to run the test suite at location X and post the results of all those tests to URL Y. We then have a collector at that URL that puts the results into a database and a web page to display the results. This is straight forward to do and lets us add more configurations to test very easily.
We plan to post next on using Selenium on Rails and how we built our own DSL. Here is a video of our test suite in action:
Tags: selenium, selenium testing,ruby on rails,
white box testing, selenium on rails, dsl, domain specific language, selenium core, selenium ide, customink, tshirts, custom tshirts
