Making Selenium Test Writing Easier

Testing software is painful. Testing AJAX based sites is really really painful. Here at CustomInk we have a very AJAXy site in the CustomInk design lab (http://www.customink.com/lab) and early on in its development we knew that automated testing was going to be necessary. We didn't have a QA department or the technical staff necessary to do black box testing so we knew that we had to try to squeeze as much as we could out of automated testing. The tool we chose for automating our web based testing was Selenium, for the reasons mentioned in previous posts.

The first test that we tried to automate was one that added a piece of clipart to a shirt and then made sure that the clipart could not be drug off of the shirt. It took two developers, three days to write one test and this is what it looked like:

Don't bother trying to read that, the point is that it was super complicated and very painful to write. Here are some highlights of it's most awful parts:

1. Doing one thing always took multiple commands - A simple statement like "Load the lab' would often turn into 5 - 10 Selenium commands
2. AJAX testing is all based around the Selenium command waitForCondition which involves ugly inlined javascript
3. Copy and Paste == Selenium code reuse
4. Selenium commands require xpaths or css locators that tightly couple your tests to the underlying implementation. Basically this means that every time an implementation detail changes (like the id of a div) all of your tests will break. Search and replace will then become your friend

Our problem had turned from "How to implement automated testing" to "How to make test writing for Selenium easy." In thinking about this problem we realized that this is not an issue with Selenium. Really Selenium is great for what it does, provide a generic framework for testing a generic application. We weren't try to test a generic application though, we were trying to test our application and there was never going to be an open source testing framework for our application.

We concluded that using straight Selenium was not a good idea and that we had to try to build something on top of Selenium to make test writing easier. This was our focus the whole time, making test writing easier. We tried several approaches including using the Selenium IDE and adding our own javascript functions into user-extensions.js - none of these worked. Eventually we used Selenium on Rails to write our own Domain Specific Language (DSL) which allowed us to write our tests in the language of our business rather than in the language of Selenium. This approach was the winner because it allowed us to turn our adding clipart and trying to move it off the shirt from the previous Selenium code into this:

This approach had several advantages:

1. The keywords were easy to use and made sense in the context of our application. The words used here were the words of the business.
2. It provides a layer of abstraction on top of the implementation details. In no place do be reference elements by name or use xpaths or css selectors. This is all handled in the DSL itself.
   
3. No more copy and pasting, instead commonly copied and pasted Selenium code is rolled up into a single command that actually makes sense when you read it.
   
4. The ability to actually use Ruby code when necessary. If you want to do one operation 5000 times you can with a simple for loop around that keyword.
5. No javascript! All of the javascript for the Selenium waitForCondition commands is hidden in the DSL.
   

The biggest downside to this is that the DSL implementation is a living piece of code that has to be maintained. You can not just write it and then expect to just walk away from it. You are going to want to add things gradually and hide all of the implementation details in the DSL implementation. This means that whenever the underlying technical implementation of your web site/application changes you need to change the DSL implementation as well (but not your tests).

The idea of a Domain Specific Language (DSL) is actually not a new one at all and in fact the term means different things to different people. In this case we are using it to mean a High Level Language, based on the language of your business, that is built on top of another high level language (Ruby in this case). Really it is a way for you to construct commands so that people can write programs that look and flow like a conversation about your business. This is why a DSL was perfect for our situation, we had a generic framework that we wanted to make test a specific application. That is what a DSL does, take a generic programming language and allow you to make one that is specific for your needs.

Our DSL is now around 80 commands and supports a couple of hundred tests. We run it on multiple operating systems on multiple browsers in parallel. Whole books have been written on DSL construction but here are our tips on getting started writing your own DSL for Selenium tests:

1. The goal here is to make test writing easy, so start by writing up three or four tests that cover the basic core functionality of your application. Then read them out loud, see if they actually make sense. Only once you have these tests just the way you want them should you sit down and try to make them work.
   
2. When you do actually start implementation, just worry about getting the basic functionality for your sample tests up and working. Once you have the initial commands working then add new commands as needed.
   
3. Don't try to design all your commands up front - you will never be able to. The people who should be designing the new commands should be the ones who will be writing the tests - not a sub-committee.
4. If you see any of the following in a test you need to write another DSL command: element ids, css selectors, xpath or javascript.
   
5. Always error on the side of adding another command. Yes you could make due with a command called "verify_position" and then just do the math to figure out if it is centered or not but that makes tests harder to write. Just go ahead and add a "verify_centered" command and make it easy to do forever.
6. Every command should contain at least one assert. You want to assert as much as possible. Since you are writing a DSL anyway why not have each command assert that it is working correctly? Then, any situation that you use it in will always assert that the application is in the correct state.
   

There is a lot more to say about testing DSLs but really you just need to try it in order to understand how well it is going to work in your situation. It is not a magic bullet. It is not an open source application that can be downloaded, have a one click installer and automatically test your application. It is a technique and like any technique it should only be applied where it makes sense.

Our next post is going to talk about the specifics of implementing a DSL using Selenium on Rails.


Tags: selenium, selenium testing,ruby on rails,
white box testing, selenium on rails, dsl, domain specific language, test writing easy, customink, tshirts, custom tshirts