Friday, November 9, 2007

Improving Selenium Performance in IE6

If you have played around with a Selenium Suite of any size you have probably noticed that running it in IE6 is anywhere from 5 to 50 times slower than running it in Firefox. Part of this comes from the fact that IE6's Javascript implementation is really, really slow compared to the Firefox (or even IE7) implementation. The other part comes from the fact that most versions of Selenium Core, especially 0.8.3 that CustomInk currently uses, leak memory in IE6. This means that every test that runs in a suite runs slower than the previous test. There are two simple methods that CustomInk has adopted to deal with these problems.

Never, Ever, use Xpath Selectors
The easiest way to get big speed ups in IE6 for your Selenium Test Suite is to not use XPath selectors, ever. At CustomInk our initial test suite used XPath selectors almost exclusively. By removing these and replacing them with CSS and ID selectors we achieved a 10x speed up. Doing XPath in IE6 has to be completely implemented in Javascript, which is really slow in IE6. By removing it and replacing the XPath selectors with primarily CSS selectors you can achieve a very large speed up at a relatively low cost.

Never, Ever, have a Test Suite Larger than 20 tests
The other big speed up that we achieved at CustomInk was to take one huge Test Suite and break it up into many small Test Suites. This change helped across all browsers. Chunking up the tests and resetting the browser before starting on the next suite allows all of the memory that was being used/leaked to be cleared out. At CustomInk we fooled around with a lot of different suite sizes and found that 20 was about the right size.

This is the only way that we have found to really get around the IE6 memory leaking problem. We tried to dig into the Selenium Javascript and were not able to make much headway. Even if the Selenium Core didn't leak at all, resetting the browser every 20 tests or so will always be good for performance.

With these two simple rules CustomInk was able to get a 15x increase in performance. Even with these changes our IE6 tests still take 5x as long as our Firefox tests. If anyone has any other performance tips/tricks please post them and share with everyone.