1/4 million Selenium test executions for WhenWise

The total test executions for WhenWise app reached a milestone today: a quarter of million test executions over 2.5 years. That translates 267 test executions per day. Frankly, I care little about this stats, it is just a minor milestone for WhenWise (the total executions of my other app ClinicWise is over 610,000). By the way, both WhenWise and ClinicWise are developed (and maintained) in my spare time.

What really matters to me is that our Continuous Testing process, which enables us pushing updates to the production frequently. How frequent exactly? I am not fixated at a particular figure: such as "Facebook's twice a day". I do production releases as necessary, sometimes 3 within a day or 2 in a week, it really does not concern me. I just push the new release to the production after getting a green build (passing all automated UI tests) on BuildWise. We trust our automated tests.

Some will have doubts, as many UI test automation attempts failed, such as:

1. What's the size of WhenWise UI test suite?

443 Selenium tests, raw Selenium WebDriver tests in RSpec syntax. The test scripts follow the design conventions as I wrote in my book: "Practical Web Test Automation". You can also get the free sample ones on Github.

2. How long is a test run of all WhenWise App?

56 minutes with 6 BuildWise agents (as shown in the above screenshot). If running all these tests one by one, the execution would take about 5 hours. I know, it is not exact Kent Beck's 10-minute build (one of the XP practices). I could shorten it further by adding more build machines to our test lab. With the quick feedback feature of BuildWise, I am comfortable with the current set up.

3. How do you do defect tracking?

No defect tracking at all. As a matter of fact, we never used any form of defect tracking at AgileWay from day one, for each of our highly-claimed apps (TestWise, ClinicWise, BuildWise, SiteWise and WhenWise). Simply, there is never the need for defect tracking: we create one automated test to replicate the issue, and include it in the regression test suite.

4. How do you achieve high reliability of the UI test execution?

Executing UI tests sequentially (often in traditional CI servers) won't work, a single test step failure (WhenWise has ~24000 test steps) results in a broken build. In my opinion, executing functional tests in parallel with the features of Auto-Retry and Dynamic ordering (as found in BuildWise) is the key to the success of Continuous Testing ( and therefore, Agile and DevOps).

5. How do you maintain test scripts (as app changes often)?

First of all, The effort of maintaining automated test scripts is much more than creating them. Be aware of that.

Design your test scripts well, using Page Object pattern and use an efficient testing tool (I use TestWise).

6. Maybe WhenWise is a simple web app?

No, WhenWise is a feature-rich and dynamic web app, it is more complex than most medium-size software project (has at least 143 web pages, as indicated from the number of page classes we created in our Selenium test suite).