It is time to get realistic about test automation

You'll only have to take a look at any number of job openings in the software testing space to instantly see that test automation is a hot topic. Teams and organizations are investing significant amounts of money in automation, all in an attempt to keep pace with the ever increasing speed and agility with which they want to deliver software. As a result, they are constantly on the lookout for employees or external consultants that can help them create and implement automation effectively.

When done right, automation can definitely be a very effective means of checking whether certain parts of a software system conform to predefined behavioral characteristics. However, I see a lot of automation efforts that do not live up to the expectations people have of it, and that's rapidly becoming a problem. Inefficient, untrustworthy and/or badly written automation is likely to slow down software development instead of delivering the fast, reliable feedback that was promised. In this article, I'd like to dive deeper into the reasons why automation projects fail to deliver, by means of three examples of mismatched expectations, as well as what we all can (and quite frankly, have to) do in order to make things right.

Expectation: testing can be automated and test automation can replace testers

Sadly, a lot of organizations still believe that test automation is an activity that replaces the work their testers are currently doing. They assume that what is often referred to as 'manual testing' (which in itself is a term that shouldn't be used, unless you're testing manuals) can be replaced by code. Essentially, that the critical thinking, observation, experimentation and learning that testers perform can be replaced by machines executing predefined sequences of clicks, keystrokes and API calls.

If you're a tester and all you do is execute and check off scenarios that were defined by someone else and yell 'pass' or 'fail' at the end of each scenario, then you might indeed be in trouble. But as we all know, that's not what proper testing looks like. Testing encompasses many other activities, and most of them cannot be automated. Risk analysis, communication and mitigation. Determining whether the product provides value to people who matter (the definition of 'quality' as it is given by James Bach and Michael Bolton, based on an earlier definition by Jerry Weinberg). Critical thinking. And the list goes on. All part of testing, all inautomatable.

We should start seeing test automation for what it is: an activity that (when done well, but more on that later) supports testing. We should stop regarding it as something that can replace testing. Surely, a solid set of valuable automated checks can provide fast feedback on whether or not a system or application still conforms to some predefined, deterministic expectations. This frees up time for testers to perform more interesting and complex activities. It is a fallacy, though, to expect and act as if these checks can replace those very testers.

Expectation: we need to turn our testers into automation engineers

Another falsehood, and one that's closely related to the previous one, is that testers should be turned into automation engineers, and that becoming an automation engineer is the only way to 'survive' as a tester.

Let me be clear on this: I think that becoming familiar with the concepts, principles and (im-)possibilities of test automation is a valuable (and rapidly becoming an indispensable) skill for every tester. That doesn't mean that testers should feel forced to radically change what they're doing and become fulltime automation engineers instead. For some, a transition towards a more automation-heavy role will make sense. If you're interested in automation, have to deal with it regularly or feel that it could be a useful means to support your testing activities, by all means spend some of your time and effort to become proficient in it.

My point is that it should not be a forced transition, if only because not all good testers make equally good automation engineers, and vice versa. We need to stop blindly pushing people away from testing and towards automation. It's not either-or, it's both-and.

Expectation: the current way we provide automation training is useful

Finally, if you do want to improve your proficiency in test automation, you're bound to face another problem: there is preciously little proper education available.

Now, I hear you saying 'What? But there are so many courses, tutorials and workshops out there!'. That's true, and some of those are actually very good. The main problem with the vast majority of these courses, though, is that they are tool-focused, meaning they'll teach you how to perform tricks in a certain tool. And I know, I have delivered -and still deliver from time to time- similar courses and workshops myself. Still, I've made it a habit to always try and paint a broader picture:

• How does the tool fit into your overall software testing and development process?
• What alternatives are available?
• What can the tool not do?

These are all pieces of the test automation puzzle, pieces that are underrepresented in too many test automation courses.

We should all be aware that being well-versed in a specific tool is only a very small part of becoming proficient in test automation. Instead of more tool-oriented education, I'd like to see courses that teach you, among many other things:

• how to manage expectations around test automation
• how to decide if automation is a valuable addition to a specific development activity in the first place, and if so, what would be a good implementation strategy
• that 1-on-1 conversion of existing (regression) test scripts into automated checks often is not an effective strategy for automation
• how common software development patterns and practices apply equally to the creation of automated checks

In short, I'd like to see more education that presents a realistic view about test automation. Now, there are already some good resources out there. The Automation in Testing course by Mark Winteringham and Richard Bradshaw, that will be part of TestBash Brighton in March next year, looks promising. The month after that Ard Kramer and I will host a workshop on reasonable expectations in test automation at TestBash Netherlands. I am confident that they are a good start, but much more needs to change in the way people can learn how to be a good test automation engineer. It is one of the things I hope to spend much more effort towards in 2018.

I'd like to conclude by saying that automation can be a powerful means of enhancing testing efforts and making them more effective. However, this can only become a reality once both the testing community itself and everybody that is informed by means of testing activities comes to terms with what automation can and cannot do for you. There's a lot of work to be done!

About me: I'm an independent test consultant who loves helping clients bring their test automation efforts to the next level. You can contact me here on LinkedIn, via @_basdijkstra on Twitter or through my blog OnTestAutomation.