My definition of test automation

Earlier this week, Jason Arbon posted an article on LinkedIn titled 'Checking vs Testing'. The first sentence of article, which is:

"Too much energy, creativity, and time are spent debating 'testing versus checking' and 'manual versus automated'."

particularly resonated with me. I'm not sure I agree with the remainder of the article to the same extent, especially because I think that too much time and energy is spent criticizing other people, what they do and the way they think, too, but that first sentence is something I do agree with wholeheartedly, and it's what triggered me to write this article.

Please keep in mind that this is not a response to that article, nor is it an attempt to settle a discussion. I'd rather steer clear of the discussion altogether and spend my time doing other things instead. It's just me giving my view on what constitutes 'test automation'. Maybe it's useful to some of you, maybe it's not. Your call :)

Whenever I teach a course on test automation (and that has been quite regularly the last couple of years), I often start with a question to get a feel for what people already know about or have heard about test automation:

What do you talk about when you talk about ‘test automation’?

The answers usually involve things like 'running regression tests automatically', 'testing more in the same time' and 'having more time for the interesting stuff'. After asking a couple of follow-up questions, it often turns out that what people know about test automation is typically limited to what Jason in his article calls 'regression automation'.

After that, I share my definition of test automation, which includes regression automation, but is broader than that:

"Using tools to perform specific testing tasks more efficiently"

Why this definition?

Let's take a closer look by dissecting it into three parts.

"Using tools"

It shouldn't come as a surprise that tools come up when I talk about test automation. Without tools, there's no test automation. The range of tools I tend to discuss in my training, though, go beyond just test execution tools (like Selenium or Postman), because I think it's important that testers (and others involved in test automation) know something about other types of tools, too.

Stubbing and service virtualization tools. Mocking frameworks. Reporting libraries. Test data generators. Log file analyzers. All part of the set of tools I think anyone involved in test automation should have in their tool belt, because these all play a part in what I talk about when I talk about test automation. You'll hopefully see what I mean when we take a look at the other parts of my definition of test automation.

"To perform specific testing tasks"

When I talk about test automation, I don't like to talk about 'manual versus automated'. I think this is a false dichotomy, that it isn't either/or. One isn't better or cooler than the other, because there's nothing to compare. Test automation does not replace manual testing, and therefore implementing test automation does not absolve you from doing manual testing. Instead, test automation is an activity that I think is meant to support testing.

There are lots of activities in testing that cannot be automated. There are also activities in testing that can be automated (i.e., performed or supported by tools). People like Michael Bolton and James Bach have covered this extensively, and I generally agree with what they have written. I still use the term 'test automation', though, when I talk about using tools to support testing. I also sometimes still find myself writing or talking about 'manual testing'. I'll gladly leave that discussion to others. Shoot me.

"More efficiently"

The last part of my definition of test automation is something I still find overlooked often (there's a reason it's a big part of a case study I present in my 'Introduction to test automation' course). Even when you find that specific testing activities can be performed or supported by tools, it doesn't mean that you should do so. Sometimes, not automating these activities is the better (more efficient) option. Maximum automation (code) coverage isn't something to strive for. Nor is 'automating all our regression tests'. Optimal coverage and optimal use of tools is.

Or, as Peter Drucker said so eloquently:

"There is nothing quite so useless, as doing with great efficiency, something that should not be done at all."

I'd love to read your views. In the meantime, I'm off to learning and teaching more about how to use tools to perform specific testing tasks more efficiently.