Test automation

• We should not automate.
• Automation does not make testing faster.
• Test automation is not testing.

Doesn’t make too much sense, does it? Of course, those statements above are a little bit exaggerated. But only a little bit. I am going to prove some points about test automation, which are represented by those radical claims above.

Test automation is a topic, which appears pretty often during discussions about testing. Some people are excited about it, some are reluctant, there are some people between them – whatever. We, people who deal with testing – we do talk about test automation. Through all my years in testing I’ve heard about test automation many times. I have also seen some test automation efforts, which can be considered successful. However, most of them are not. From the other hand it has to be admitted that more and more good cheap or free tools are available on the market. They are not incomplete, beta versions – they are mature, hardened in many battle fields, stable tools. Also, industry as a whole seem to have more and more mature treatment of the subject. However, there is still something to improve, which made me write this article.

We should not automate

First of all we set wrong strategic goal – we should not try to automate the tests. We should toolify them. What do I mean by “toolifying”? I mean support the testing process by tools. In most cases – software tools, but in some cases, the tool could be a proper organization of work or test data. When we chose the right tools, which really help, then from time to time we may notice, that if we add a little bit more, the tests will be executed automatically, with minimal human intervention.

Automate vs toolify may sound like a subtle semantic difference, with no actual significant and practical meaning. Let me give an example to explain my point better.

You have to regression test a billing system for telecom company. You do it by setting up some scenarios - activation of accounts, activation of services, change of some services, deactivation etc. Then you simulate phone calls and print the bills.

If our priority would be to automate, we would look at manual tasks (mainly GUI actions) and will look for opportunities to automate them. There are decent free tools around, which will allow you to do it. You will/will try to automate all clicking in GUI scenarios.

If our priority is to toolify, we would look at the whole process and will try to identify where do we spend most of our time. Then we will think how to organize our work, to save as much time as possible.

I need to add an important comment here. One may consider this opposition above (automate vs toolify) a little bit artificial. One would say – if we are given a task to automate it does not mean that we will not be smart enough to identify quick wins area. Planning to automate is not guaranteed to blind people on important traits of a testing process. That’s right. However, goals do impact the results. You may get good results by setting slightly wrong goals. However, if your goal is not the right one, it is much more likely that the results will also be not as good as they could be.

When we automate we start writing programs/scripts, which will click on GUI windows. If we toolify we will notice that clicking GUI controls actually does not take that much time. Finding a test data does. We will notice that if a particular scenario takes 10 minutes to run, 9 minutes are spent on finding what has to be typed in GUI controls. Once we have these data, typing them and clicking take 1 minute. So if our goal is to toolify, before we automate a single GUI action, we will write a script or SQL statements, which will give us a test data, information, which we need to type in GUI controls.

Full automation in most cases is very, very difficult to achieve and infeasible. And the industry seem to understand that. If we take the functionality to be tested and we decide on the very beginning that our goal it to automate the tests, and then follow this decision, we take huge risk. During the creation of the automation framework we may encounter many problems, originating from the fact that most of software is produced in order to implement some business logic rather than to be easy to automate. Before the automation framework is ready, we get nothing in return. And even if it is ready, some change in a tested software might limit its usefulness.

When instead of automating, we try to understand the testing process, reorganize it accordingly and apply proper tools for it, after implementing each improvement we get something in return. Even if we do not implement everything we planned, we are getting the benefits from what is already implemented. If we do it rightly, we test more and more effectively, and one day it may happen (but it does not have to happen always) that we realize that we need 1 or 2 small things and we will have some tests automated.

When I see some automation efforts, I am under an impression that the test automation is a goal in itself. This is wrong. The goal in itself is to make testing faster and more reliable. And all of this has to be delivered in feasible manner. It sounds obvious but it is many times forgotten. I’ve seen a couple of automation frameworks, were the major delivery was the automation of some operations. But the testing itself was more difficult and time consuming with the framework than without it.

Automation does not make testing faster

The second problem with automation is more related to getting a “go” from decision makers, and later keeping them convinced that automation was worth the effort. In other words – it is about expectations setting. People who talk about test automation give wrong promises about what test automation really delivers. Most of people asked about the main benefit of automation would answer – speed. Presentations about test automation for stakeholder will for sure contain a diagram showing how much faster tests are run. But that’s not the main value delivered by test automation. The main one is this – because of test automation we will be able to run often some testcases, which without test automation would not be feasible. That’s the value of test automation. When we have the core functionality tests automated, we will be running them after every single shipment. We will not test quicker than without this core functionality tests automated. Without a test automation, we will simply not regression test core functionality. We will accept the risk and skip those tests, because running them often will not be feasible.

However, when we talk to stakeholders, to business to convince them to give money for automation, we promise exactly that. That instead of running tests in 10 days you will need 0,5 day. Then it is absolutely logical for stakeholders and business people to reason as follows:

“If it’s so great then we may expect that the testing effort in each future project will be reduced by 95%. “

And once they see this promise not kept, they feel cheated.

Test automation does not deliver the testing duration reduction – test automation will increase the reliability of the testing process. It will minimize the risk of delivering a software with a bug in a core functionality. You reduce the time spent on investigation of bugs in the core functionality. And there are some other ways in which more reliable software reduces the testing effort. But directly test automation does not provide the value of reducing the testing time.

“How come”, you may ask. Isn’t an author building some thought provoking claim on some logical trick, which actually is nothing more than a trick? Well, it is obvious when you thing about test effort distribution during the testing process. In most cases, when a new functionality is coming, it does not make sense to automate it. It is new, immature, so it is very likely to change. Its behavior is not that well defined and fixed to justify the test automation effort. Therefore, most of the new functionality is tested manually. If you provide a new functionality, you are very likely to touch the core, the functionalities delivered in the past. Regression test would be nice. But if you deal with a huge system, which is out there for a couple of years, full regression test would be quite an effort. So you reduce the regression test effort significantly or you even skip it. If a new functionality requires 1 month of tests, full regression test may take even 2 months. With test automation regression test takes 2 days. Business or whoever pays for testing will be reluctant about adding 2-3 days of testing but they will probably accept it, considering the reliability they get in return. From the other hand they are very unlikely to accept increasing the testing effort by 200%. They would rather take a risk and pray, that if manual tests of a new functionality don't not catch any core functionality problems, then the core functionality will probably not broken.

Test automation is not testing

This one statement is not exaggerated and radical. It is logical and rational. Running test automation is testing. Automating tests is not. It is a development. And for development you hire developers, not testers.

Testing is about checking if the software behaves in an expected way. Sometimes “expected” is measurable and well defined, sometimes not.

Test automation is about writing a program, which behaves in well defined, precisely described manner. So it is a development, not testing. Managers seem to often overlook this difference and they give the task of test automation to testers. And of course it is possible that a tester will be a good developer and thus a good test automator. The same way as a talented plumber can a be a good carpenter. However, you are much likely to get a furniture you need if you hire a carpenter.