What makes a good test automation
In this post, I am going to write what are important factors you have to keep in mind when implementing a test automation framework. I will also point your attention to some common mistakes done during automation.
As I have mentioned in my post about automation in general, the testers should strive to test quicker, more reliably and at smaller costs. Test automation makes sense if it helps in achieving this. If it does not, it should not be done. Test automation should never be a goal in itself. Hence, before deciding “let’s automate” try to think what would make your life easier in a particular functional area. Is there any activity, which is really painfull, really time consuming but in the same time, too important to skip. The perfect example of such a task is finding a test data for a scenario or a set of scenarios. Pretty often the main effort in test case execution is to find the right test data – the right customer with some specific configuration of services, the right phone number, which is suitable for our test. Once we have it, execution of a test case is a simple thing. So, if we have a particular test case, which is often repeated, it may occur that we will speed its execution significantly if we create a sophisticated select, which will find the data we need.
Let me now present you a counter example, when a test automation looks like a goal in itself. I have seen the following mistake repeated many time in test automation. A sophisticated framework is created for performing programmatically some operations, which normally are done manually. GUI operations are the major representatives here. So we have a several GUI operations automated. We gather those automated test cases in a test suite. Each test needs an input data. They are given in a text file, and finding those data are by no means automated. Thus, whoever is going to launch those automated test cases, he/she will have to find the test data. In most cases that takes a lot of time. Much more then the actuall execution of test cases manually. Ok. We’ve found the data, we put them in the input text file. We launch our test suite. Due to some event, unexpected delay, the test suite breaks in the middle. We have to restart it. But we need new input data. So we have to search for them again. Shall we perform our tests manually, we would have been done with them already. But we use automation – we have to find the data again and restart the tests and pray that nothing unexpected happens.
This is an example of automating or supporting tests by tools in the wrong place.
Below you can find the key points about good test automation and also good test support tools:
1. Repeatability
Provide tools, methods and procedures, which will allow you to easily repeat the same test (or a group of tests) as many times as you want. If your test framework executes all tests in 5 minutes but requires half a day to clean up and prepare for the next run – you are not doing good test automation. Start working on reducing the setup time or your test automation solution will not be used. And I am not talking about reducing the time of a part of effort needed to rerun the test suite. I am talking about everything what is needed. I am talking about everything which has to be done after the last test suite execution in order to press <ENTER> for the next execution of the same suite. If 99% of those preparations take 10 minutes, and 1% takes 5 hours, that means you have a problem with repeatability.
2. Easy resumption
If you have a group of automated test cases, which you run in a group, you must provide a mechanism, which will allow you to continue test execution from the place it stopped (or just quickly restart).
This is yet another key feature, which people tend to overlook. There is a sophisticated test suite, but once it breaks in some point, there is no way to resume from there.
Why this is a key feature? Why it is in not only a nice to have?
Because your automated test suites will break in the middle. Of course they will. Otherwise, what’s the point of testing a piece of software if you assume it is not going to break? We, the testers, know, that there is not software which is bug free. Therefore creating a framework, which assumes that in most cases it will run with no problems from a start to an end is a logical contradiction.
If in case of breach you are unable to determine were exactly the test broke and you unable either to make it continue after fixing the problem or quickly restart the test suite, that means that you do not have your test automated. That means that your “test automation framework” is yet another source of problems and delays rather than help.
3. Gather enough data to start diagnosis and debugging once the suite breaks in some point. Gather logs, organize them in such a way that it is easy to find logs related to a particular test case. Provide timestamps. Make automatic screenshots, data dumps.
4. Ease of use. Ease of use comprises of points 1,2 and 3. However, that’s not enough. Your framework can be sophisticated and may allow you to do impressive things. However, there should also be a quick way of running it in some default mode, so not only test team elite can run it, but other testers as well.
These were the traits, which test automation framework should have.
Now let’s talk about some additional factors, which we should have in mind when automating tests:
0. Understand were is it where you spend most of time doing the same tedious job. These are the candidates for support by smart tools. In most cases you will find out, that execution of the test scenarios, actual action with the tested software, is not such a thing. It is a fraction of time, which you spend on preparing the test data and then later on verifying the results.
I am repeating myself here. And I have intentionally given this point number 0, because this is the most important thing you have to watch when automating. It is like a business case in PRINCE2 project management methodology. If there is not business case, there is no project. If there is no efficiency gain, there is no test automation.
1. Learn to write your own tools in some scripting language. If you feel uncomfortable about learning how to program, you can stop reading here. And you can forget about test automation. It’s not for you.
2. To expand on previous point – test automation is a development activity, not testing. Running the automated test suite is testing. Implementing test automation is not. When you test, you run the software and check if it behaves as expected. When you automate, you write a piece of software which has to implement some requirements. This a pure development. Therefore good test automation requires a developer.
3. Remember that you do not write deliverables for the end user. Defensiveness, careful verification of input parameters – this is all very nice. But it takes time and effort. And your tool may be obsolete in next 2 months. You need to find a balance – you should of course implement some safeguards so your tool does not do dangerous things, but don’t go too far in making it idiot proof. In the first place, idiots should not be allowed to test the system.
4. Learn how to use well known unix tools like grep, sed, cat, cut, uniq, sort etc. When you combine them smartly, the results could be really amazing
5. Learn sql – it is a powerful language. However, many people don’t know how to utilize this power since they stop their learning after understanding the “where” clause and what is a join.
6. Data processing is about text processing. Data may have some interim formats, they can be encrypted, packed, binary but finally they represent some information for human. So it has to be textuallized. Therefore master text processing tools. Learn regular expressions. Learn test substitution functions in some mature scripting language like python or perl.
And that’s it. Really. The rest is your smartness, development skills, understanding of tools you use and understanding of the subject matter in which operates the software which is being tested.