Effectiveness vs Efficiency in Test Automation
We start developing an automation script. We fall in love with the script. Hence, we develop it with care, so that it looks elegant, consumes less time, less memory & less CPU. We follow fail-fast approach in automation development. So, when the script fails, we expect a defect in the SUT. Simple right?!
However, there can be situations in which an automation script can fail without any defect in SUT. Yes. Because, automating a test is not always about developing code to test the scenario. It is more than just writing code for each steps in test plan. Three important factors has to be considered before starting to automate a test.
- Limitations of the test-bed
- Known limitations of the SUT (System Under Test).
- Limitations of the tool used for automation
There can be scenarios where a test can fail due to unstable testbed. So, our automation code need to be developed with handlers for environmental issue. Generally, a test case attempts to put the SUT under certain situation and validates if the SUT behaves as expected and recovers the SUT into normal state. There can be specific scenarios known to break the SUT irrecoverably. At nascent stages of product development these cases need not be automated unless acceptable steps are found to recover the testbed and SUT in it. One team member may say "if only this tool has some specific capability, we can automate it easily". Well,! we are limited by the tool we use, aren't we?. Nope, one must use creativity to remove that limitation. We are limited only by our thinking! :-)
When we automate a test with all the above consideration, the code may not look elegant or may consume bit more time than expected. However, our priority should be effectiveness of the test automation not the efficiency. Because efficient script that gives inconclusive results is useless. It is like having a pair of beautiful eyes that doesn't help to see any objects in front.