Is it a myth that automation testing would conquer exploratory & manual testing?

Cutting-edge mind-set that growing these days gives such impression that every organization and IT corporate want to push automation testing as the substitute of hard core manual functional testing that has various legs including exploratory testing, sanity & smoke testing, regression testing, re-testing & feature testing.

Now it’s very much evident that automation testing is certainly unchallenged option for regression, sanity & smoke testing where existing & unchanged application areas or scenarios gets tested again and again for baseline functional stability perspective. However, this concept cannot be hold true while we talk about exploratory testing and feature testing that includes new functional implementation and implements that were never considered for testing earlier.

Ideally, automation testing can only play complementary role but unsound expectation like automation tests should replace manual tests, is utterly erroneous. The biggest issue is that most of the organizations & technology gigs ignore the biggest drawback of test automation. This is code (or script if you prefer) written and needs itself testing & corresponding change maintenance. Even a well-designed script needs to run properly and shown the right results. So testing on this is always a mandatory task. Of course we can debate that the benefit of running the same script multiple times outperforms the time to verify it. I would not disagree. My concern is that after running thousands test cases and shown as "passed" what check and balances you have to accept the results without any doubts that your automated scripts may were not right.

One of the myths that circulating around is automation test would work equally for every type of application & domain, is again an ambiguous expectations. Nature of work & technicality might decide on amount of coverage and usability that actually turn out to be true return of investment.

Although automation test has vast and compelling rumour that it will reduced number of escape defects and requirements of manual functional testers however I have observed that actual ground reality is differ than the high volume expectations. It’s because expectations toward automation tests are bit ambiguous, rather than considering automation test as best substitute of repeated regression tests we are coming on thinking that automation test is actually a substitute of initial function test that are being carried out manually by following requirement driven test case execution. In fact, exploratory testing is now completely out of scene which actually an important and critical test technique to uncover requirement gaps and important business improvements.

Basically automation testing is divided in three zones, performance, functional and security. It’s evident that doing performance and security testing using automation tool and technique is certainly a value add and effective approach, you can even cover over 90% application area by automation as it will merely require high and new changes in scripting until unless changes do not impacting core areas or functions. As far as automation of functional zone is concern, it can only compliment the stable and baseline area until unless new changes & implementation are not thoroughly tested against the actual business requirement.

Looking at vicissitudes expectancy the industry line should be changed to think about empowering unit test automation that can be carried out along with code & functional development. It will actually turn out to be the critical & game changer steps that will actually signify outcome when less number of failures passed on to testing phase. By this way less number of manpower would be needed during testing phase as cycles would be of short duration as less number of failure would be observed & reported.

While testing reincarnation took place around year 2000, it was never conceptualized that testing will be used to catch regular defects or obvious failures of software development. It was actually, considered as complementary inspection of system to uncover potential escape issues or requirement gaps before handing over system to client. To make sure intact coverage, requirement traceability metrics were considered as the important reference factor and merely any of obvious issues were escapes and caught during testing phase until unless they are out of box scenarios. There were shortcomings in approaches & methodologies which were subsequently improved from waterfall to spiral and spiral to v-model and at some extent till agile. However, with these innovations & improvements quickness has actually overpowered the quality. We made utmost quick approaches to organize and signed off business requirements, quickness during architecture designing and similar during functional coding. Finally, similar expectations are set during testing phase that does it as quick as possible via automation. Concrete steps has to be taken for making sure that initial business requirements are thoroughly reviewed w.r.t. to indirect association and consequences. Similarly the approach hold true for architecting & coding phase. Coding guidelines and code-testing (unit & white box) should be mandated as the integral part of every development team and leads/manager should be hold responsible for any deviation. Likewise manual & automation test scoping & feasibility have to be clearly discussed, planned and circulated to avoid unnecessary expectations from executive management & client.

Last but not the least, if we closely look at our on-going engineering approach & kind of failures are being reported by testers & clients, we can easily zeroed on that we are certainly missing or compromising some important facts or areas during our engineering cycle which are really important to instrument the quality deliverables.