The Pros & Cons of Test Automation

Test automation has varying degrees of business impact both positive and negative depending on factors such as the size (KLOC), complexity, and design of the software to be tested.

From a cost perspective, automation typically requires a large initial investment to establish the infrastructure, test framework, test library, and reporting mechanisms. However, the ongoing effort over time is lessened and can be spread across maintenance of the framework, tests, and tools; freeing up more time to focus on defect prevention.

Automation's greatest strength is the ability to perform scheduled and on-demand regression testing after code changes have been made. Any time code is touched there is a chance to introduce variability in pre-existing code and therefore introduce defects. Manual regression testing consumes more and more time as the test item matures. Automation effectively reduces the time and cost required to evaluate code changes.

Automated unit testing encourages a quality mindset through test driven development. Unit tests and functional code are both written to validate requirements. The smallest functional units can be stubbed out to have various test operations performed on them. This is helpful when there is focus on Built-In Quality within an organization.

A test automation framework enables testing to be performed in parallel on many systems. This is a significant benefit when performing platform and localization testing.

Automation removes variability in the test process to a large degree. It ensures tests are executed the same way each time. It is important that automation tests are correct otherwise it is likely that false positives (and negatives) are reported, incurring cost to the business in the form of needless rework.

There are several pros and cons to Automation, many of which I have described in this article, however there are areas where Test Automation is no substitute for Manual Testing. User Acceptance Testing, for example is a valuable Manual Testing strategy for capturing the human response when interacting with software. The emotions, comfort, and feedback the user feels is not something that can be included in automation. An application may be secure, robust, scalable, resource efficient, and functional but if it is difficult to use or evokes the wrong response from the user it is useless.

In summary, Test Automation is a powerful tool. When applied appropriately, it can reduce the cost of production by removing waste in the development and test process and reduce cycle-time.  Test Automation is part of an overall strategy to deliver quality products to market consistently.