Manual Testing and Test Automation: 10 Considerations

Once a QA team has made the case for automation and is moving from doing all testing manually to introducing automated testing, the change process often comes with some challenges that need to be addressed.

A few examples:

• If the automation of test cases is owned by only a few people, and not everyone on the test team, there is a risk that the ownership of product quality falls between the cracks.
• If a DevOps team sends out new, “one-size fits all” directions and protocols for software releases, it might result in development and test teams losing equilibrium: They are still accountable for product quality, but they no longer have the power to do something about it.

To increase the chances of success when equipping a QA team with a test automation tool, it’s important to realize that manual testing and test automation complement each other. Both have strengths and weaknesses. Below are ten aspects to consider when figuring out how your QA team can benefit from test automation.

 

CONSIDERATION #1: Flexibility in testing

Manual testing comes with the inherent benefit of offering testers full control and flexibility of how to execute single tests. Testing manually allows fast, ad hoc testing and is crucial for signing off new features during a sprint.

Test automation should only be applied to more stable features. This means that automation cases for features are built one or two sprints after the implementation of the features in question.

Read our post about agile testing to learn more about this approach to automation.

 

CONSIDERATION #2: Regression testing

Manual regression testing is incredibly time consuming and comes with a range of issues:

• Uncertainty about whether tests are executed the same way every time.
• Executing the full regression suite manually creates a bottleneck in the release cycle making it rigid.
• Due to time constraints, manual regression testing is usually not feasible every time the software changes.

For these reasons and more, regression testing is an ideal target for automation. Automated regression testing comes with following benefits:

• Robots never sleep. This significantly extends the time window for running the regression suite.
• Automated cases are executed the same way every time producing credible test results.
• The time spent building an automated test case is a one-time cost.
• An automated test case can be reused as many times as needed.

Read more about automated regression testing as part of a sprint.

CONSIDERATION #3: Scalability

Manual testing can only be scaled with more people and hours being allocated to the given project. More test cases require more hands, which doesn’t scale well.

Automation enables a high level of scalability as you can simply just add more test executors (i.e. “robots” or “agents”.)

 

CONSIDERATION #4: The Human Touch in DevOps

DevOps is all about streamlining the release pipeline, and automation plays a key role in this. Test automation, specifically, is completely in line with DevOps principles, for various reasons:

• Automated execution
• Tests are triggered as part of a release pipeline
• The development team is provided with feedback quickly
• Reporting is highly structured

However, there are elements of the DevOps pipeline that require a human touch:

• As mentioned earlier, testing of new features in a sprint manually comes with some advantages.
• Analyzing and managing reasons for why automated test cases fail is still very much a task that require a human mind.
• Building automation cases is a manual task.

Read more about test automation in a DevOps world.

 

CONSIDERATION #5: Processes and ownership

When you apply automation to the DevOps pipeline, make sure that testers still have – and feel they have – a say in how the pipeline is defined. Testers are accountable for product quality and they should decide how often builds are pushed to test environments, when automated test should run, and how test results are distributed and formatted.

A DevOps pipeline should support the software development process, not the other way around. In the same way, a test automation platform should support—not dictate—the defined pipeline.

Test automation supports the DevOps pipeline which in turn supports the software development process.

A test automation platform should come with plugins for pipeline orchestrators (Jenkins, Bamboo, TFS, etc.) and/or with APIs for executing test cases and to read test results.

Not two organizations or products are the same, so it’s important that a test automation platform has the necessary flexibility to support many types of software development processes.

 

CONSIDERATION #6: Change management

Introducing agile processes and DevOps is a big change for organizations, and management must equip their teams to being able to handle this transformation.

Testers should have a chance to take a step back and reflect on how things are done today and whether processes can be improved. In a QA team both developers and testers should have a say, and each team member should be empowered to make the necessary changes.

Some test automation platforms are restrictive in the sense that they force testers into one way of working. To avoid this be aware that the testers—not the tool--should decide on the processes related to test automation. This helps ensure ownership in a change process.

 

CONSIDERATION #7: Tooling

Introducing an automation tool comes with the risk that a long learning curve or technical challenges take up too much time and require a lot of support between team members. This takes away time and focus from testers’ primary tasks (building test cases, analyzing requirements, reporting, etc.).

When choosing an automation platform, make sure that the evaluation is done thoroughly. Every member of a QA team should be able to work with it, and the tool should be a good fit in three aspects: technology, processes, and organization.

From a technical perspective, you can use the following requirements for the evaluation:

• Which applications are you automating? (Web, Desktop, Citrix, other?)
• Should the automation platform be deployed on-premises or in the cloud – or a combination?
• Should test cases run locally, on cloud-based services, or both?
• How well does the platform integrate with the release pipeline?

Read “Get started with test automation” to learn more about how to evaluate automation tools.

Criteria for evaluating a test automation tool: Organizational fit, technology fit, and process fit.

 

CONSIDERATION #8: Skills

The essence of testing is to guard and improve the quality of an application through a series of tasks: Test case definition, requirement analysis, test execution, regression testing, result reporting etc. All these disciplines are still relevant and required when test automation is introduced.

However, it is crucial to ensure that the team can reap the benefits of test automation. Otherwise it’s just another set of tasks added on top of the already existing tasks. Upgrade the skills of the test team through training and make sure to select an automation platform that matches the skills of the team.

When it comes to the skill sets required for test automation, there are basically two different paradigms. One is programming-based requiring testers to code when building automation cases. Automation tools of this paradigm are designed with a developer mindset and assume that testers have a deep understanding of how the underlying technicalities of the system under test.

The other approach is to let testers build automation cases by working with a visual designer. With this method, testers can fully utilize their domain knowledge (see below), without having to worry about an application’s underlying code.

Learn more about the differences between the two approaches to test automation.

 

CONSIDERATION #9: Domain knowledge

A deep understanding of an application never goes out of fashion! As mentioned above, the concept of knowing a software well is often confused with having to know the code the software is made of. To be clear:

Programming skills are not—and should not be—a default part of a tester’s skill set, but domain knowledge will always be part of the testing profession – automation or not.

Utilizing automation is based on the tester’s knowledge about an application and how it should be tested. Make sure to nurse and value domain knowledge – it is the basis for good software quality.

Automation success is dependent on the collective knowledge of the entire test team. A robot is not a sentient being – it does what humans instruct it to do. So, implementing automation is about utilizing a team’s collective knowledge in the development of automation cases and making sure that automation becomes part of the daily work routine – in the same way that writing test scripts is a natural part of manual testing.

Continued reading: "Will AI Take Over Test Automation?"

 

CONSIDERATION #10: Organization

Test teams are often organized around products – or projects for changing products.

With or without automation, this a valid and efficient way of organizing teams. It results in dedicated, relatively static teams with a well-kept pool of domain knowledge, which is exactly what is needed for creating and maintaining a regression suite for an application. Make sure to preserve all processes in which testers are picking up additional domain knowledge about the application in question.

Success with automation requires that the entire test team is on-boarded and receives proper training. Otherwise, the overall responsibility for the product quality is neglected.

All members of a test team play an important role in automation – but not necessarily the same role. For example, testers with some technical proficiency can take on organizing reusable components and working with DevOps to include test automation in the release pipeline. Testers with a high level of domain knowledge can oversee creating and maintaining test cases and monitoring results of automated tests.

All members of a test team, incl. domain specialists, technical testers, and test management play an important role in making test automation work.