You're faced with conflicting testing results. How do you navigate between automated and manual findings?

1 Understand Results

When faced with conflicting test results, your first step is to thoroughly understand what each method is telling you. Automated tests are great for checking predefined scenarios quickly, but they might miss nuances that a human tester would catch. On the other hand, manual testing allows for exploratory approaches and can identify issues that were not initially programmed into automated tests. Analyze the context in which the discrepancies occur—is it a false positive from the automated test, or did the manual test uncover a real issue that the automation missed?

2 Verify Accuracy

Once you've understood the conflicting results, verify the accuracy of both the automated and manual tests. Check if the automated test scripts are up-to-date with the latest application changes. For manual tests, ensure that the tester has correctly understood the requirements and executed the test cases accurately. Sometimes, environmental factors such as network issues or database states can affect test outcomes. Ensure that both testing methods were performed under similar conditions to rule out external influences.

3 Evaluate Coverage

Evaluating test coverage is crucial when reconciling automated and manual findings. Automated tests might have a broad reach but can lack depth in certain areas, while manual tests can provide in-depth coverage but be limited in scope. Compare the test cases from both methods to see if there are gaps in coverage that might explain the conflicting results. It's essential to ensure that both automated and manual tests are comprehensive and complement each other.

4 Communicate Findings

Effective communication between the QA team and other stakeholders is vital when dealing with conflicting test results. Present the findings from both automated and manual tests to developers, product managers, and other relevant parties. Discuss the implications of the discrepancies and collaborate to determine the root cause. This collective approach can help in making informed decisions about addressing the issues uncovered by the tests.

5 Refine Processes

After discussing the conflicting results with your team, it's time to refine your QA processes. If automated tests are consistently missing bugs that manual testing catches, consider enhancing your automation scripts or incorporating new test cases. Conversely, if manual testing is unreliable, look into additional training for testers or revising the test plans. Continuous improvement of both automated and manual testing processes is key to minimizing discrepancies in the future.

6 Document Learnings

Documenting the insights gained from resolving conflicting test results is an often overlooked but critical step. Record the nature of the discrepancies, the investigative process, and the resolutions achieved. This documentation can serve as a valuable reference for future testing cycles and help prevent similar conflicts from arising. It also contributes to building a knowledge base that strengthens your QA practices over time.

7 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?