What debugging techniques are most effective for failed automated tests?

A debugger is the best tool to review code behavior while executing realistic scenarios. Software development starts with an algorithm which usually handles multiple use cases. Generally, each use case has common and variable behaviors. Testing the common behavior is relatively straightforward. Troubleshooting failed automation cases for variable behaviors requires special attention. Debuggers let developers monitor the value of specific parameters, run the program until a specific code section or a function, then run the following code step by step to see the behavior of the program and identify the reason for the unexpected behavior. In short, running a debugger to learn the system behavior helps finding the failure reason of a test case.

While developing code or analyzing a core file, using a debugger might be the best way to test and understand the system behavior. In contrast, while analyzing a production issue, using a debugger may not be an option unless there is sufficient information to reproduce the issue. Logs help to understand the scenario or anomaly causing the issue. In other words, logs might be the only resource to find out what happened in the field. Thus, engineers need to capture useful information in the logs which will give them a clear view of the scenario. Good engineers consider use cases and possible scenarios to generate logs providing most information with minimal overhead as generation of each log uses system resources and slows down the execution.

For many years we have included debug logging within the application logs when developing products. When debug logging is enabled we log the execution of each method its parameters and outcomes, including try/catch exceptions along the transactions execution path. We find this a good method when debugging complicated execution paths. Especially adding new modules in to existing applications.

When a new feature is developed, several test cases are created to make sure that acceptance criteria for the feature is satisfied by the code change. Some of these test cases are automated to rerun them in every new release. Depending on the purpose of the test case, test results need to be analyzed differently to determine the impact of the software changes. Some system behavior changes can be determined and fixed via simply finding and addressing the failure cases. In contrast, some cases such as performance degradations may require detailed review of test case results. For example, modifying the database schema may slightly slow down the system. In these cases, reviewing test results in detail to find all anomalies is the best practice.

In my experience, I often feel that embracing the power of simplicity is less valued. Simplifying test cases in automated unit tests significantly aids in easy debugging by focusing on specific functionalities or scenarios. These streamlined test cases help isolate issues within the codebase, making it simpler to identify and rectify errors. By reducing complexity and targeting precise behaviors, simplified test cases pinpoint problems more effectively, facilitating a quicker and more efficient debugging process. This focused approach accelerates the resolution of issues, ensuring a smoother development cycle and higher-quality software outputs.

Test cases should be created by focusing on two things. First, the test case needs to determine if the code behaves according to the requirements. Second, it should be easy to find the reason for test case failure. To make sure that requirements are satisfied, test case reviews and requirement traceability matrices are used. The simplicity criteria deals with the ease and maintainability of test cases. Having multiple test cases address a single requirement makes it easy to troubleshoot test case failures while providing necessary and sufficient coverage for requirement verification. Simpler cases addressing a single step or a condition helps developers find the failure reason without analyzing complex scenarios.

Our mobile framework is an API only application our test harness connects to the application using the API. We use the test harness to provide system level/user level application testing.

Automated test cases can be run during every software check in or in nightly builds. Running automated tests frequently can identify software issues early in the software process and save time and money. However, it requires a framework simulating a production-like environment. A test harness facilitates the testing by controlling the test inputs and outputs and simulating missing components. We choose a test harness which is compatible with our software interfaces and programming languages. We use open source tools when available and develop simulators to support proprietary interfaces. Regardless of how it is created, the test harness acts as the most important component of a reliable and repeatable test environment.