What can software engineers learn from software engineering research to improve testing?

1 Testing techniques

Software engineers often ask themselves how they can test their code effectively and efficiently. Research has proposed and evaluated various testing techniques to help them reach different goals, such as finding bugs, increasing coverage, reducing redundancy, or optimizing performance. Test case generation involves creating test inputs and expected outputs for a given system or component, with strategies like random, combinatorial, model-based, or mutation-based. Test case selection is the process of choosing a subset of test cases from a larger set based on criteria like fault detection, diversity, or prioritization. It can be done through coverage-based, change-based, or risk-based methods. Finally, test case execution runs the test cases on the system or component and compares the actual outputs with the expected outputs; frameworks, libraries, or platforms can be used to automate and facilitate this process.

2 Testing processes

Software engineers often need to determine how to organize and manage their testing activities within the software development lifecycle. Software engineering research has proposed a number of testing processes that can help plan, coordinate, monitor, and improve testing efforts. Test-driven development, for example, involves writing test cases before writing code and then using them to guide and verify the code implementation. This can improve design, quality, maintainability of code, as well as reduce feedback loops and debugging time. Continuous testing involves running tests frequently and automatically throughout the software development cycle. This helps detect and fix errors early, ensure code consistency and compatibility, and accelerate delivery of software products and services. Agile testing aligns testing activities with agile principles and practices such as collaboration, communication, feedback, and adaptation. This helps involve stakeholders, deliver value, respond to changes, and cope with uncertainty in software projects.

3 Testing challenges

Software engineers must face the challenge of testing complex and evolving software systems. To address this, software engineering research has identified various testing challenges to help overcome the limitations and risks of testing. For example, when testing distributed systems, which consist of multiple components that communicate and cooperate over a network, you must consider concurrency, coordination, consistency, and scalability. Adaptive systems, which can change behavior or structure in response to changes in their environment or requirements, pose issues of uncertainty, variability, and unpredictability. Additionally, security-critical systems must protect the confidentiality, integrity, and availability of their data and resources from malicious attacks; complexity, diversity, and evolution of threats and vulnerabilities are key considerations here. To test these systems effectively, you may use simulation, emulation, orchestration, runtime verification, self-testing, learning-based testing, penetration testing, fuzzing, or symbolic execution techniques.

4 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?