The Case for Test Cases

In my 40+ year career in software engineering, one of the most daunting problems that I have observed facing software projects is production of usable, accurate documentation of a system.

Over the years, I have used a variety of approaches to address one of the fundamental problems of systems: keeping the documentation in lock-step with the actual software. Typically, we developed a rich system specification based on what we hoped were good user requirements. We made attempts during actual development to make minor updates to the specification to ensure it described the developed software. However, once the software was in production, it was easy to make the classic error of correcting a defect and not bothering to update the documentation. After all, the support engineering function was to keep things working correctly, not necessarily to write documentation.

I attempted to address this problem in the 80s by designing software generation tools which literally built the software from the documentation. This was a nice step forward in the software engineering process, because in reduced the chance that the software documentation would drift during development. However, the support issues remain with this approach, although it could be partially addressed by “re-generating” the software.

Many commercial software houses attempted to solve this problem in the 90s by publishing detailed release notes which attempted to document the changes to the software. With this approach, the documentation is now in two places, the original specification and the release notes. This is not very handy for someone who wants to really understand how something works.

With the arrival of UML, much effort was put on front-end development of rich use cases. These often provided a better understanding and language of how the software worked. However, they did not address the continual problem of documentation drift.

With the advent of agile approaches and test-driven development, there now seem to be a light at the end of the tunnel. On a recent agile project, we co-developed test cases and software simultaneously, with the users driving the test case development. Where possible, the developers converted these into automated test cases. Each iteration was required to pass all test cases, so you had no regression during the development phase. See https://blog.parasoft.com/bid/52034/how-can-test-driven-development-tdd-be-automated for a discussion of this approach.

With this approach, you might ask, “Where is the system specification?” The answer is that the specification is the test plan consisting of the suite of test cases. Through use of modern software support tools, you can ‘bind’ the software module to the appropriate test cases. The support engineer now cannot make a defect correction without modifying the related test cases to reflect the current behavior of the system. This seems to be a near ‘silver bullet’ in the evolution of software engineering.