The Art of Software Testing by Glenford Myers

This is a book which every test manager should keep under his pillow. In a sense, it is also intellectually provocative and meant to restore the trust in testing, as a discipline, and to break myths such that it is living on the back of development and can make economically sense only based on development. Book is essentialy about software testing but, as explained in the preface (upper right and lower left pictures), it intends to be a "practical, both feet on the ground" handbook, seriously going to its philosophical and economic fundamentals. 

Furthermore, there is no testing (or inspection, or any type of verification) method in IEC61508-7 not to refer to this book. This could be due, on the one hand side, to the "sparcity" of testing literature, as the author claims in the preface, at the time of writing it, 1978, but on the other, to the lack of mature testing literature to compe with complex software development nowadays too. 

What is the book about?

It starts up front, challenging the reader to do a self assessment of his/her testing "abilities".

As you can see in the right picture, he is asked to write a series of test cases that would "adequately" check a simple, trivial program.

Some possible test cases are presented right on the next page (in the picture here below), concluding in a "moralizing" way:

Testing a trivial program such as this is not an easy task. And if this is true, consider the difficulty of testing a 100 000 statement air-traffic-control system, a compiler or even a mundane payroll program.

One does not have to mention that software on todays' cars are magnitude orders larger than the 100 000 statements mentioned here. 

How is the book structured?

After the introductory short chapter about the self-assessment, Chapters 2 and 3 are more about "non-computer-based" testing (inspections, code reviews, walkthroughs..). with more focus on the "economics" of testing, in Chapter 2, then the next 3 chapters are about different "computer-based" test methods. Last two chapters are about debugging and concrete tools and techniques.

Economics and philosophy of testing

Many could find stunning the most important claim made in this book, around which everything else evolves: testing is the process of executing a program with the intent of finding errors (and not to check program correctness, as many of us would think). Lots of important things are extrapolated from that:

Testing is, as such, a destructive process

... even sadistic, claims the author :). For us, this is not a trivial thing to cope with it, since it goes against the constructive and goal-oriented way of the human nature. If a tester is told to finish a task when all his tests are "passed", or "successful", then he will write tests to lead him to that. It is therefore imperative to encourage the testers to set as their testing goals, and to design and write tests, aiming to find errors, instead of "confirming correct functionality". There are in the book references in this sense, to another engineering magnum opus, The Psychology of Computer Programming, by G. M. Weinberg, which I also plan to present in this series. From this derive the next conclusions:

A test should be considered successful when it finds an error, and unsuccessful, when it does not

This changes test result as well as test completion (as we'll see it next) criteria fundamentally. It almost completely reverts current mindset, where test completion is bound to the amount of program (or product) features, and where green charts, bars and pies make all test managers happy and give the appearance of quality checks in order to make investors or managers happy and eager to continue throwing money at the project. But all of that sounds a bit contradictory and looks like setting some goals impossible to be achieved. How can you finally release something to the customer if you're stuck in testing it endlessly? Here comes in the economic aspect of testing. One cannot test all possible program (software) combinations in a finite time (especillay given nowadays complexity and size), therefore test methods, as well as a good test strategy, should guide the tester to find as many errors as possible in a limited amount of time. How should such a strategy look like (also in the picture below):

1.    Start by estimating the total number of errors in a program

2.    How many of those errors can be found by testing?

3. Plan testing effort and define testing completion criteria based on that estimate

This is a completely different mindset than the existing, pervasive one (as I said at the beginning), because testing activities and test completion criteria depend on errors estimates, NOT on product features, therefore test methods and strategy become essential. Here is where test case design methods come into play. Which of them have the highest probability of detecting most of errors? Generically they are split into two groups: black and white box testing, but this is only in regards to the access to the internal logic.

There are also the top-down and bottom-up approaches, with their advantages and disadvantages.

This is where this book meets test methods prescriptions in ISO26262 (because this also relies on this book). When the standard requires a "testing strategy" and recommends (or highly recommends) some methods for a certain ASIL, it does it on the grounds that the higher the ASIL, the more testing activities are required in order to strive to detect more errors. Some "fancy" methods like error guessing, or MC/DC (modified condition/decision coverage), or more common ones, like stress or performance testing, come from that and are highly recommended for higher ASILs (essentially due to economic nature of ASIL).

Testing principles

Another aspect which I took from this book and did not want at all to leave it mentioned and shared, is the testing philosophy behind the entire strategy and all of the methods (picture below). I found especially interesting the probabilistic approximation of "possible other errors " (beyond the already found ones). 

Last chapter is about some concrete testing tools and techniques, like static flow analysis tools, test coverage monitors, or program correctness provers, software-error data collections, use of predictive models to estimate maintenance software costs after production, methods to estimate software mean time to failures. Looking back 40 years, when this book was written, those could look old-fashioned from today's perspective. Most of them are living today via sofisticated and complex test tooling and software, some of them were abandoned due to their economic un-feasability, but all of them were part of some pioneering work in this engineering branch.

Who should read this book?

.. obviously test engineers and managers, for all the reasons outlined above.

Safety engineers and managers - mainly because few meticulously researched where some of "mysterious" testing methods popped up in ISO26262 from, and also because quite many have a hard time to justify why some some methods are required for some ASILs and why not for the others.

Managers - to better understand the complexity of testing activities, the amount of time and budget it requires and it deserves.

Developers, or development engineers, or programmers, to sort out their possible preconceptions about testing ... and also because it is a good piece of engineering literature.

What are the take aways

Have you ever wondered what's the difference between stress, performance or volume testing? This book will tell you. Or, have you asked yourself why specifications have to be tested, or verified too? Or why there should be other person testing the program of somebody else? When do you need condition coverage tests for software? Again, this book will guide you. But maybe the most important take away is to grant testing the importance and the role it deserves. It is a very creative and intellectually-challenging activity, by no means "inferior" to development.

Conclusion

Reader might be tempted to think about this book as one only about psychology of testing and about non-practical blabla bringing no concrete business value. My hope is that everything I wrote here helped to deconstruct such preconceptions and will make you read the book.