Saying a lot while writing very little using test matrices.

Very often in testing we need to think about and describe a very large set of possible test problems. Perhaps are asking for clarification on requirements. Maybe we anticipate testing certain behaviors, or we are trying to clarify priorities and risks. Perhaps we are reporting on what testing we did do.

In any of those situations, there are often cases where the permutations get very large. One approach would be to describe every permutation, every possible test condition, in a list. This is time consuming, tedious, and is the fastest way to lose the attention of anybody you are communicating with. We need ways to say a lot in very little space that is easy to understand.

The following is one example - the test matrix. The content is an excerpt from my book "Writing Test Plans Made Simple".

Explaining Combinations of Tests with Matrices and Tables

Behind flat lists, one of the most common and popular ways to model tests is with some sort of matrix. They are popular because many tests are about combining multiple variables or actions together to see an effect. Matrices are usually represented as tables where the rows and columns are combined together to describe a set of tests. They are a simple construct with a lot of power.

The simplest matrix usually represents the test problem as if all the values in every column were combined together to comprise the tests. The end result is multiplicative. Consider the following example:

Test Issue: How well does Web-Calc hand different types of dual operand mathematical formulas?

Methodology: ?Create formulas of the form “<X> <Operator> <Y>”, using each value of X, Y and Operator from the following table:

In the example above, there are 8 values for X, 8 values for Y and 7 values for Operator. This results in 8x8x7=648 tests. Imagine if all of those tests had been written as a list. It would have gone on for pages and pages, and almost nobody would have the patience or energy to read the entire list effectively. But, represented as a table, all of the tests can be perceived and reviewed without even having to flip pages. Everything can be seen in one eyeful.

I call this the, “one eyeful rule” of test analysis. Generally, people do really well understanding a problem if they can read it all in one image. If they have to scroll, flip, or change views, the problem is hard to understand, as they have to remember information during all the flipping.?It is also important that whatever is presented is still legible in one eyeful.

Don’t take the “one eyeful rule” so seriously that you do something unnatural in the test analysis, but definitely be mindful of it. You will often find that modeling the problem in a way where it can be understood all at once will force you to see things you didn’t before.

There is a lot more that can be done with matrices and tables, and indeed a good deal of testing theory rests on using such tables as a means of constructing tests that maximize coverage without having to execute every single test in the analysis. In this book, I am more concerned with introducing you to these simple, yet powerful tools as a way of making the writing easier.?

I cover more strategies such as above in "Writing Test Plans Made Easy." It describes several ways of quickly describing the questions and tests for an application.