The Test Shade is Always Gray: Beyond the Notion of Black & White-Box Testing

There are only a few contrasting terms in the testing world which have helped me in understanding the subject better. The distinction between black-box testing and white-box testing is one of them.

However, over a period of time, as I experimented more and more, this distinction became limiting. It puts an upper bar on the testing thought process. So, if I split my 20 years in testing into two halves, in the second half, this distinction started fading away more and more. Although I can acknowledge why these two terms exist and are used, I consciously try to break out these two thought cages. You can do that too if you want.

Before you read further. This discussion is not about what is called what. That does not concern me. I concern myself with limitations on thought that any kind of thinking brings. The binary thinking of black and white box testing as a thought constraint is what I am challenging here.

Back to Basics

Let's get to the basics first.

Testers define different terms differently. It does not matter how you define it. For the purpose of this distinction, I need a starting point. So, I chose to pick up definitions from Wikipedia (I know, I know. Doesn't matter by the time this article ends.).

Here's what Wikipedia says about Black-Box Testing and White-Box Testing, verbatim:

Black-box testing is a method of software testing that examines the functionality of an application without peering into its internal structures or workings. This method of test can be applied virtually to every level of software testing: unit, integration, system and acceptance. It is sometimes referred to as specification-based testing. - Wikipedia, Retrieved on 17 July, 2023.

White-box testing (also known as clear box testing, glass box testing, transparent box testing, and structural testing) is a method of software testing that tests internal structures or workings of an application, as opposed to its functionality (i.e. black-box testing). - Wikipedia, Retrieved on 17 July, 2023.

Some initial thoughts for you from these definitions:

• Should black box testing be only about functionality testing? What has the notion of a black-box got to with constraining it to only one Quality dimension?
• If you are sitting on a table along with a developer having a cup of coffee and the developer ends up mentioning some internals. As a self-labeled black-box tester, are you doing black-box testing any more? (k aapka dharm bhrasht ho gaya?)
• Should white box testing only about internal structures - why not functionality? Btw what is the meaning of functionality? When you test a function by calling it, aren't you testing its functionality?

When you start with a categorisation mind-set, it no more remains about just distinction between 2 words/phrases, you will start segregating a whole world of concepts into two boxes and end up force-fitting. This force-fitting then indirectly puts limitations on thinking and the quality of testing that one does.

Here's an exercise for you: although unrelated, but just do a thorough search (beyond your own thought) to find what different testers mean by error, mistake, fault, defect, bug, failure, discrepancy, issue, problem, cause, root cause. Have fun! But please spare me as rather than fun I find all this funny :-).

A Directional Bias

In my thought process when I see a thought leading in only one direction, I question myself why?

This has helped me break many biases in my mind as well as understand constraints.

The usual black-box testing thought process has a subtle directional bias to it.

As a tester we typically talk to a test object via an interface. What is to the right of / behind that interface can be thought of as a black box. That's what I infer from the way testers talk. There is an assumption in this as if know everything about what is on the left where we sit, the Primary Point of Control .

What most testers fail to realise is that this is how the directional bias comes into play:

• I know very little about the left side, but I'll ignore that.
• I know some details about the right, but I'll ignore them. If I am consciously using them, it's ok. (Nobody told me it was chicken. I thought it is cheese. Now let me continue eating it, and still claim to be a vegetarian).

The truth, though, is something else and in itself is a liberating one:

Here's a break-down of this thought process:

• Rather than as a Test Actor, the one who conducts a test, you need to shift to the role of a Test Thinker to think in this manner. On the same lines, rather than the Primary Point of Control, it's more of a Point of Thought.
• The Gray Zone is your knowledge zone. In practice, it is a mix of what you are supposed to know based on the traditional outlook of black and white boxes. This also includes knowledge beyond these traditional boxes, where you are not just concerned with a specific implementation but bank on yours and others experience, constant learning about other similar implementations, your imaginations, your notions etc. The Gray Zone is a personal box. You have yours. I've mine.
• Everything outside of this Gray Zone is the Black Zone (box). Most of the Test Object, the participant objects, the users, the world, the business and so on. Experience and deliberate learning keeps bringing some chunks of these black zones to your Gray Zone - when you learn a little better about how those things actually work (the internals) and are working for a specific context/implementation.

Following is a more generalised version of the previous visual which further generalises and simplifies it:

Further Thoughts for Self-Labeled Black-Box and White-Box Testers

• Knowing code will not make you biased. We are biased anyways :-). Exploring internals, thinking about them will only enrich your testing. You can look at a code, generate a test idea and execute it at the same interface as you are used to in black-box testing. Without looking at the code, an insight into architecture, how protocols work will help you too.

Example: You fill a form in a web application deliberately with wrong data and when you click submit button, there is an error message as expected. Most testers relate this functionality to the web application as a black box and are happy with it. Let's break it down a few steps. Who rejected this form data? Client-side JavaScript or the web application server? This single answer can further build the basis for a security test. If it were just a client side rejection, you can apply the GOLF Heuristic - Go One Layer Further and conduct the same test to find a different answer.

• Knowing a particular code and directly writing tests for it is what one often calls white box testing. Please understand there's always a black box. At some point in this white box testing you are also doing black box testing.

Example: You are testing a file creating functionality in a code base. This code makes a library call to a third party lib. Did you go through its code? Or that lib is a black box? What about the call that this said library is making to the built-in library of the programming language? And then a subsequent call to the OS's user mode? And then to kernel mode? and so on. You see, at some point this white-box thought process will hit a black box as far as you are concerned. It will be outside your Gray Zone.

Further Thoughts Beyond Such Labels

As a tester it will help you to think that you are in the middle of something rather than at the edge of something.

Look both ways. Rather, look around from your current locus/point of thought.

E.g. when you are designing a scenario at the top most layer of your system, even then you don't know the internals of a system that consumes your system or how a user is actually going to use it.

The other direction of thought, is already discussed above.

This article, if you read between the lines, is an extension to what I wrote in the article on the GOLF Heuristic - Go One Layer Further .

That's all for now.

Related Articles:

20 Years of Contradictions

Pluralism and the Infinite Testing Schools

The Infinity Model of Imperfect Quality for Fallible Testers

An Undefinition - What is a Test?

AEIOU - The Vowel Model of Thinking for Pluralistic Testing

The Ambidexterity Continuum of Testing

Uncaging, the Perpetual Translation Engine and the Deltas

Think-Words for Testers

Metacognition - Biases, Problems, Abstractions and Variables

Uncaging the Types of Testing

?A Critique of Some Popular Testing Principles

A Straw Man Called Regression Testing or Why Regression Testing is Exploratory