My thoughts on thinking

My colleague Blake Norrish recently posted this article on one of my favourite topics: how to think like a tester. I agree with many, if not all, of his points, but in the interest of discourse I wanted to add a few of my own thoughts and comments.

Inverted thinking and the burden of proof

Blake starts by talking about “inverted thinking” and the “burden of proof”, stating that when testing software, you must start by assuming that the software does not work, or you will not find any bugs. I interpret “inverted thinking” as a mitigation of confirmation bias. In the context of software testing, confirmation bias means being oblivious to indications that the software is not working, and only noticing results that in our mind confirms that the software does in fact work.?

I find cognitive biases immensely fascinating, and highly relevant in most situations, and maybe in particular when it comes to software testing. Conformation bias is probably the bias I think about the most, closely followed by priming bias. As a Swedish expat, another personal favourite is the IKEA effect: valuing something you created yourself higher than you would have valued the same thing, had someone else created it. In software development, this manifests itself as placing disproportionally high value on, or faith in, our own code.

When I read the “burden of proof” I immediately think of the scientific method and hypothesis testing. Once thing I learnt in academia is that you can never prove a hypothesis or theory, you can only prove them wrong. Testing works the same way: you can never prove that the software works, you can only identify ways in which it does not work. A test that passes only shows that something worked once under a very specific set of circumstances, and that result cannot logically be extrapolated to apply to all circumstances, and every time the test is run. That is a logical fallacy.

Empathy and roll-playing

I agree with Blake that being able to take on a different perspective, and see the software from the end user’s viewpoint is a critical skill of a good tester, but I am not sure I would put as much emphasis on empathy and roll-playing. I think what you really need to be able to put yourself in somebody else’s position and understand how they would approach and use the software is curiosity, and likely also humility.

I have never done any kind of acting or roll-playing, and it is something that is very alien to me. I do think that I am empathic, but with a focus on understanding the feelings of others rather than sharing them. Yet I think I am a good tester, and that I can see things from different perspectives, being a good user advocate. Partially because I am humble enough to understand that the software was not built exclusively for me, and partially because I am very curious about and want to understand other people, and how users could possibly interact with the product.

Challenging assumptions

Blake points out that we talk about challenging assumptions so often that it has turned into a cliché, and I agree. I also think we still need to think and talk about it, and that Blake was right to include it.

Challenging assumptions means questioning, which is an essential part of critical thinking, which in turn is an essential part of software testing, and life in general. Questioning is another way to mitigate cognitive biases, in particular priming bias. However, being skilled at critical thinking and questioning includes recognizing where to draw the line, or maybe more bluntly, the point at which there is diminishing return, and instead increasing frustration potentially leading to unnecessary and destructive conflict. I agree with Blake that challenging assumptions is not an excuse to turn every discussion into an esoteric philosophical debate.

Blake states that the critical skill is to evaluate assumptions, and I agree. The process of challenging, or questioning, has no value in itself in the context of testing, only the output of the process does. Keep in mind that critical thinking is the process of analyzing information to form a judgement; the purpose is to arrive at some kind of conclusion.

Intuitive thinking and exploratory behaviour

I wholeheartedly agree that testing is a non-linear and nondeterministic activity that requires intuitive thinking and exploratory behaviour, and I strongly encourage everyone to look up Blake’s reading suggestions.

My only addition to this topic is a brief discussion on intuition. We often interpret intuition as something innate, a skill or ability we have rather than acquire. I think this is an unfortunate view that reduces the value of intuition, and our willingness to trust intuition.

Intuition allows us to grasp something without cognitive reason, but our intuition is still informed by our experiences and our other skills. The only difference is that we do not have to, and cannot, consciously apply it. The beauty of intuition is that since it does not rely on our conscious cognitive reasoning, it is probably the most creative and unbiased skill we have since we have no direct control over it. We just need to allow ourselves to listen to and trust it.

Recognition of human nature

I do find it funny how we like to think of technology as something neutral and distinctly disconnected from humans, whereas all technology is, as Blake correctly implies, a product of our human nature, with all our flaws and shortcomings built into it. Rather than futilely trying to make technology less human, I think we need to make it more human, and I agree with Blake that we must recognize that any and all technology is the result of human efforts.

Summary

I really enjoyed reading Blake’s article, and especially how it triggered all these thoughts and prompted me to sit down and think about software testing in a way that I have not done for quite a while. Discussing and debating is an essential part of how I grow, and a necessary step for me to refine and “test” my own ideas. Thank you, Blake, for sharing your thoughts.