The Travelers of Testing and a Journey of Streets

Twenty years is not a long enough time. It’s too short to write the kind of things I am going to say here. However, twenty years of working in the field of testing is long enough to understand and realise how little I know. It's a long enough time to do experiments which are only partially successful. It’s also a pretty long time to realize that there are people out there who have explored a dimension of it, which is completely alien to me.

A Poem of Travel

My mind is not wired for building long term memories. I forget a lot of stuff. The opening of the Punjabi poem that I am going to share is not even the exact poem that I had read. I was 17. That was a long time ago. I’ve forgotten it despite its impact on me and my style of thinking. What remains is a version of it in my mind. I don’t remember its author’s name. Some years back I tried finding that out. I couldn’t.

?????? ?? ???

??? ?? ????

????? ?? ?? ?????? ????

?? ???? ????? ???? ???? ????

?? ????? ?? ??? ??

??? ?????? ???

“The history traveler, with a baggage of inventions on his head, started to walk.

And then someone sneezed behind him, and history forgot its way.”

?(In many Indian cultures, sneezing at the beginning of anything is considered a bad omen in the sense depicted in the poem).

This poem is so important that I ended up reciting it during the only meeting (and a pretty short one) with James Bach years ago. The topic was belongingness. To explain why I don’t belong to any particular school of thought, I recited the poem and translated.

The poem further goes on to tell that Time needs a highway to travel, but the travelers often venture into the streets and never come back.

I’ve always interpreted this poem in terms of what happens to knowledge in a profession. In short, this poem is the story of testing and testers for me.

I don’t belong to any specific school of thought. I am a pluralist. I believe that testing, as a field of knowledge, seeks travelers to travel on its highway. Its ask is not the streets into which many and most experts ventured and alas, never came back. Followers followed.

The One-Way Streets

????? ?? ?? ?? ?????? ?? ??? ?? ??????

??? ????? ??? ???? ?? ???? ??? ???

Someone reached his destination as soon as he left home

Someone, just like me, remained in a journey for life

?~ Ahmad Faraz

The journey of a highway can be boring and intimidating to some. The landscapes could be appealing. So, it’s not uncommon to take a break, venture into the roads and streets to explore.

To manage complexity in the subject of testing, we coined terms and drew distinctions. The act of drawing such distinctions gave us an opportunity to go deeper. This is also the way humans innovate and develop unique styles. What did we do with these solutions later? Did we amalgamate these solutions or let the distinctions stay in their respective universes?

There is a fundamental question to be raised here.

Do we have the intelligence for contradicting ideas to co-exist in our minds? At the least, do we intend to do so?

If not, how do we plan to fill the inherent gaps in one style that could be filled from another style of doing things?

Around the time I got serious about testing, Exploratory Testing, Context Driven Testing and Risk Based Testing were some such distinctions and types. I am taking these names as examples, and also because the related work had a big impact on me. They highlighted important aspects of the art and craft of testing. They gave an opportunity for testers to develop methods, techniques, tools, heuristics and so on for focusing on those aspects. To this day, some of the related work I read during that time, helps me.

This was also the time when some of the well-known testing certifications were taking shape. The premise was to create a ground for organised learning paths and evaluation of a tester’s testing skills. That was the promise they made to the overall ecosystem.

It was also the time when I was fighting my own demons. Rather than on a street of innovation, I was on a path of ego. I was focused on performance engineering and I looked down upon “manual testing” or even functional testing. I considered my own work superior to these and by its extension, me as a superior tester. I am not proud of that time.

A few years later though, I came down from this high cloud. I realised how deeply flawed my thinking was.

It was the time when I expected the other streets to join the highway back. Exploration, Context and Risk should have flown back to the mainstream of knowledge rather than staying as a long-term distinction of types. They never truly did.

I started questioning this in my circle. I asked, shouldn’t every tester develop exploration skills? If there is something called context-driven testing, what is non-context driven testing? Can we still consider that testing? If there is something called risk-based testing, what is non-risk-based testing? Can we still consider that testing? In a way I believed in the importance of these ideas more than their originators. That’s the irony. However, humans are usually happy with categorised, even cut-down versions of concepts as they are more manageable. In short, I was shut down. We live in a world where more than the idea itself, what matters is who is promoting the idea. I was (and I am still) a poor, small promoter, I guess.

Another irony is that the followers of the leaders/adventurers who create and venture into these new streets end up picking only punch lines and one-liners from their leaders. They don’t truly or deeply study their leader’s work.

One such example is that when the main minds behind Exploratory Testing, conceptually merged it years later, and said it’s what is Testing (although that means a street replacing a highway), many and most of the followers continued using Exploratory Testing as a term even to this day.

On the same lines, the certifications became more of a business and ecosystem problem rather than fulfilling the original premise of learning and self-evaluation.

Certifications which tried to replace them mostly failed to scale but never admitted openly to how this absence of scale raised a question on the whole premise of those certifications in the context of overall ecosystem. The problem of a good certification at scale is an unsolved problem to this day. Whichever exist are street certifications. And more often than a not, even a poor version of what that street demands.

On the certifications front, I volunteered in various capacities to bring about some changes. I must confess, across all work that I’ve done in my career, I wasn’t able to do much worthwhile in this area. The changes I could bring were too small. I created a certification too. It had its own flaws. I tried to improve it. That didn’t sell because all participants continued to fail. Are certifications about knowledge or for practical purposes they have been systematically toned down to the level of knowledge that exists in the wild amongst testers? Is it a design/creativity constraint on part of the creators or is it a business compromise?

Automation in testing had the same fate. Black-box testing vs White box testing were such streets too. Static and Dynamic testing were these streets as well.

Pick a term and you’ll find how silly, questionable walls have been created around those concepts. This caging has killed many opportunities of furthering knowledge in testing.

The types of testing slowly became types of testers. The distinctions became self-imposed cages on skills. Black-box tester is not an uncommon word to hear these days, for example. Exploratory testers, White box testers, automation testers (what does it even mean? ),… - the list goes on.

In the false dilemma of superiority in an x-vs-y game of words, common sense often takes a back seat.

Critical concepts like controlled repetition are often frowned upon, treated as cheap skills and often made straw men to sell agendas. Well established concepts like that of regression testing are discussed less and less, letting misinformation propagate, as no one can take credit for inventing their names.

In short, everything else what one does not sell himself/herself is snake-oil, biased or plainly wrong. If the word context has any weight in testing, how can such extreme opinions of right or wrong exist in the same universe as the word context?

The distinctions and the vocabulary which were supposed to be furthering the craft ended up damaging it. This damage is at a point, where the highway is almost forsaken. What we are left with is a notional highway and most, if not all, travelers traveling on the streets.

The Ignored Streets

Most of the testers are focused on testing functionality by treating test objects as black boxes. These streets make the most noises. A damage of this majority bias was the lost potential of generalisation of concepts from specialisation streets.

Those who tend to specialise in fields like performance engineering, security engineering etc stop using even the word Tester in their profile at some stage.

In these fields, many non-testers have great innovations to their credit which we all as testers could have learned from. However, it has been as if those who were focused on functional testing took charge of what the core of testing should be about. The core test design techniques, for example, exist in the shape and form which suits basic functional testing needs and continue to be taught and discussed in that manner. We could have collaboratively helped these streets in joining the highway. But how we could have done that, when the foundational thought was to move away.

As an example, Equivalence Class Partitioning, a classic foundational testing technique is heavily limited when it is seen only in the way it is seen in most testing books. There is functional testing bias. Usage of this technique can be seen in performance engineering world as well in a more open-ended manner. One has to apply sampling to use cases, test data, interactions etc. to come up with a workload distribution model. Similarly, the same approach is used in security engineering. That’s the forward direction. In a backward direction of thought, there are many extensions to the technique used in these fields, e.g. the concept of payload construction, the special meaning of special characters, sequencing of characters etc which can be brought back to the mainstream stance on equivalence class partitioning. We can do the same to boundary value analysis, decision tables, state transition testing and so on. This bi-directional exchange of thought is lost when the streets decide never to join the highway.

It can be better understood with some further examples.

Most of the testers have heard and loosely used the term - Defect Taxonomy.

Just do a cursory search. Try to find out which field of work and who has done decent work in the field of defect taxonomies. Barring a thesis guided by Cem Kaner, a few pages in Boris Beizer work, and some random articles/blogs in the functional testing world, there’s nothing. Now check how the security engineering world has treated this area. Look at the maturity of taxonomies there. Check CWE website, for example. While we as testers were busy in proving which of our streets are better than others, there was relevant knowledge creation happening elsewhere. We could have learned the patterns. We could have built on those patterns to use them in various other areas of testing. We didn’t.

Another example? Look at white-box testing. What have we done as a testing community in this area? Search for publicly accessible information. If one wants to do white box testing, do we really have rich white-box testing literature created by testers for testers? Compare it with the black-box counterpart to understand the majority bias.

What about testing tools? How many of these tools are created by testers for testers?

You might be tempted to take a few names as answers here. Pause and think. Is the totality of their work a justification to the questions I raise here?

There are only a few and far pockets of hope. Not sufficient. The names that come to your mind need more encouragement, reach and space, otherwise the black-box, functional bias is here to stay and dominate most of the discussions in the field. If their knowledge remains in respective streets, I don’t foresee any concrete large-scale change happening in the coming decades, in the way we treat overall testing knowledge.

For how long, are we going to be busy in debating and drawing distinctions amongst error vs mistake vs fault vs bug vs defect vs failure vs discrepancy vs issue vs incident? Manual vs automated testing? Can we throw these words/distinctions at a real-world problem and solve it?

In a world which does not take testing seriously and “anyone can do testing” is a popular opinion, should testers too behave in a way to treat testing in this manner? Yes, anybody can talk about testing. Doing testing and doing it in a focused way is what takes deeper thinking.

When a tester makes testing as a street problem, s/he is (even if unknowingly) giving a nod to the statement that testing is easy.

James Bach in a recent comment on my post on LinkedIn said that he had expected that over a period of years, every tester will have his/her own mental model of testing and will defend it with reasoning. That’s an important and beautiful thing to say. In my opinion the streets are not transient, because most testers are followers. The day when each tester has his/her own mental model, the highway will automatically unfold. As that is just hope, till that point I guess the leading voices themselves will need to show the path from the street to the highway.

We are testers. Despite carrying unique, and many a times contradicting, ideas, we share this bond.

When I use the word tester here in this article, I am not addressing guests – those who are in testing temporarily and would walk away at the first opportunity.

Those of you and me, who love testing and are here to stay, can’t we just get along? To prove an idea is great, should we continue on the path of constantly proving that the other idea is less worthy?

?Let me take an example.

“Manual Testing” as a term is considered demeaning. So, some testers challenge this term. The thought process is more or less the same when somebody uses the word ‘resource’ for a human. If it is used to demean, I can understand the reasoning here. I can respect when someone interrupts and objects to the usage of this term to say - there’s nothing called manual testing.

Having said that, I see this at the other extreme now as well. The moment somebody uses the word ‘manual testing’, many testers without even thinking or reading the content fully, pounce on it. “There is nothing called manual testing”. “What do you mean by manual testing”? Such a question or remark is the only point of discussion they care about. All of this, without attempting to understand whether the person used the term to demean human role in testing. Goal displacement? Isn’t ignoring a testing colleague’s core point of discussion also demeaning? Aren’t most of such remarks done by testers who want to sound smart and ridicule others? Without exaggeration, this extreme in the testing universe is seen when an influencer who himself shares copy-pasted contents on testing interview questions, says – “Humans have the complex neurological system which gives them an amazing sense of experience.” This goes unchallenged as expected. After all, it will be politically incorrect to do so and will be an invitation to get trolled. The damage it does is that it dilutes similar commentary by those few testers out there, who have spent their career in exploring and talking about human role in testing.

Words are important. I get that. But only the words are not important. Context in which they are used is equally important. More than both of these put together, applied knowledge is important.

Testing debates are frequently stuck these days in the war of word choices rather than the subject. If the goal is awareness, then let me point out something. There are only a few testers who are publicly vocal. The silent ones are increasingly becoming disinterested in testing subject knowledge. The reason is that the vocal testers give them the impression of more of an Instagram/YouTube influencer constantly either delivering the same message for the 100th time or blindly roasting others over word choices.

Are we selling popular opinions in the disguise of debates or ideas? Have we converted the idea of a “type of testing” to a “type of tester” and then to “type of testers” who cuddle together in costume parties?

Do we actively ignore misinformation as long as it originates from testers in our own street and is in favor of what we promote? Are we serving testing or personal agendas?

A New Street

As we speak, a new street is already founded and is claiming its own share of travelers.

The Artificial Intelligence/Machine Learning street, already has and will have many experiments and innovations relevant to testing. Across the streets, there are contrasting emotions and reactions – that of excitement as well as that of fear. The experiments on the street are about testing AI/ML systems as well as using AI/ML systems in testing. There are a lot of opportunities.

Watch out, or this is going to be another one of those streets which does not come back to the highway. You’ll think AI/ML will innovate new stuff in testing. It will do so. However, it will do so on its own street. If we don’t change our mindset, this street will never join the highway.

Those who want to travel the street, remember that it has a testing context and in the absence of larger context of knowledge already created elsewhere, you will be either re-inventing some wheels unnecessarily or utilising information and tools already available.

Evidence of both of the above problems co-existing is out there as I write this article. It’s just the beginning.

A Call From the Highway

I am a minuscule part of testing community as well as what the testing highway is about. I don’t wear an expert hat. I am a student. My relation with testing is that of servitude and love. In my limited circle, I show the variety of streets. I’ve raised the questions in this article innumerous times. I’ve not taken sides. I don’t have any favourites. I don’t write to abide. I don’t write to sound nice.

I decided during the first few years of my testing career to travel on the highway of testing. I constantly venture into the streets to learn new ideas but I come back to the highway with the new learnings.

Many things which need to be done cannot be done alone.

Do you empathise with the theme of what I wrote here? Are you in a street and would want your work to reach its full potential? Are you already a traveler of highway, trying to make sense of it all?

Let’s join hands. Let’s talk and put collaborative efforts in creating a body of knowledge which amalgamates knowledge from the streets back to the highway.

It will not happen automatically. Abandonment of the highway took explicit steps. Amalgamation will need explicit steps too and even more involved efforts.

In case you’ve read it all, thank you for your patience. That’s all for now, with the following food for thought:

?????? ??? ??? ?? ???? ??? ????? ?? ????? ??

????? ??? ?? ??? ??? ?? ????? ????? ??

Although there are drops and drops of water in a river

A drop is the drop in which one can see the river

?~ Krishna Bihari Noor

PS: I wrote this article for Tea Time for Testers magazine. Republished here as it is a part of the continuity of my thought process in this year in the form of LI articles, posts, webinars, AMAs, talks etc.