Beyond Automation: Mark Winspear on the human touch in testing

From time to time, a buzz will appear in the world of IT, including testing, which suggests a seismic shift to ways of working is on the way.? When this happens, it’s important to dig into and understand the reality of the real-world impact and effects of these developments as part of evolving good practice, but also to take time to educate people outside of the specialism as to what it actually means. I’m going to talk through an example of something here that you may have heard – variations of “All testing can be automated” or “Manual testing won’t exist in X years’ time”.?

?

Firstly, it’s important to point out that this is an article to inform and enlighten its readers, so to give you some pertinent aspects of my background and mindset, I am a huge supporter of test automation and have delivered highly successful test automation frameworks in two organisations, one of which was responsible for calculating ￡bn in taxes (Any issues with that system absolutely would have made news headlines!). If done correctly, automation will certainly save time and money, it’s an important part of any delivery and should be considered in all circumstances.?

Automated checking

I used the phrase “Test automation” in the previous paragraph as it is a term familiar to many in the field of IT, but to move forwards, I encourage you to rethink two terms used when it comes to testing, reframing them so that instead of using “Manual testing” and “Automated testing”, you instead have the terms “Testing” and “Automated checking” in your minds. I will go on to explore why they are very different things, but with both being essential components of delivery.?

?

In the same way that developers use tools to automate certain tasks, testers use tools, including those which can automate some of the pre-determined, repeatable checks that can be made on a system, and which save time compared to a human executing the same checks. Is this testing? Well, a human tester certainly should have assessed the requirements and designs and crafted the tests to check the outputs given a set of inputs, but during execution, this narrow window of…

 check output > pass/fail > move to next check        

?…is all that is executed.? It’s important to understand that a controlled environment is necessary to run reliable automated tests which may not simulate real-world conditions of a dynamic and unpredictable environment. During execution, there is also no observation of the system, no adaption, and no exploration. Therefore, we can’t rely solely on automation.?

What is testing?

So, if that is automated checking, what is testing?? Testing is a risk identification, exploration, and reduction exercise. The identification and exploration cannot be done in any other way than via a curious, experienced, skilled, and creative human tester. Some of the risk reduction can then be done by machine because that expert hand has crafted something to do that, but a human tester has Contextual Understanding – understanding business requirements, user behaviour, real-word scenarios, industry regulations, changing accessibility, security or other non-functional guidelines and standards. Human testers excel in recognising and adapting to changing contextual factors during testing.??

?

Freeman & Pryce explain the evolution of testing, especially in the agile world as: “Testing is no longer just about keeping defects from the users; instead, it’s about helping the team to understand the features that the users need and to deliver those features reliably and predictably”.

In my view, testers are the glue between the Product Owner and Developers/ Engineers. They are who the Product Owner look to give them confidence that the developed system is good quality and fit for purpose by understanding and advising on testing levels and coverage and knowing when to apply the best approach (Context-driven testing). They practice critical thinking to deal confidently, thoughtfully, and pragmatically with difficult and complex situations. They accept and deal with ambiguity, situational specific results, and partial answers.?

?

So, focusing in on the tester collaboratively working with team to assess the requirements.? What are they doing there? They are thinking creatively (a skill all good testers possess), they are identifying potential test scenarios and asking questions which drive out further and clearer requirements, they are thinking about how these fit into the overall system and what the impact could be on the whole system, and this might drive out more requirements and more testing, and they are engaging? business users to discover how they use a system or process and thinking about how that could be tested, or how it could be misused in order to model tests.?

Live example

Let’s go on to peer into a specific example of some testing in action: A team is introducing a new application or site. We are in PI planning and the question comes up (or is raised by a tester potentially) about how we can ensure that the application doesn’t fall over when people access it.? The obvious answer is that we test it – of course, and part of that is exploring the risk profile…?

?

Tester: When will people use this? Do we expect peaks of usage in the day, week, month or at specific times of the year? Will we have campaigns that will drive people to the application??

Scrum Master: That’s a good thought – let’s discuss adding a new Feature for Campaigns with the ART Team.???

[team discussion occurs]?

Tester: How many people do we think will use it at low load/ average load/ high load? Let’s quantify what Low, Average and High means. Should the service degrade gracefully or how should it react when usage exceeds our expectations, either as a spike or prolonged load???

[more team discussion]?

Scrum Master: Ok, we have some good acceptance criteria to describe those scenarios with placeholders, but we’ll raise a story, go away and do some analysis so we can plug in meaningful numbers that we are prepared to stand behind.??

[A new story is raised]??

Tester: What monitoring and alerting is going to be in place??

[more team discussion and a new story to implement monitoring and alerting results]?

Tester: [Makes a note to test this monitoring and alerting].? What infrastructure shall we setup to test on? ?Should the environment be scaled to production level, so testing is meaningful in the real world?? That could be expensive, or shall we look at something that could be scaled up and down? And what do we actually mean by scaled – we need to define and agree on that.??

Analysis

As you can see by this (highly simplified) exchange, we’re testing – we’re exploring risk and exploring how we can meaningfully put in place tests to reduce that risk. Can this be automated? A tester can craft and automate the actual performance testing once we have defined the base and peak load and designed the tests we’re going to perform on that loaded system, but the conversation, the thinking, the exploration, the heuristics used, the design, analysis and breaking down the various elements cannot. (Note - In reality, some of this discussion about environments would be upfront to enable the creation of a ‘walking skeleton’ to test the infrastructure before we start building anything.)?

?

The skill of asking effective questions is pivotal for testers, as it drives deeper understanding and uncovering of issues that might otherwise go unnoticed. Skilled testers engage in critical thinking and curiosity, probing beyond surface-level functionality to identify potential flaws, ambiguities, and areas for improvement. This ability to ask insightful questions ensures comprehensive coverage of test scenarios, leading to more robust and reliable systems. By continuously challenging assumptions and seeking clarity, testers contribute to early detection of defects, often before any implementation has started, they enhance collaboration with developers and stakeholders, and ultimately facilitate smoother and more efficient delivery. The result is higher-quality products that better meet user needs and reduce the risk of post-release issues.

Misconceptions

Modern testers are engaged with teams, thinking about use and misuse of products, exploring, identifying, collaborating, mitigating risk through testing, but they are also coaching the team in quality practices, finding opportunities to shift testing left, for example, pairing with developers on improving unit tests.?

?

Modern testing also employs exploratory testing which is not the same as ad-hoc testing. Exploratory testing has structure and intent. While ad-hoc testing is informal and lacks a predefined approach, exploratory testing is more systematic and involves simultaneous learning, test design, and test execution. Exploratory testing is usually guided by charters—brief mission statements that define the scope, objectives, and focus areas of the testing session. These charters help ensure that exploratory testing is goal-oriented and covers critical aspects of the application, unlike ad-hoc testing, which can be random and unstructured. Charters provide a framework for testers to be creative and investigative while maintaining a clear direction and purpose.??(For more on charters, see Elisabeth Hendrickson 's excellent book 'Explore It!')

Why is this important?

If done correctly, test automation can absolutely save time and money.? Even in the non-software world, my Test & Quality team seeks out and implements opportunities for automation, for example, testing network device functionality and connectivity, and smoke suites to run before and after any releases on the network to ensure we pick up and can resolve any issues before users do.?

?

Automation sounds exciting, and it is, so what we need to ensure is that we are accurately presenting the benefits.? Many of us will have seen statistical headlines that show by using test automation, we’ve saved any number of years’ manual effort. It makes a great headline, right? But often when you dig into the numbers, the basis and assumptions for these metrics can be unsound.? In reality, would we execute the same 10,000 tests a day, every day for the next year? No, we would have sub-sets of targeted regression suites and smoke suites that were run in the appropriate environment at the appropriate trigger points.? This is the comparison in effort we have to use to ensure that we have credibility.? And it’s not a one-time exercise, the human tester must assess constantly which sets of tests are required for each scenario, updating, creating and retiring the tests, parameters, data and the test suites to reflect changes in the application, libraries, network, integrations, tools, browser and devices.??

?

The importance of proper testing cannot be understated.? I wrote an article about the Post Office scandal, which demonstrates what happens when testing is not done properly, which in that case was ￡bn financial losses and a tragic human cost, a loss of businesses, properties and sadly lives. In a multi-supplier environment, a unified, client-led approach to testing is critical for ensuring quality. We will often create a 'Plan on a Page' which shows where testing fits into the team approach at each stage. This lightweight approach ensures everyone is on the same page when it comes to testing which is critical when you consider, like the Post Office should have done, that the client carries the risk.?

Mark Winspear is a highly experienced professional in Test & Quality who has dedicated his career to building and leading teams that drive excellence and efficiency with his leadership philosophy rooted in the 'One team' principle. Through his leadership, his current Test & Quality team achieved a Net Promoter Score of +92 (NPS is an industry recognised customer experience metric used to gauge customer loyalty, satisfaction, and enthusiasm on a scale between -100 and +100, by asking the consumer directly about their experience), so this is an outstanding achievement.?

Connect with Mark on LinkedIn and please do leave your thoughts and comments