Don’t let AI be a distraction; if pen and paper are the best solution, use them.

Over the past 20 years in the testing field, I've seen it all: from pure manual testing to continuous automated testing integrated into DevOps pipelines, and everything in between.

I've lived through the highs and lows of each era: the initial excitement of new tools, processes, or concepts, and the eventual realization that there is no silver bullet.

Every new advancement brings its own set of advantages and drawbacks. These innovations are not meant to replace the old methods but to mix, match, or extend them, allowing different approaches to be applied based on the problem at hand.

As extensively discussed with the concept of the testing pyramid and its variations, each testing approach has its specific time and place. Whether you are shifting testing left or right, no single approach, tool, or process can replace the others. Instead, they should be planned and executed based on the context and as part of a cohesive whole.

According to Gartner, we are currently in the era of AI-Augmented testing. As systems grow more complex and the pressure to deliver faster with higher quality increases, software testing can leverage AI advancements to enhance both efficiency and effectiveness.

In an era where AI is being integrated into virtually everything, it's reasonable to foresee that AI can play a key role in addressing software testing challenges.

Industry research consistently shows that AI is a top priority for executives. For instance, according to the BCG AI Radar, 89% of surveyed leaders consider AI and GenAI among their top three tech priorities for 2024, while 85% plan to boost their spending on AI and GenAI.

I witness this firsthand on a daily basis through conversations with various companies and their teams, who are actively seeking to purchase AI-infused testing tools.

When I inquire about their underlying problem and why it's crucial to invest in an AI-based tool, I frequently hear poor responses and weak justifications.

I often wonder if any tool that addresses the issue wouldn't be enough. Yet, I frequently hear that AI-based tools are a top priority set by senior management, and there is a budget that needs to be spent quickly.

Sadly, the hype around AI is so intense that companies are prioritizing the purchase of solutions without first clarifying the problems they need to address.

This drives tools vendors across the board to quickly update their roadmaps to include “AI features” in their products, in an effort to seem modern and trendy and improve their chances of being selected by potential buyers.

Unfortunately, in many instances these AI capabilities are just smoke and mirrors or a shallow implementation of AI with no clear and measurable benefit to the user. Many so-called AI tools or features are simply wrappers around ChatGPT or smart algorithms, rather than genuine AI, GenAI or machine learning implementations.

Don’t misunderstand me, I’m not against AI. In fact, I’m building testing tools infused with AI. I firmly believe that as the AI field continues to evolve, it will lay the groundwork for creating more advanced and intelligent tools to support the testing discipline.

This post serves as a reminder for anyone seeking to improve their testing with AI: clearly define the problem you're trying to solve, understand the metrics involved, and establish a clear criteria of what success looks like.

Don’t let AI be a distraction; if pen and paper are the best solution, use them.

Always keep in mind that AI is just a tool, and a fool with a tool is still a fool. Avoid succumbing to the shiny object syndrome and adopting tools just because of the hype.

AI has the potential to significantly boost testing efficiency in various areas, including accelerating test generation and maintenance, addressing challenges in test automation (such as flaky test detection, self-healing tests, and root cause analysis), and managing test data and environments. These are just a few of the many possible applications.

However, AI cannot replace experienced testers, creativity, or good judgment. Realizing the full advantages of AI demands skilled testers capable of identifying what truly matters amid the noise generated by the AI tools.

Testers who get the most from AI are going to be those who are experienced and smart enough to use the tools to their advantage and are able to judge what is right or wrong.

On the positive side, testers who effectively take advantage of AI advancements will enhance their productivity, completing tasks more quickly and efficiently, which will give them more time to dedicate to activities that involve reasoning, creativity, and collaboration with their colleagues.