Forget about finding the perfect testing tool

In the software testing world, there is a lot of debate around testing tools. Whether it be the merits of Selenium vs Cypress vs Playwright, a comparison between UI Automation tools, or the many tools out there that are designed to help a team with various stages of test and defect management. The lists of tools out there continue to grow and if you were to believe a lot of the vendors out there – almost all of them are going to revolutionize your testing approach.

However, the reality couldn’t be more off. Not only are any of these tools unlikely to offer the value you are expecting from a testing and automation perspective. But there is a good chance you probably won’t even need them or at least most of the features that they offer.

Don’t get me wrong. This is not an article to criticize the approach of vendors or to even be critical of testing tools themselves. In fact, testing tools are incredibly useful and many of the testing tools out there provide value to you as an organization. Some of which I will be exploring in this blog over the year ahead.

However, the problem is that it’s not the tool that determines your success, but rather your testing approach and automation frameworks are what will form the foundation of your testing and quality successes. Tools are simply a means to achieve that success and it's purely how you use them that counts. And so rather than trying to look for tools to fix your testing issues, focus on the following things instead.

Now, as with all things, there are caveats and in the case of software development, it's vital to have tools like code repositories, CI pipelines, and defect management tools. You can’t always delay in trying to make the right decisions with these and so often I would suggest going with the many available open-source solutions if you are part of a new small team or utilizing the existing approaches if part of a larger organization and work with those to at least have something in place before approaching the below steps.

Understand the problem

Too many companies are stuck trying to find quick fixes for problems they don’t fully understand or rush into trying out solutions before fully addressing the real issue they are trying to solve. A tool – no matter how good - is never going to patch over your quality and testing issues and so before even looking into tool evaluation, take a snapshot of your existing software development environment and try and understand the existing quality and testing issues, the process and ways of working and try and address the issues identified without looking into tools.

You can do this by looking at the existing data you already possess around your software defects and testing processes, existing metrics, and by speaking to people in the teams. ?Embedding yourself into teams for a period of time is often the best approach and will allow you to first hand see some of the challenges that are being faced and then be best placed to know how to address the problem.

And depending on the issues or experience uncovered, it is often not a testing tool that is going to make the greatest impact on addressing any quality concerns and probably a matter of internal processes, team culture, and software design that will lead to the greatest gains. And those should be the things that you should address and rather avoid looking at tools until you have solved some of these issues.

Culture goes a long way

In my experience, most testing issues are best addressed by fixing team culture or an internal process. So, even if you are on the market for a new testing tool to address some challenges along the way, get your culture right first.

Things, like having full team ownership of software quality and testing, having developers take accountability for their unit tests and getting testing expertise involved early in the project, and providing them with a voice in design decisions, are the biggest things that will turn any testing and quality effort around.

These are things that tools cannot solve and will help you get your team and process into a healthy place where tools can actually start to add value. Not doing this effectively will likely mean that any tools you are looking to introduce or implement to improve software quality, testing, or team collaboration are unlikely to achieve the ROI you hoped for, and you will end up spending money unnecessarily. ?

Design for Testability

This is a complex issue that entire books have been written about. However, it remains vitally important that the software being developed is designed with testability in mind. There is no point in trying to implement testing frameworks and using tools if the application doesn’t support the testing strategy or integrate with the specific tools. So, do the research on what makes the software more testable and then you can start to put these measures in place in your design and development process. ?

If you’ve addressed the previous step on testing culture and gotten testing experts involved in software design, this should solve a large part of the problem.

Focus on the framework

The true foundation of any test and automation success lies in the framework that is in use. A framework is a foundation on which your tests are written and is what drives the respective test runners/tools to be able to execute against the chosen API/applications. It’s how you structure your different tests in a way that makes them more maintainable while removing the core logic to centralized modules and having functions that provide the relevant reporting that drive the most value.

Most of the time, these frameworks can be designed completely independently – though often developed in conjunction with some open-source approach to make things easier for the organization and reduce the ROI of the process. However, even in the latter example, teams need to be wary of becoming too reliant on a specific approach purely just because it makes your initial automation job easier. As an example, in the JavaScript world, there are many incredible packages and frameworks out there which can make it quick and easy to develop automated tests against a variety of different types of applications using proven frameworks and approaches to test writing. However, as with many open-source solutions – they are not always maintained, sometimes run into security vulnerability issues, or are simply made obsolete by newer and better packages, and then significant rework is introduced to realign the framework to the new packages.

This is not to say that rework and maintenance are necessarily bad things if it keeps the software and testing moving forward and up to date with the latest trends. You just don’t want to incur excess maintenance because you were tied into a specific approach that doesn’t adapt well to a new set of frameworks or tools.

Focus on your own unique solution

Companies, development philosophies, and applications are quite varied and different, whereas tools may be developed toward a specific solution or approach that might not be conducive to your own unique situation.

To maximize the effectiveness of a tool, you often need to use it in a certain manner or follow a set way of working. Most companies though often operate in their own unique ways or have diverse teams that all operate differently based on their need and culture. This means that teams either have to change their ways to get the most out of the tool or end up simply just trying to do the bare minimum to adhere to their current processes, nullifying any benefit they may derive. And from experience, I would say that the latter often wins out. ?

This is especially true for many of the different features that tools offer, which many companies barely use. Whether it be specialized reporting or traceability options, unique automation features that supposedly “streamline” the automation process, or various forms of analytics and AI that are designed to save development time. These are all useful things, but features like this can never be effective if the companies underlying approach to testing, design, and frameworks are not operating effectively.

This is not a limitation in the tools or their effectiveness, but rather just simply the reality that software development has become so varied and with many different types of applications and lifecycles, that it is very difficult for any tool to fit a given situation perfectly.

So, instead of trying to place emphasis on tools in your automation strategies or tech branding, rather focus on the culture in your organization and ensure that the company understands what it takes to build quality software. Once you have a universal buy-in that can save a lot of additional time as people will ensure the requirements and code are now individually of higher quality, then you want to focus on the formal test process and automation frameworks in place to ensure that you are in a position to test your software effectively. And it's only once you get these right that tools will only become useful.

Understand the real cost of tooling

This might seem a bit late to introduce this important side to the conversation. And while it’s a very important aspect of the tool conversation, it's one that is difficult to fully evaluate until you’ve really addressed specific testing and quality issues, got the right culture and framework in place, and understand the unique landscape that your teams fit in and how a tool might best be utilized in that given landscape.

Only then should we look into the cost of a testing tool that can be used. The truth is that all tools cost something. Even an open-source tool that might not have any direct licensing costs, will have a cost to set it up, migrate into your current ecosystem, and maintain. And it’s those costs that often add up the most long-term in an organization and so it’s worth evaluating the amount of time that you can afford to put into that effort long-term and using that as a baseline for where a commercial tool might save you value. Too many companies jump to tooling decisions too quickly based on pure licensing fees and arbitrary benefits without understanding the full cost of effort in adopting, training, and supporting a tool or changing approaches and this is often why the usage of tools doesn’t lead to the right return on investment, even many years later.

I’m not going to advocate for any specific approach here between open source or commercial tools, as both have their merits. ?You can read more about the different pros and cons here . What is important is that you fully understand the long-term maintenance costs of different tools and don’t just focus on the direct licensing costs of any given tool.

Be Tool Agnostic

It’s also important to not become too reliant on tools. Tools are great, but they evolve and there are constantly companies or open-source projects coming out with better approaches and features. If your approach to testing is too built around a particular tool and the way it works, you are unable to explore new better tools as they become available. And while you don’t want to jump around too much either in the search of the perfect tool that doesn’t exist, you definitely don’t want to be stuck in an old way of development and testing when better approaches exist.

Additionally, many of the best feature-laden tools aren’t free and while they are more convenient than having to get different open-source tools to speak to each other, you ideally want the freedom to find the best tools and approaches that work for you. And you can’t do this if you’ve built everything around one particular toolset.

Thankfully most tools these days do offer simpler ways of importing and exporting data that make transitioning between tools and the dreaded vendor lock a lot less likely, but even then, there are often certain procedures or built-in features in tools that you can become reliant on and make it difficult to switch a tool should you ever need to. ?

At the end of the day, it is simply just not worth it to try and build your testing processes and frameworks around tools and you need to go rather than focus on working with a framework that is agnostic of the different tools that are in use.

So, in closing, I would like to reiterate that I am not against the use of any specific tooling approach. In my career, I’ve used a variety of open-source and commercial solutions to varying degrees of success. It’s just never the tool that is generally the problem or solution in any area and it's all down to the implementation of your quality and testing strategy and tools can only supplement that. No tool can overcome a bad strategy and it really should be one of the last things that companies focus on when it comes to software testing.?