Making a tool agnostic test framework
In my previous article, I wrote about building a scalable test framework that would meet the needs of your organization now and for the future. One of the biggest requirements in achieving this is to ensure your framework is not reliant on tools. A big mistake that organizations and teams make when they build their framework is to choose a tool that meets their automation needs and then build their framework around that. The problem is that tools and technologies change and what might be the best tool now is not necessarily going to be the best tool for your future needs.
Along with this, there is also a need to often utilize multiple tools based on the different needs of your software, whether it be front-end, back-end, device specific or often even security and performance testing. Building a framework around each of the different tools that you need is not only expensive, but a nightmare to maintain as your test grows.
Below is a short list of just some of the benefits you are likely to gain through building your own tool agnostic framework:
- Reduced maintenance overhead long-term
- Reduce learning curve for engineers as they only need to learn one framework
- Freedom to choose the tools you want without need to build a new framework
- Utilize multiple test tools within one framework
- Perform multiple levels and types of testing in one framework, include functional, unit, API, security and performance testing
Be prepared to see prolonged ROI
One of the reasons companies do this though is down to the speed of development. Building a framework around your tool is often a lot simpler because the tools come with an inherent set of functions and features that you can easily leverage in automating tests in a short space of time. The obvious problem with this is that you as a team then tie yourself to that tool and should you need to ever add or change a tool to your framework, you are unable to do this. This is difficult for companies, as they understandably want to see a fast ROI on any investment of effort they make (the average ROI on any automation framework is between 1-2 years, so it can be a frustration for businesses), yet in building a framework companies need to actually be willing to invest heavily in building these things all on their own. This might add an extra 3-6 months to the development of any framework, but the long-term benefits are easily worth it if you willing to be patient.
How to Build Your Framework
So what exactly do you build your framework with? Well, that depends on the coding skills in your company and the types of application you need to test. From experience, I have found building a Java-based framework the most extensible option, but if you are in need for high-performance execution, this might not be the best option for your business. Most testing tools these days can work with a variety of different languages and even if they don’t natively speak your language of choice, it is relatively easy to write a module that can translate this on your behalf. Just don’t let your tools dictate the language you choose.
Choosing your language is obviously the easy part. In designing an agnostic test framework you need to be prepared to write a lot of the following functions that tools often take care of for you:
- Reporting – Determining what metrics and how you want your tests to report on.
- Function Libraries – Test tools often include a wide set of function libraries that dictate how it behaves around certain browsers, UI objects and databases. While tools can still be used for this execution, your framework needs to include functions that handle this.
- Test Validation – Every test framework needs to have preset criteria for the types of things a test needs to look for and verify as passed or failed Test tools include a lot of these built-in and you will need to define these yourself.
- Test Input – Your framework needs to define how tests write tests that can be translated and executed. Catering for different type of data-driven file formats is often useful
- Error Handling – You will need to develop this whether you use a tool agnostic framework or not. Many tools though have a wide set of inherent features that assist with unexpected behaviours and your framework will need to take care of this internally.
- Object Repositories – While tools can still be used to help identify and execute objects in any given UI, they often interact with objects in a specific way.
From this brief list above it might sound like the creation of a tool agnostic framework is practically like building your own test tool and you would not be that far off. While you would still require an actual tool to help manipulate objects and execute your tests, you should not rely on it for anything else. Your framework at the end of the day is a software product all on its own that should be able to be invoked by anyone in the organization requiring to do some testing. It should be built, maintained and managed just the way you would any other product. I’ll be talking more on that in my next article in how designing a modular framework requires software engineering best practices.