Getting the most out of your Test Automation

The first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. The second is that automation applied to an inefficient operation will magnify the inefficiency.

Bill Gates

I might have met a handful of those who have made automation work for them. If you are reading this article you are one of them who cares about how to bring efficiency to your Test Automation to achieve better return on investment. With the right set of processes and tools getting the most out of your Test Automation is certainly achievable.

Start with your team:

Is your team well suited to automate on their own?

Should your team consists of only Test Engineers, will your company be ready to invest in a Record and Playback automation tool which requires very little learning curve?

Do you want to bring expert or external help to start with the automation?

You may want to bring in automation experts in case you would want to hive off automation as a separate discipline or take external help to build a minimal framework that can later be included as a part of your automation ecosystem.

In my opinion automation works best if your team does it for two reasons:

a. They will get to learn something new plus they would easily understand the importance of automation and how it could help reduce testing timeline

b. They will also know which test cases are critical and must be automated

So it pays to up-skill your team by investing in an automation tool or offer training to do automation on their own. I have put together a number of links in this article that could help self-train. You may also engage a consultant during the up-skill phase to ramp-up this strategy.

Automate early

The biggest benefit of automating early is the return on investment.

Just like how testing early helps you identify issues early, automation also helps in finding issues early if you wish to adopt automation using the Shift-left approach.

Also, it helps you run frequently helping gain confidence in a release and therefore covering the costs of developing the scripts.

Objectives to consider while automating early

a. Forget end-to-end automation:

This Google Testing Blogpost discusses at length why end-to-end automation is not an optimal approach.

Why?

You will have to deal with a lot of flakiness if there are several integration points involved. A script failure could also take time to debug based on the complexity of the test script.

The more end-to-end tests that you own the longer it takes to run. The last thing you would want is automation taking more time to run whereas a Test Engineer would have completed it quickly.

b. Automate APIs first:

If your product use APIs, better to start with automating them first. Your UI layer is only good as the data it consumes. Also, your team would mostly use a tool to test the APIs, so why not automate it instead?

Use a contract driven approach to start with API automation. A contract is simply put “an API specification that contains end-points with requests and response along with the data type”.

The Developer and the Test Engineer can work out the contract during the design phase. The developer can publish the contract in a word document or WIKI or in a widely used knowledge base format. The Test Engineer can now start with API automation using the contract and run it whenever the Developer’s code is deployed.

Keep in mind to unit test individual APIs before you start with the integration testing. For a better understanding read this blog by Bas. Tools like Pact can come in handy for API testing.

c. Automate user workflows next:

Whether it is the shift-left approach or automating in a regression cycle, when you automate UIs, always automate a user story.

For ex: In an e-commerce website the user would most likely search for a product that they have in mind, view the product description, add it to cart, make the payment and buy it.

Although it sounds like an end-to-end case it is a story that binds all your UIs in a single workflow. This will also make it easy to debug scripts by taking you to the exact step should a script fail. It will help you complete the integration tests from an API perspective as well.

Automate what is necessary

a. Catch up with complex scenarios:

More often than not you may ignore a few test scenarios or postpone it for the future.

Do catch up later to see if you can automate it. See if there is additional work required in the framework or get creative in solving the problem using a different approach.

Keep in mind to automate only such scenarios that the user would absolutely use. You may as well cover this in a regular regression cycle if there are too many interaction points.

For ex: Few workflows in an e-commerce application may require a third party integration, like receive OTP or fill out a payment form. You may work with the developer to develop a script or demo numbers to automate such workflows.

b. Automate for better coverage

If you are starting with UI automation, then plan to start with a breadth of scenarios that will give adequate coverage to all modules in the system.

This will help visualise how automation will work and help plan coverage for new modules in the future. Automating a large set of scenarios for each UI module can have adverse side-effects like longer run times not fetching enough issues, script flakiness etc…

Making automation work

Assert expected behaviour

Most automation Test Engineers fail to understand that Automation complements your Testing. When you test manually you assert a system’s behaviour.

Automation fails when the script does not sufficiently assert the expected behaviour. This leads to bugs in production.

Work with the Product Manager, Developer to understand which element in an UI is critical to automation and add assertions to every scenario based on the test data and input.

Select the right test data

While automation helps in testing large volumes of test data, using Testing Techniques like Equivalence Partitioning, Boundary Value Analysis to select the right set of test data can save run time. Too much test data can be an overkill while at the same time you must note that critical input conditions must be always validated for system integrity.

Run early and run often

Once you have a decent set of API and UI scripts select the critical tests and split them into Build Verification, Sanity and Regression buckets.

Make the best use of CI/CD tools like Jenkins, GitLab or even Cronjob to run automation everyday or as frequently as possible.

Rerun failed tests

Flakiness in automation is a given, for a number of factors that could affect your scripts — poor network, environment issues, Workflow change, UI elements change, tools can be flaky or race conditions can affect your test runs.

Use build tool plugins like Maven or Gradle failsafe to rerun failed scenarios. This will eliminate unnecessary manual re-runs to see if the error conditions are due to flakiness or script.

Optimise your run time

It will be wise to automate certain test cases only if it takes more time for a Test Engineer to complete it.

Tips to reduce automation run times:

• Use parallel tests. Most frameworks allow you to run tests in parallel using distributed or fragmented approach
• Systematically review the time taken for each test case, reduce the number of steps or unnecessary assertions
• Remove unused test cases or tests that you think will no longer find bugs
• Identify overheads in the code, try to address them

Involve your development team:

• Your developer will know a thing or two about optimising the code. Have your developers review automation code as well as pull requests
• Learn coding best practices that could help optimise the runtime

Coach your team to make use of automation:

Have your team review daily reports and provide feedback.

Have them analyse failures or replicate them manually.

Get them interested in the testing time saved using automation which could help in writing better stories for UI automation.

Measuring ROI

While the key measure for automation is the time it saves, the primary measure that I would go for

• Is the number of bugs found using automation
• Time saved in the regression cycle
• The coverage % of my automation tests.

Conclusion

Automation is key to any Software Test Life Cycle. Automation test scripts have a very short life span while the costs involved in creating them can be huge. Be it adopting a test tool or involving an expert to build frameworks and to maintain them, it is important to get the maximum bang out of test automation within a short span. I hope this article will help you achieve it.