5 Key Test Metrics To Measure

Requirements Coverage:

Test Coverage can be a very vague metric if not defined properly. It will also not mean anything with a vague definition. However, it is very helpful if measured in the right way. One of the key to measure functional test coverage is by being able to track the tests to the requirements. This can be achieved by having a strategy/process in place for writing/creating tests. The number of test cases often doesn’t guarantee a good functional coverage.

There are tools that can monitor the code coverage when tests are run and can provide more details into test coverage. But there can be too much irrelevant metrics with this approach.

Testing Cycle:

One of the factors that determine the efficiency of the testing process is the length of testing cycle. Testing cycle is the overall time required to complete the necessary functional testing (automation and manual) before the software can be released. The automation test strategy should consider the testing cycle in designing the solution. A higher number of automation test cases is not necessarily the best solution, as it might increase the testing cycle and there by delay the release. The tests should be optimized based on the testing cycles measured in the previous releases.

Defect Age:

Testing does not end with finding defects. A metric that can give insight into cost of fixing a defect is the Defect Age. The Defect Age is the total effort required from uncovering it to deploying the fix after retesting. Defect Age helps improve the quality of the code over a period of time and enables the team to focus on the key pain points and figure out a quicker turnaround time for fixing defects.

Automation Test Ratio (UI : API)

A mature automation test suite would have a good balance of UI and API level tests. Neither one of them can completely replace the other. The scenarios should be automated at the appropriate layers for efficient testing. A clear understanding of the application architecture is required to segregate the tests at different layers without compromising on coverage. This is key to having an efficient test cycle time.

Maintenance Cost:

Automation test framework maintenance cost is usually determined at the beginning . Often it is represented as a percentage of the total effort to create the test suite. However maintenance effort becomes compounding over a period of time if the framework/strategy does not cater for maintenance time. Maintenance effort should be measured for every release and the test strategy/framework needs to be optimised to keep it minimal. There is never a consistent maintenance effort. Often this effort goes unseen and results in unexpected delays in testing cycle.