Using metrics for test coverage reporting

Imagine you are working on a mobile app project with releases every 2 weeks. Say a manager wants a report on test coverage for each release before they start the release roll out. How might you report on it?

One approach may be to have a checklist or a bunch of test cases documented somewhere and going through it with every release. This is fine for getting started but can become hard to maintain as a project grows.

I’ve worked on many teams with this approach and it always gets to a point where no one wants to mantian these tests or to keep them lean yet useful. No one wants to delete a test that was created 6 months ago even if it’s no longer adding value.

I’ve often used an analytics framework to help with this type of reporting.

For example, say each screen in the app triggers a unique page view event. Say version 1.0 of the app has 20 of these unique page events. As the team is testing version 1.1 of the app they trigger 18 page events you could say 90% of the app has had some test coverage.

Maybe the missing screens are harder to reach features or lower risk, for example an outage message. 90% screen view coverage might be enough for you to go, “no one has observed any new or critical issues with the testing so far, we might be good enough to start rolling out the next release”.

There are some drawbacks to this approach, new features that add more screens might mean that coverage is over 100% compared to the last version. Different user flows that have the same screen names might be missed too.

One benefit is that it can be easy to create custom dashboards depending on the analytics framework you are using. I will often set up a dash board with page views of old version and page views of new version side by side in a table view to help compare the two.

If you’d like to read more here is more related content, checkout these posts:

Metrics and Quality

Bug bashes

Risk based approach

Mobile app test strategy

This blog post is a series on data related topics co written with Uzma who is publishing content over at medium, will update this blog post with her content when she publishes it.