Software Testing Metrics and KPIs

Original Article Link: https://www.swtestacademy.com/software-testing-metrics-kpi/

In this article, I will explain you several software testing metrics and KPIs and why we need them and how should we use them. This article based on my experiences and understanding. Also, I will use several quotes from various books and articles. They are listed at References part of the original article.

I want to start with metrics. Metrics can be very useful as well as very harmful to your development and testing lifecycle. It depends on how to interpret and use them. In any kind of organization, people (managers, testers, developers, etc.) generally talk about metrics and how to do measurements in a right way. Some of them use very dangerous metrics to assess team members’ performance, on the other hand, some of them use relevant, meaningful, and insightful metrics to improve their processes, efficiency, knowledge, productivity, communication, collaboration and also psychology. If you measure the correct metrics in a right way and transparently, they will guide you to understand the team progress towards certain goals and show the team’s successes and deficiencies.

In lean software development, we should focus on to use concise metrics that lead our continuous improvement. In reference [1], there is a quotation from Implementing Lean Software Development: From Concept to Cash, by Mary and Tom Poppendieck[2007], stated that fundamental lean measurement is the time it takes to go “from concept to cash,” from a customer’s feature request to delivered software. They call this measurement “cycle time.” The focus is on the team’s ability to “repeatedly and reliably” deliver new business value. Then, the team tries to continuously improve their process and reduce the cycle time.

These kinds of metrics need “whole team” approach because all the team members’ efforts enhance these metrics results. Time-based metrics are critical to enhancing our speed. We should ask ourselves some questions based on speed, latency, and velocity. In the agile world, one of the most used metrics is Team Velocity. It shows us, how many stories point (SP) that the team tackles during a single sprint and this is one of the fundamental key metrics of Scrum. It is calculated at the end of each sprint by summing up the completed User Story points.

Lean development focuses to delight end users (customers), which should be the goal for all the team. Thus, we need to measure also business metrics such as Return on Investment (ROI) and business value metrics. If the main business goal of the software is to reach and gain more customers, we need to deliver a software which serves this purpose.

In another reference, Sarialio?lu states that “without any metrics and measures, software testing becomes a meaningless and unreal activity. Imagine, in any project, that you are not aware of the total effort you have expended and the total number of defects you have found. Under these circumstances, how can you identify the weakest points and bottlenecks?” [2] This approach is more metric based and with this way you should talk with the numbers such as:

• Total Test Effort and Test Efficiency (with schedule and cost variances)
• Test Execution Metrics (passed/failed/in progress/blocked etc.)
• Test Effectiveness (number of defects found in system/total number of defects found in all phases)
• Total number of Defects and their Priorities, Severities, Root Causes (dev, test, UAT, stage, live)
• Defect Turnaround Time (Developer’s defect fixing time)
• Defect Rejection Ratio (Ratio of rejected/invalid/false/duplicate defects)
• Defect Reopen Ratio (Ratio of successfully fixed defects)
• Defect Density (per development day or line of code)
• Defect Detection Ratio (per day or per testing effort)
• Test Coverage, Requirement Coverage, and so on…

In “Mobile Testing Tips” book [5], metrics importance stated as follows: “without the knowledge, you would obtain through proper test metrics, you would not know how good/bad your testing was and which part of the development life cycle was underperforming.” The whole team approach also critical on the metrics that you will measure and report. Some of the tips are listed below for the rest of them I suggest you read the book.

• Tell people why metrics are necessary.
• Explain each metric that your gather to all team members and stakeholders not only the test team.
• Make people believe in metrics.
• Try to be informative, gentle, and simple in your reports.
• Try to evaluate/monitor/measure processes and products rather than individuals.
• Try to report point in time and trend metrics.
• Try to add your comments and interpretation with your metrics.
• Try to be %100 objective.
• Metrics should be 7X24 accessible

Also, metrics categorized in three sections in the book. These are test resources, test processes, and defects. Resources metrics are about time, budget, people, effort, efficiency, etc. Process metrics is about test case numbers, statuses, requirement coverages, etc. Defects metrics is about the number of defects, defect statuses, defect rejection ratio, defects reopen ratio, defects root causes, defects platforms, defects types, etc. At last, the book states that metrics make the test process transparent, visible, countable, and controllable and in a way allows you to repair your weak areas and manage your team more effectively.

Some approaches such as Rapid Software Testing (RST) – Bach & Bolton states that you need to use discussion rather than KPIs and objective metrics. Bach emphasized that you need to gather relevant evidence through testing and other means. Then discuss that evidence. [3]

Also, Bach wrote that in “Lessons Learned in Software Testing” [4] book at Lesson 65, “Never use the bug-tracking system to monitor programmers’ performance”. If you report a programmer’s huge number of defects, he gives his all to fix his bugs and tries to postpone all other his tasks. Also, the other crucial mistake is to attack and embarrass a developer for his bugs. This will cause very big problems on team collaboration and whole team approach. The other developer team members also respond this action very defensively and they will start to argue on each bug and they don’t accept most of them. They generally tell, “This works on my machine”, “Have you tried it after clear the browser cache with CTRL+F5”, “This is a duplicate bug”, “It is not written in requirements”, and so on. Also, the worst thing is they may start to attack you on your testing methods, approaches, strategies, and skills. This causes a terrific mess in the team, big problems on team members’ communications, and reduce team efficiency.

And also Lesson 66 tells us “Never use the bug-tracking system to monitor testers’ performance”. If you start to evaluate the testers with the number of bugs they found, they may start to behave not intended way. They are starting to focus on easy bug such as all kinds of cosmetic defects, they try to focus only bug counts rather than questioning the requirements and examine all kinds of edge cases. They may report the same bugs several times and this also irritates the developers and leads waste of time. Testers less likely to spend their time for coaching to other testers or self-improvement activities etc. Also, it affects their psychology in a bad way. For example, in team A, if developers wrote unit tests which have %99 coverage and they also run main business flow tests before testing phase and the test environment, test data, network, etc. are very stable, then in these situations tester X may not find too many defects. On the other hand, in team B, if developers do not have the necessary things to do before the testing phase, this time tester Y may find too many defects. In these conditions, if you assess tester X and tester Y with the bug counts, this will be very unfair. You need to understand the reasons of the bugs; you need to question them not just only take into consideration the bug counts.

I totally agree with Bach on lessons 65 and 66. Assessing developers and testers with the bug counts leads too many problems. You need to focus on reasons of the bugs and improve your system, process, methodologies, strategies, plans, etc. There are several lessons related bugs in “Lessons Learned in Software Testing” book, it is worth to read it.

In 2004, Cem Kaner and Walter P. Bond published an article on metrics [6]. In this article’s first section, they stated that some companies established metric programs to conform the criteria established in the CMMi, TMMi, models and fewer of them succeed with them. These metric programs are very costly Fenton [7] estimates a cost of %4 of the development budget. Robert Austin [8] gave information about measurement distortions and dysfunctions. The main idea of that paper is if a company managed by using the measurement results and that measurements (metrics) are inadequately validated, insufficiently understood, and not tightly linked to the attributes they are intended to measure, then all of these actions cause the measurement distortions. Kaner and Bond proposed a new approach: the use of multidimensional evaluation to obtain the measurement of an attribute of interest. They conclude their paper in this way: “There are too many simplistic metrics that don’t capture the essence of whatever it is that they are supposed to measure. There are too many uses of simplistic measures that don’t even recognize what attributes are supposedly being measured. Starting from a detailed analysis of the task or attribute under study might lead to more complex, and more qualitative, metrics, but we believe that it will also lead to more meaningful and therefore more useful data.”

In my opinion, in this way, you may get very useful data to help and improve the testers. But in practice, it takes too much time and this will be understood as “micromanaging the details of tester’s job” from the testers and also if the team practicing an agile development framework, it will be much harder to follow this approach.

 In an another article [9], below suggestions is provided:

• When using metrics, we should go beyond metrics and seek qualitative explanations or “stories” being told about metrics.
• Measure humans and their work in numbers and use metrics to gain more information, use them as clues to solve and uncover deeper issues.
• Do not use metrics to judge the work of a human, do not reward or punish individual’s work. Programmer/Tester productivity metric is something that you need to avoid.
• Create an environment where metrics misuse can be minimized.
• Metrics are meant to help you think, not to do the thinking for you – Alberto Savio

I also agree above suggestions and from this point, I want to share some metrics with you. Actually, metrics may be endless. You may need to use metrics based on your software development life cycle, development framework, company culture, goals, etc. I will present you some software testing metrics below.

You can find the several metrics & KPIs and their details on original article page. Here is the link: https://www.swtestacademy.com/software-testing-metrics-kpi/

The last paragraph is crucial I want to share it here also.

As you seen and wrote in this article, metrics are limitless. Using some of them as a KPI may be dangerous. You can use them to see the status of your situation and try to question your system, methodology, deficiencies, for continuous improvement. If you use them as a rewarding or punishing tool, it will create an enormous pressure and stress on your team. Try to use them in a right way and effectively based on your beliefs, philosophy, organization culture, etc. but whatever you will do, please be Fair! Seek first to understand, then act! Empathize! Create Trust and Love! Help your team! Believe in them! These are my advice to be a “good” maestro.