Software Metrics--Testing

Software metrics are quantitative measures that enable software people to gain insight into the efficacy of the software process and the projects that are conducted using the process as a framework. Basic quality and productivity data are collected. These data are then analyzed, compared against past averages, and assessed to determine whether quality and productivity improvements have occurred. Metrics are also used to pinpoint problem areas so that remedies can be developed and the software process can be improved.

Three Types of Software Metrics are available

• Product metrics
• Project metrics
• Process metrics

Product metrics describe the characteristics of the product such as size, complexity, design features, performance, and quality level.

Project metrics describe the project team’s ability to perform the project execution; Examples include the number of software developers, the staffing pattern over the life cycle of the software, cost, schedule, and productivity. Some metrics belong to multiple categories. For example, the in-process quality metrics of a project are both process metrics and project metrics.

Process metrics can be used to improve the efficacy of an existing process used in software development and maintenance. Examples include the effectiveness of defect removal during development, the pattern of bad fixes, and the Backlog of development and test team.

Product Metrics

Time to find a defect

                This is the effort required to find a defect. This shows how fast the defects are being found.

Time to find a defect = The cumulative hours spent on test execution and logging defects /    The number of defects entered during the same period.

Time to solve a defect

                This is the effort required to solve a defect. This shows how fast the defects are being solved.

Time to solve a defect =   The cumulative hours spent on solving a defect /

The number of defects solved during the same period. 

Defect severity index

                An index representing the average of the severity of the defects. Provides a direct measurement of the quality of the product—specifically, reliability, fault tolerance and stability. Two measures are required to compute the defect severity index, a number is assigned against each severity level:  4(Critical), 3 (Major), 2 (Minor), 1 (Cosmetic). Multiply each remark by its severity level number and add the totals, divide this by the total number of defects to determine the defect severity index.

 

 

Defect severity index= Total Cumulative Severity Index /   Total number of defects found

Test Coverage

                Defined as the extent to which testing covers the product’s complete functionality. This metric is an indication of the completeness of the testing.

Test Coverage= Number of Test Cases Executed X 100   /       Total Number of test cases

  Test Case Effectiveness

                The extent to which test cases are able to find defects. This metric provides an indication of the effectiveness of the test cases and the stability of the software.

Test Case Effectiveness =   Number of Test Cases that Found Defects X 100 /

                                                     Total Number of Test cases Executed

 

Defect Density

                Number of defects found per 1000 lines of code. – This metric indicated the quality of the product under test.

Defect Density =               Number of Defects Found   X 100  /   Total Number of Lines of Codes (thousands)

 Project Metrics

Test planning performance

                The planned value related to the actual value. Shows how well estimation was done.

 

Test planning Performance =  Actual Effort Spent on Testing ( In Person Hours or Days) X 100 /

                                                                Planned Effort for Testing (In Person Hours or Days)

 

Test effort

                Test effort is the amount of work spent, in hours or days or weeks in testing, in relation to the effort spent in the development activities, will give us an indication of the level of investment in testing. This information can also be used to estimate similar projects in the future.

Test effort =  Amount of Effort spent on Testing (In Hours or Days)  X 100  /

                                Total Effort Spent on Project (In Hours or Days)

 

Productivity

                Measure  the performance of an individual.

Productivity =  Actual output  X 100 /    Expected Output

• Number of Testcases Developed
• Number of Testcases Executed
• Number of Defects Verified

Process Metrics

Bad fix ratio

                Indicates the effectiveness of the defect-resolution process, plus indirect indications as to the maintainability of the software.

Bad Fix Ratio  =           total Number of Bad Fixes     X  100 /   Total Number of Fixes

 

Defect Removal efficiency

                Indicates the efficiency of defect removal methods, as well as indirect measurement of the quality of the product. Higher the Ratio better the quality of the product

Defect Removal Efficiency =    Number of Defects found During Testing       X  100  /

                                (Number of defects found During Testing + Number of defects found at customer place)

 

Development Backlog

                The number of defects that are yet to be resolved by the development team. Indicated how well the software engineers are coping with the testing efforts.

Testing Backlog

                The number of resolved defects that are yet to be retested by the testing team. Indicates how well the test engineers are coping with the development efforts.

Software Quality Metrics

Although there are many measures of software quality, correctness, maintainability, integrity and usability provide useful insight.

Correctness

A program must operate correctly. Correctness is the degree to which the software performs the required functions accurately. One of the most common measures is Defects per KLOC. KLOC means thousands (Kilo) Of Lines of Code.) KLOC is a way of measuring the size of a computer program by counting the number of lines of source code a program has.

 Maintainability

Maintainability is the ease with which a program can be corrected if an error occurs. Since there is no direct way of measuring this, an indirect way has been used to measure this. MTTC (Mean time to change) is one such measure. It measures when an error is found, how much time it takes to analyze the change, design the modification, implement it and test it.

Integrity

This measures the system’s ability to withstand attacks to its security. In order to measure integrity two additional parameters are threat and security need to be defined. Threat – probability that an attack of certain type will happen over a period of time. Security – probability that an attack of certain type will be removed over a period of time.

Integrity = Summation [(1- threat)  X  (1  -  security)]

Usability

How usable is your software application? This important characteristic of your application is measured in terms of the following characteristics:

• Physical / Intellectual skill required to learn the system
• Time required to become moderately efficient in the system.
• The net increase in productivity by use of the new system.
• Subjective assessment(usually in the form of questionnaire on the new system)

Size – Oriented Metrics

          These are derived by normalizing quality and/or productivity measures by considering the “size’’ of the software that has been produced. Lines of code often used as normalization value.

Typical Size- Oriented Metrics

• Defects per KLOC
• Dollars per KLOC
• Pages of documentation per KLOC
• Errors per person month
• LOC per person month
• Dollars per page of documentation

 

Why we need Metrics?

Software Testing Metrics are useful for evaluating the health, quality, and progress of a software testing effort. It would be almost impossible to quantify, explain, or demonstrate software quality without these metrics.

“We cannot improve what we cannot measure.”

“We cannot control what we cannot measure.”

AND TEST METRICS HELPS IN

• Take decision for next phase of activities.
• Easy tracking of the project status and also helps in presenting the statistics to the senior management.
• Analyze the risks associated in a deeper way
• Evidence of the claim or prediction
• Understand the type of improvement required to success the future projects

 

Conclusion:

Software testing metrics are the quantitative indication of capacity, dimension, amount or size of some attribute of a process or product.

“Metrics can be defined as STANDARDS OF MEASUREMENT”.