12 Steps to Useful Software Metrics (Part 1 - Introduction & Selecting Metrics)

Abstract: 12 Steps to Useful Software Metrics introduces the reader to a practical process for establishing and tailoring a software metrics program that focuses on goals and information needs. The process provides a practical, systematic, start-to-finish method of selecting, designing, and implementing software metrics. It outlines a cookbook method that the reader can use to simplify the journey from software metrics in concept to delivered information.

?Introduction to the Twelve Steps

There are multitudes of possible software metrics based on all of the possible software entities and all the possible attributes of each of those entities. How do we pick the metrics that are right for our organizations? The first four steps defined in this article will illustrate how to identify metrics customers and then utilize the goal/question/metric paradigm to select the software metrics that match the information needs of those customers. Steps 5-10 present the process of designing and tailoring the selected metrics, including definitions, measurement function, measurement method, decision criteria, reporting mechanisms, and additional qualifiers. The last two steps deal with implementation issues, including data collection and managing the impact of human factors on metrics.

When I started doing software metrics, there seemed to be two schools of thought. The first said collect data on everything and then analyze the data to find correlation, meaning, or information. The second school of thought was what I call the shotgun method of metrics. This method usually involved collecting and reporting on the current "hot" metrics measurement or using whatever data was available as a by-product of software development to produce metrics.?

These methods are both what I call the Jeopardy approach to metrics. You know the game show Jeopardy – where they start with the answer, and the contestants try to guess what the question is. In the Jeopardy approach to metrics, we start with the metric and try to guess what it tells us about our software processes, products, or services.

There are problems with both of these methods. The problem with the first method is that if we consider all of the possible software entities and their possible attributes that can be measured, there are too many measures. It would be easy to drown an organization in the enormity of the task of trying to measure everything. One of my favorite quotes talks about "spending all of our time reporting on the nothing we are doing because we are spending all of our time reporting." The problem with the second method can be illustrated in Watts Humphrey's quote, "There are so many possible measures in a complex software process that some random selection of metrics will not likely turn up anything of value." [Humphrey-89]?In other words, Murphy's Law applies to software metrics. The one item that is not measured is the one item that should be measured.

There has been a fundamental shift in the philosophy of software metrics. Software metrics programs are now being designed to provide the specific information necessary to manage software projects and improve software products, processes, and services. Organizational, project, and task goals are determined in advance, and metrics are selected based on those goals. These metrics are used to determine our effectiveness in meeting our goals. The foundation of this approach is aimed at making practitioners ask not so much "What should I measure?" but "Why am I measuring?" or "What business needs does the organization wish its measurement initiative to address?" [Goodman-93]?b

Measuring software is a powerful way to track progress towards our goals. As Grady states, "Without such measures for managing software, it is difficult for any organization to understand whether it is successful, and it is difficult to resist frequent strategy changes." [Grady-92]?Appropriately selected metrics can help both management and engineers maintain their focus on their goals.

Step 1 – Identify Metrics Customers

The first step of the "12 Steps to Useful Software Metrics" is to identify the customers for each metric. The customer of the metric is the person (or people) who will be making decisions or taking action based upon the metric; the person/people who need the information supplied by the metric.?

There are many different types of customers for a metrics program. This diversity adds complexity to the program because each customer may have different information requirements. Customers may include:

Functional Management: These people are interested in applying greater control to the software development process, reducing risk, and maximizing return on investment.

Project Management: These people are interested in being able to accurately predict and control project size, effort, resources, budgets, and schedules. Interested in controlling the projects they are in charge of and communicating facts to their management.

Software Engineers/Programmers: The people who do software development are interested in making informed decisions about their work and work products. These people are responsible for collecting a significant amount of the data required for the metrics program.

Test Managers/Testers:?The people responsible for verification and validation activities are interested in finding as many new defects as possible in the time allocated to testing and obtaining confidence that the software works as specified. These people are also responsible for collecting a significant amount of the required data.

Specialists: Individuals performing specialized functions (e.g., Marketing, Software Quality Assurance, Process Engineering, Software Configuration Management, Audits and Assessments, Customer Technical Assistance) are interested in quantitative information upon which they can base their decisions, finding, and recommendations.

Customers/Users: The people who purchase and use the software are interested in the on-time delivery of high-quality software products and reducing the overall ownership cost.

If a metric does not have a customer, it should not be produced. Metrics are expensive to collect, report, and analyze, so if no one is using a metric, producing it is a waste of time and money.?

The customers' information requirements should always drive the metrics program. Otherwise, we may end up with a product without a market and a program that wastes time and money. By recognizing potential customers and involving those customers early in the metric definition effort, the chances of success are significantly increased.

Step 2 – Target Goals

Basili and Rombach [Basili-88] define a Goal/Question/Metric paradigm that provides an excellent mechanism for defining a goal-based measurement program. The Goal/Question/Metric paradigm concept is to identify our goals, determine the questions that need to be answered to determine if we are meeting or moving towards those goals, and then select metrics that provide information to help answer those questions. As illustrated in Figure 1, a goal may spawn more than one questions, a question can relate to more than one goal, and a metric can provide information to answer more than one question.

The second step in setting up a metrics program is to select one or more measurable goals. The goals we select to use in the Goal/Question/Metric will vary depending on the level we are considering for our metrics. At the organizational level, we typically examine high-level strategic goals like being the low-cost provider, maintaining a high level of customer satisfaction, or meeting projected revenue or profit margin targets. We typically look at goals that emphasize project management and control issues or project level requirements and objectives at the project level. These goals typically reflect the project success factors like on-time delivery, finishing the project within budget, or delivering software with the required level of quality or performance. We consider goals that emphasize task success factors at the specific task level.??These are often expressed in terms of the entry and exit criteria for the task.

When talking to our customers, we may find many of their individual needs are related to the same goal or problem but expressed from their perspective or in the terminology of their specialty. Many times, what we hear is their frustrations.

For example, the Project Manager may need to improve the way project schedules are estimated. The Functional Manager is worried about late deliveries. The practitioners complain about overtime and not having enough time to do things correctly. The Test Manager states that by the time the test group gets the software, it's too late to test it completely before shipment.

When selecting metrics, we need to listen to these customers and, where possible, consolidate their various goals or problems into statements that will help define the metrics that are needed by our organization or team.?

In our example, all these individuals are asking for an improved and realistic schedule estimation process.

Step 3 – Ask Questions

The third step is to define the questions that need to be answered to ensure that each goal is obtained. For example, if our goal was to ship only defect-free software, we might select the questions:

·????????Is the software product adequately tested?

·????????How many defects are still undetected?

·????????Are all known defects corrected?

Step 4 – Select Metrics

The fourth step is to select metrics that provide the information needed to answer these questions. Each selected metric now has a clear objective -- to answer one or more of the questions that need to be answered to determine if we are moving toward our goals or meeting our goals.?

When selecting metrics, we must be practical, realistic, and pragmatic. Avoid the "ivory-tower" perspective completely removed from the existing software-engineering environment. Start with what is possible within the current process. Once we have a few successes, our customers will be open to more radical ideas -- and may even come up with a few of their own.

Also, remember software metrics don't solve problems. People solve problems. Software metrics act as indicators and provide information so people can make more informed decisions and intelligent choices.

An individual metric performs one of four functions. Metrics can help us understand more about our software products, processes, and services. Metrics can be used to evaluate our software products, processes, and services against established standards and goals. Metrics can provide the information we need to control resources and processes used to produce our software. Metrics can be used to predict attributes of software entities in the future. [based on Humphrey-89]. A comprehensive metric program includes metrics that perform all of these functions.

A requirements statement for each metric can be formally defined in terms of one of these four functions, the attribute of the entity being measured and the goal for the measurement. This leads to the following metrics objective template:

An example of the use of this template for the “percentage of known defects corrected” metric would be:

A requirements statement for each metric can be formally defined in terms of one of these four functions, the attribute of the entity being measured, and the measurement goal. This statement leads to the following metrics requirements statement template:Having a clearly defined and documented requirements statement for each metric has the following benefits:

• Provides a rigor and discipline that helps ensure a well-defined metric based on customer goals
• Eliminates misunderstandings about how the metric is intended to be used
• Communicates the need for the metric, which can help in obtaining resources to implement the data collection and reporting mechanisms
• Provides the basis for the design of the metric

Next week in Part 2 of this 12 Steps to Useful Software Metric, you will learn about the steps for designing your metrics.

______________________________________________________________________

For more information about our Certified Software Quality Engineer (CSQE) Preparation course click here .

______________________________________________________________________

For more information about these webinars or to register for one or more of these webinars visit click here .

________________________________________________________________

? 2023 Westfall Team. All Rights Reserved