Measure What Matters

The first step in measurement is to make sure that it matters. As the saying goes, “what gets measured gets managed.” I’ve seen this in practice and it is a true statement. When the measure matters this really works well. It works poorly when we measure for the sake of measuring or when we measure those things that are easy to measure rather than something that is really valuable to measure. Measuring takes time and costs money, which is why the first step is to make sure that it matters.?

I worked at a mapping company for several years. I love maps! There are many parallels between the activities of mapping and measuring business operations. Maps are a simplification of the infinite complexity of the geography around us. Maps are a visualization of data, the important data that really matters. In the building of our maps we had lots of data. We knew when data was entered and what type it was. We knew when data changed and who changed it. Our engineers and data scientists could compile many different views of data entry into our map. The problem was that most of these were not terribly helpful. We had too many Key Process Indicators (KPIs) that didn’t really help us manage the business. All we really wanted to know about was the quality and the cost to make changes to our maps.

Measurement can be very costly. To measure the quality of our maps we had statisticians that built complex sampling models and geographers that flew around the world collecting data samples to gauge the quality of our data. This is a good example of measuring what matters. The high degree of quality allowed the company to win new customers and prevented new entrants with lower quality maps from winning business on price. Quality matters and we invested in the measurement capabilities to prove we had a high quality map.?

Measurement can be complex. Understanding the efficiency of map building was another matter. Here we tended to measure the data that was easily available, such as the quantity of changes in the map. This data was easily available in the database. What we really needed to know was the cost for each change and the quality of the change that was made. The cost of manually collected and entered data was much higher than that of? large automated data sources. The quality of large data sources varied greatly and often the sources contained a lot of information but it did not actually produce change in the map. Automated data collection vehicles collect massive amounts of data: every fire hydrant, telephone pole, and the distance from the sidewalk to the building. Most of that data was useless for a person’s navigation needs. Ultimately, from my personal perspective our field geography offices were the best source of high quality data. Unfortunately, while I was there we never produced a metric that could prove or disprove my hypothesis. We did produce a cost per change metric and it was an improvement over previous measures, but it still fell short of measuring what truly matters.?

In the end the company reduced its investment in field geographers and increased its investment in more automated and mass collection methods. I cannot say whether this was a good decision or bad decision. What I am sure of is that we lacked the measurement to know for sure before the decision was made. The measurements we had were insufficient to tell us the impact or at least there would be a lag of years before we could be sure of the impact. The point again is that there are consequences to measuring what is easy (the cost of a field geography team and the volume of changes) versus measuring what is useful (the quality of data and cost per change from each specific collection source).?

When I teach Lean and Six Sigma classes we usually do some form of cause and effect diagram. I ask the class if we need to go measure everything that is on the diagram. The standard answer is, “yes” it is a Six Sigma class after all. I then ask if measurement is free and everyone answers, “no.” Measurement requires tools, it requires people's time, it requires subject matter experts, and it often requires iterative redesign of the measurement process itself. Measurement is certainly not free. I ask the class to identify the causes that are most important to control and the factors that will be most important to monitor after the project is complete. These are the measurements that matter. This is where your investment in measurement will produce the largest return.?