What is the right measurement?

Improvement practitioners often find themselves pushing people, organizations, and companies to quantify their operational performance beyond just financial measures. We grouse that companies don’t invest enough in measurement. While this is a problem it can also be the case that a company has too many measures. I recently visited a company where they showed us 23 different measures that they use to manage the business. After leaving the company I commented about how much good data they had, but my colleague corrected me stating, “How can they possibly create any focus with so many measures to choose from?” So how much is just enough? It is quite a conundrum.?

Like everything we do in continuous improvement, when we set out to create measures we first have to ask, “what is important?” or “what problem are we trying to solve?” Measures serve many different purposes. They can align a team, they can tell us if the product we are shipping will meet the customers expectations, and they can expose inefficiency in the organization. The purpose can change depending on the situation. I ran many projects where the measurement was used to understand the problem, root cause the issue, and verify the solution. At the point of verifying the solution we stopped measuring. You might say we should never stop measuring, but remember measurement is not free. There needs to be an ongoing value to the measurement process. Alternatively, I worked with organizations to build or improve on measures that became the focus of the organization for years. You need to remember that if you do not measure it you cannot manage it.?

When it comes to establishing measures, I organize my thoughts around the scope of what we wish to accomplish:

1. A metric to align the organization
2. A metric to monitor ongoing quality or efficiency performance
3. A metric to resolve a current state issue

A metric to align the organization

Metrics to align the organization should be based on the core purpose of the organization. These will often have a financial component since they are the result of aggregate activities. In manufacturing we may be looking at contribution margin per employee. In a distribution center we may be looking at on time delivery or on time delivered in full metrics. These measures should have good source data from the underlying enterprise data systems. The data sources need to be validated by the accounting and IT organizations to ensure good data quality. The data should be reliably produced in reports on a monthly or weekly basis. We need to be able to connect these measures back to the financial performance of the organization. For example, if on time delivery is good, sales people can drive more business, and generate greater revenue. This should be the starting point for all measurement selection. I like to use Critical to Quality Trees or Critical to Process Trees to show the connections.?

When considering metrics to align an organization, less is more. Twenty-three metrics at this level only confuse the organization. Which one is most important? In the course of business we will often suboptimize one measure over another; the organization must be aligned to avoid conflict when this happens. This was discussed in Jim Collin’s book “Good to Great” which is a must read on this topic.?

A metric to monitor ongoing quality or efficiency performance

Looking at a metric once a month or once a week is not enough. We want to know that there is a problem before a customer calls with a complaint. Management must have hourly and daily validations of both the quality of output and the efficiency of the operations. These metrics will take the form of defect counts, output counts per hour, changeover times, down times, call wait times… the number of metrics at this level can be endless. This is where a company may have 20, 50, 100 different metrics. Regardless of an organization's size, when we dive down to a team it should be clear that there are 3 to 5 metrics that are important to monitor. I often find that the data is less reliable at this level. This is due to the fact that there are more manual collection activities and less data validation activities. An operator may be logging numbers in a log book about specific down time or defects. These logs may have inaccuracies, or worse, a lack of attention can often omit large chunks of data. Measurement System Analysis, as previously discussed, is very important to understand the quality of the data. At this level though, I am okay with losing a little on the quality to reduce cycle time of data gathering. I don’t want to wait for a validated report in order to recognize a management problem. I want this information in real time, so that we can react. As modern MES, WMS, and service systems become more advanced we do not have to make that trade off, but those are investments in system infrastructure that many companies are not ready for just yet.

A metric to resolve some current state issues

Most of my career has been focused on leading initiatives. Every initiative should start with a problem statement and objective that can be quantified. Early in my career I struggled with the reality that the data I wanted was often not readily available. This is where it becomes important to have the skills to create measurement processes. It is likely that we will be asking someone to track something that they have not tracked in the past. How much effort will this take? Will they need a specific measurement device? Will they need to be trained on how to use it? These are important questions to ensure a repeatable and reproducible measurement. Additionally, how will the data be used once it is collected? Most likely we will be digging down further, so we may need to collect metadata along with core measurements. This will allow us to make comparisons, look for trends, and performance correlation analysis.?

Finally, it is important to think about how to construct a measure that exposes the problem. The executives may be okay looking at a simple proportion of shipments that were on time against a total number, but that does not really tell the full picture of the problem. This is a discrete measure and we can learn much more from a continuous measure such as the number of days late or the shipment variance: days early or days late averaged and trended over time. It is likely that we will be collecting only samples of data over a short period of time. We have to think about how many samples we need and how much error is in this new measurement process we created. Again, we need to perform the measurement system analysis.?

Getting the measurements right is tough. We do not want to confuse the organization with lots of unnecessary information, but we do need to quantify our performance, our problems, and manage based on data. Getting this right is an iterative learning process. We have to think about it differently depending on what we wish to accomplish. The key is to try, test, and change and don’t give up!