Dashboard Metrics Don't Tell The Whole Story

We’ve all had the experience of looking at a chart or a component on a dashboard or slide and thinking:?so what??

It’s hard to tell what one number means on a chart unless you understand the metric and definition, you’ve been following that number since the last time it was updated, and you know when it will be updated next.

This points to a key problem with metrics in general and dashboards in specific: they don’t help you get context on the next option and help you know what to do.

In this case, we see that there is a label and a number. If you don’t know what “Quality Leads” are and what this metric has been in the past, you’re in the dark when trying to evaluate this number.

We are not demonstrating how this metric fits into the overall lead process.

Poorly Labeled Metrics are lying to you

These badly constructed metrics are not truthful for a few reasons:

• The time horizon is unclear -?you see a current number and it’s not clear what time period this covers
• What’s a good value?- you don’t know the history or variability of this number
• Lineage is missing?- it’s not clear how to relate one metric to another

Setting a time horizon is not difficult – you state the period the metric covers – and you typically do this by suggesting both the time horizon and the cadence in the name or description of a metric. You’ve probably seen a version of this metric renamed “Daily Quality Leads” or “Weekly Quality Leads.”

After you set the time expectation, it’s time to make it clearer what this metric really delivers. I define a metric as “Something that changes from x to y by when.” We also need to know the definition of the underlying data to make it available in a description or other informational area of the metric.

To sum up, we need to know:

• the definition, e.g. “the leads that were marked ‘Quality’, because a process changed their status or high product usage was seen among those leads/trialers”
• the time frame, e.g. “last 7 days” - if you don’t state it, you’d guess that this is a continuous number or a discrete number where every week this is updated. Make it clear when to expect the next update.
• the headline number, e.g. the count

Knowing the definition leads to the next question.

What’s a good number?

Now that you have an idea of what you’re looking at, there are two things that will elevate a metric into a valuable observation:

• Knowledge of the past range and variability of this metric
• Have an idea of what to do about it

When you consider a number, how will you know if it’s good or bad? If it’s a new metric or a sparsely populated range it may be hard to compare the current value with a previous value. If you don’t look at the median or the average value over a number of measurements, it’s difficult to see any seasonality or fluctuation.

Look at the number. Now that you know whether it’s good or bad, how do you understand its impact in context? “Good” or “bad” could have been set in an arbitrary way instead of being the input for some other part of the process.

If a metric feeds another metric (or is sourced by another metric), you’d want to be able to zoom out (or in) to see the bigger picture.

What would a better metric do?

Here’s one example of what better metrics might do to display more context, uncover the source of data, and suggest the reliability of the metrics.

Let’s take a look.

The first difference is to raise the clarity of the time period, the population, and the change over time. By adding an indicator of change since the last period, you can get a sense of the directionality of the metric.

A better metric might also give you a sense of the population size, label the time period, and provide a trending indicator for that metric over time.

A better metric would also define the metric and deliver links to other related definitions and metrics. By showing us the lineage of this data –?where are we in the chain of connected metrics that make a funnel or a process –?a better metric shows us how this metric fits in the overall whole.

If you squint, you can also imagine the ability to jump to a list of records shown or visualized by this metric, along with suggestions for what to do next to change this metric.

Metrics as a generalized pattern for work

The above example uses one process to show how metrics connect in a business. But what if you use the same logic to think about more generalized patterns? Then, you’d create links between different metrics owned by different groups to measure (and perhaps start to predict, if you got enough measurements) how things are going to go.

This sounds like an interesting idea to pursue, and hard to link the performance of metrics as this is a multivariate problem. One idea I’ve thought about is to link the individual metrics and the processes that reference them into a graph where you can explore outcomes by finding the nearness of a metric to a positive outcome. At worst, this might help people to understand how their work is related to metrics owned by other people or departments. At best, it might start highlighting the most important metrics for the business and trending them.

What’s the takeaway??Making metrics better by adding labels and definitions will help the whole team. You can use this knowledge to better establish the relatedness between metrics and have more context to explain when things go well or aren’t going as well as you like.

(this was originally published on Nov 7, 2022 at Data Operations , where I write topics like this weekly.)