17 types of metrics to know

tl;dr All metrics are numbers but that doesn't mean they behave all equal when analyzing them. Knowing these types of metrics helps you to understand a) am i using the right metric, b) what are potential pitfalls and c) what else could i use

What are Metrics in first place?

Metrics are one of the - if not the - most important ingredient in analytic reports. They are numbers saying something about your business or process and you can compare them to other metrics to see if things work out or not for you.

When building a report you always start with a single table. Each row contains a fact, an observation or a case, which could be business objects like orders, products and users or observations like how many orders happened on a particular day in a particular city. The columns are either dimensions or metrics describing individual attributes of the row objects. Metrics are always numbers while dimensions are mostly textual/categorial attributes or dates.

Any kind of report on that table, independently of whether it will be a bar chart, line graph, pie chart or just a table itself, would group (aggregate) the rows by certain dimensions in order to show what happened on different days or months or within different cites, etc.. The crucial thing about metrics is that if rows are grouped, e.g. analyzing orders in different cities, metrics can be aggregated using mathematical operations like sums or averages.

(Note: you may have more than one table in the beginning, but right before any report you should have joined/prepared everything in one table, the so called reporting layer)

Categorizing Metrics

There can be a lot of metrics in a business, thousands in fact. We did create metric frameworks for customers who would like to foster the creation of data driven marketing initiatives. Team members should be able to select the best goal metrics for their initiatives by drilling down through series of filters. If you look at a metric like cost per facebook video view in campaign X you can imagine that the combinations can easily exceed 1000. Having a clear understanding of the possible types or categories of metrics will help to select the right ones for your case.

Furthermore, metrics can sometimes be misleading. If you are not sure if your metric is the right one to use, it helps to determine its category and learn what applications and pitfalls come with this type.

This article exactly tries to do this. Please note that the categorization is not along a single dimension but on multiple aspects. This means that metrics can be part of different categories.

Let's start looking at the origin of the metric data

1. Measured metrics

As the name suggests these metrics are directly measured. In general you can only measure extensive metrics (see below). In some cases, if the data comes from sources outside of your control, you may not know this. In such cases you should be careful as they may be hidden calculated or even guessed metrics.

2. Calculated Metrics

Calculated metrics can't directly be measured as they are a calculated from at least two values. Through this combination of source metrics you can eventually create a large number of calculated metrics, and many of them do (unfortunately) make some sense. Most of the following metrics types are in fact also calculated metrics.

3. Predicted or modeled metrics

In some cases the metric you need can't be measured or directly calculated from measured metrics. Cohort metrics like LTV often are of this type because some revenue is still in the future. Another example would be to predict next years revenue. In these cases you will need a model which calculates your metric based on metrics you can measure and a number of parameters that you need to provide. Generally you would estimate these parameters based on historic data and/or make assumption like the revenue will increase by 10% next year. Both will not necessarily hold also for the current data and hence predicted or modeled metrics always come with uncertainty.

A special case is interpolation. For example if you have a metric only on a monthly basis but you need to compare it to other data that is on a daily basis you could calculate daily numbers from the monthly data by interpolation. A related method is extrapolation which can calculate values outside of your current range of values in your data. It's mostly used for predicting future values.

4. Guessed metrics

Of course metrics could also guessed based on assumptions. So those number haven't been measured, calculated from measured metrics nor modeled. Those are for example often found in target metrics which define a goal for the future but are more based on gut feeling and experience from other situations. Those numbers of course have to be regarded with caution including any calculated metrics based on these metrics.

5. Filtered metrics / Sub metrics

These metrics are derived from a regular metric by filtering out rows with a certain condition. For example analyze users coming from Facebook separately from LinkedIn users. Both are basically subsets of all users. One other example is users that clicked button X. Data scientist call this kind of transformation one-hop encoding. Basically you take a dimension (e.g. channel) and then create for each channel a new metric column which is 1 if the current row has this particular channel, otherwise 0.

6. Score or index

Scores are probably the weirdest type of metric from a data scientist perspective, still they are very often used in business context. Basically they are adding up unrelated metrics. Mathematically they are mostly weighted sum like s = a1*M1+a2*M2+a3*M3, although other calculation rules may exist. This way you can add for example revenues and users even though they have different units. Scores reflect that different factors (like revenue and users) may contribute to the business success, but they require assumptions about their weights (and these may change). An example could be a RFM score (recency, frequency, monetization) which is a weighted sum of the revenue of a user, the number of transactions and the time in days since the last transaction.

You should be very carful with using scores. They are sometimes intransparent as they are mixing up a lot of things. They may work as an alerting mechanism on a very high level but once you need to find out why a score moved, you need to look at the underlying metrics.

Metrics can also be categorized according to their aggregation behaviour.

7. Extensive metrics

These metrics are the most simple ones. The important property is that they add up if you aggregate them (group several rows). They consists of metrics like counts, values or revenues. Very often those can be directly measured. Note: There is always at least one regular metric in a table even if there are no numerical columns. It's the count of rows which you can explicitly create by adding a column and put a 1 in each row.

There are also pseudo extensive metrics. Basically they are extensive metrics but summing them up doesn't make sense for us. For example age: If you want to know the age of all people in a city you usually don't want to know the sum. Rather the average (or specifically the average age per user) - but that's actually a intensive metric already.

Also calculated metrics can be extensive, but only if they are additions/ subtractions of extensive metrics, e.g. revenue - refunds.

8. Intensive metrics

Intensive metrics are are kind of the opposite of extensive metrics. They are calculated by dividing two extensive metrics and therefore don't sum up if you group together several rows. Instead you would expect something like the average of all rows. Examples are revenue per user or conversion rates like users that did B divided by the users that did A or age per user which may sound a bit weird but technically average(age) is nothing else than sum(age) divided by count(user) (see our pseudo regular metrics from above)

Attention: don't put intensive metrics directly in the tables that you start your report with. The reason is the following: If you have a table containing the average age of users per day per city and you would like to generate a report showing the average age per city you would be calculating the average of the average which is not the same as the average. In fact it would overrate days with less users. You should rather have the reporting tool (BI tool) calculate them directly from the extensive metrics on the granularity (in this case city) that the report needs. Use a custom metric in the reporting tool.

Another warning is that the two source metrics that you divide should relate to the same group of objects. So for example if you want the conversion rate from all users doing a signup to users doing an upgrade you would just divide these two counts. However if you want to have this on a daily basis, the group of users doing a signup on a particular day is certainly different from the group of users doing an upgrade. Dividing these numbers is likely to be wrong. You need cohort metrics here, even though the simple quotient may be a useful as a proxy metric.

9. Unique count metrics

Some metrics count entities that occur several times in your data set. For example a user may have created several sessions on different devices and coming from different channels. So the same user may have a Facebook session on iOS and a Google Ads session also on iOS. So far the data is correct but if you would like to generate a report only split up by device then the report would return 2 users instead of one as counts are usually additive (see above for extensive metrics). So unique counts can become very misleading when further aggregations are performed. The only way to do this correctly is to add the user id into the table for each row and then use a "count distinct" as aggregation function after grouping by device. This would still return 2 rows for the one user but the count distinct function would see that both have the same user id and only count 1.

The problem with this is, that this only works if you push the user id (or other ids) to the BI tool which usually means providing raw data which may impact performance a lot.

Therefore one should try to avoid having unique counts in the reporting layers, though that often is not possible. So instead of users we could work with sessions which would still work correctly in the above example, but may show similar issues with other reports.

In future this problem may be solved by semantic layers. Those behave differently than SQL by not providing a table of a defined granularity but actually adopt to the granularity requested by the BI tool. Still this requires to run against raw data (in the data warehouse) for each generation of the report.

There are also some special aggregation types of metrics. Usually, if a reports wants to split up by a dimension, let's say city, it will group the data set by that dimension and will do some calculations on the metrics within the individual groups of each city. However there are metrics that behave differently and will need metric data from other groups as well. Those are discussed in the following. The list is for sure not exhaustive but i think i covered the most common ones.

10. Contribution metrics

They are kind of a special case of intensive metrics. Here however we especially want to divide metrics that relate to different groups, in particular where one groups is a subset of the other. Simple cases are the percentage of facebook visitors from all visitors. This can be easily achieved by using a filtered metric and dividing it by the unfiltered or using table calculation in the reporting tool like percentage of total.

However there are more complex cases. Let's assume you are looking at revenue per user as one of important metrics. And you would like to know how much each channel contributes to the revenue per user. This is especially important if this metrics changed and you want to know where it came from. If you just take a regular intensive metric (revenue/user) and group it by channel the reporting tool would just calculate sum(revenue, grouped by channel) / count(users, grouped by channel) and these metrics wouldn't sum up to the overall revenue per user. In fact there may be some channels with an extra ordinary high revenue per user but with a low volume which means that they don't contribute a lot. Indeed you would like to have something like sum(revenue, grouped by channel) / count(users, not grouped by channel). This is called a level of detail (LOD) expression and some tools like Tableau offer this. In some cases you can also use table calculations. For example percent of total is a predefined table calculation most tools offer.

11. Cohort metric, lagging metric

Cohorts are a complex topic and are eventually beyond the scope of this article. Simply speaking it's about what belongs together. The number of signups on a day and the number of upgrades share the same date but are in fact rather unrelated. As analysts we want to know what causes what and in this sense we would like to know how much our product can convince users to upgrade after signup, independent on when these two events actually happened. If you still want a temporal resolution (i.e. knowing how this conversion rates evolves over time) you need to recalculate both events to the same date type (most likely the signup date) using the users identity. An example may be the life time value LTV of a user which sums up all revenues of a particular user (which happen at different times). You can now determine the average LTV based on the signup date of the users to get a time series of this metric.

The key to create a cohort metric is in the table that you use for the report. Instead of noting the date of the event (signup, upgrade) you take the signup date for the related user for all events. This may require a bit of joining in the table preparation. You can't do this in the reporting tool if you haven't prepared it.

The problem with this type of metric is that they are lagging. Some of the events that will still contribute to users with a signup date in the past are in the future. So the value of the metric for dates in the past will still change in future. While they saturate over time it can take even several months until they are kind of useable. Unfortunately, if you look at metrics like LTV it always looks as if you business is doing bad lately. Proxy metrics or modeled metrics may help here.

12. Windowed metrics

This type of metric will also analyze "neighboring" groups to calculate the metric result. So for example if you analyze users by day, the windowed metric "running average" would look into the current day, the day before and the day after and calculate the average number of users of the 3 days. This is often used to smooth out noise data or strong seasonality (in this case set the window size to the length of the seasonality, e.g. weekly). One pitfall could be that window may go into the future where no data exist. This could be tackled by moving the window to the past 7 days instead of looking 3 days back and 3 days ahead.

However these metrics behave usually as expected. You could calculate them before in the reporting layer or in the BI tool. Most tools provide special table functions for running averages, however, be aware that also much more complex window functions exist.

Last but not least metrics can be categorized by the (business) purpose of the metric. Again this list is not exhaustive, but the most common metric should be present. Also, metrics could potentially be part of multiple categories.

13. Key performance indicators (KPIs)

As said there are easily thousands of metrics in a business and it totally makes no sense to look at all of them. In fact a business should define a small set of metrics that are most efficient for assessing performance and steering decisions. These are the KPI or Goal metrics.

Note, KPI exist on different levels. The C-level needs different KPI than the business units and a channel manager again different ones. The most important KPI for your business or business unit is often called North star metric.

Another special case are currency KPI or central KPI. They are important in larger organizations with lot's of business units having their own KPI. Unfortunately, they often have also their own definitions of what actually should be the same KPI. One department is measuring revenue from Google Analytics data, another one from shop data directly and a 3rd is even subtracting returns. If they talk to each other than every one has a different value for the same thing and a single source of truth is missing. Here it is important to identify those currency metrics that are important to more than one business unit and define them centrally. This could be done for example in data contracts within a data mesh concept.

14. Health metric

a health metric is a metric that isn't as important as a KPI but still should be monitored to make sure everything is ok. This is often used when something is changed in the business model, in particular during an a/b test. For example if you are testing if a new layout is driving revenue you may want monitor the customer satisfaction score to make sure customers don't feel overreached by the change. Note, a metric can be a health metric in on test / change and a KPI in another one.

15. Vanity metric

A related metric is the vanity metric, which is a KPI which shouldn't be one. Vanity in this case means that it may look good (e.g. thousands of newly signed up users) but they were all promised free beer and will most likely not contribute to the business' success in future.

16. Target metric

Target metrics are target for other metrics. Ideally the other metric will reach, pass or not pass the target metric. These metrics are generally not measured but either modeled/predicted or guessed. When using them in reports make it clear they are targets.

17. Proxy metric

If you can't measure or directly calculate the metric that you really need, you could use a proxy metric. In comparison to the modeled metric, the proxy metric is not trying to mimic the desired metric but instead is an actually measure or calculated metric that you think is a good proxy for your desired metric. So for example to measure the performance of workers it may be convenient to measure worked hours instead of work output as the latter is difficult to determine or at least difficult to compare when workers are assigned varying tasks. You would assume that there maybe a factor between worked hours and work output which is potentially different for different workers. Instead of trying to determine this factor(s) (e.g. from historic data as you would do for modeled metrics) you would just use the proxy metric directly for for example determining salary. Anyhow, both, proxy metrics and modeled metrics will show uncertainty.

Another example would be to proxy LTV by dividing all revenue from a day by the number of new users. Again, this relates to different groups of users as explained above but if the distributions of these groups do not drastically change over time this may still be a good approximation. And it has the big advantage of not being a lagging metric so you could see up and downs faster (by compromising accuracy)

Takeaway

Understanding properties and pitfalls of metrics is essential when using them in analyses and reports. Differences exist with respect to how to generate the underlying data, how to deal with them when grouping and aggregating or how to interpret them in business context. If you have any questions or comments let me know.