Pyramid of Metrics

Magical guide about #product management. Part 7, article #47

What are metrics?

A metric is a qualitative or quantitative indicator that reflects a particular characteristic and the level of success of a product.

To create an application or service, one idea is not enough. It is necessary to measure and analyze indicators, investigate user reactions and refine the product. That’s what metrics were invented for.

Quantitative indicators are easier to track, so they are used more often. Based on these numbers, it is possible to draw conclusions about what is happening at all: whether users need such a product, how much they like it, whether it solves their problem. For example, you have created an application that informs motorists about the situation on the roads. It seems that the idea is good, the product is necessary, but people still do not want to use it. How to understand what the problem is? Is there something wrong with the idea, or maybe you made a mistake with the audience or promotion?

Metrics help answer these and other questions. The main thing is to know which ones, how and when to use. It is not always easy to understand this, because you can find several answers to any question, and each indicator obtained can be understood in different ways.

To make it easier for you to navigate, we will explain the basic concepts and terms, show how metrics are chosen and used in companies, and tell you where to run if you want to understand this topic more deeply.

What metrics are there and what they depend on

There are many metrics, they can be divided into different categories. For example, there are product, marketing and business metrics.

And you can single out thematic ones — according to the goals that they help to achieve. Below are some examples of such metrics.

Metrics of user engagement

They help to understand how and from where users come to the product, how many of them there are, which of them are active.

Examples of metrics:

• monthly active users — the number of active users per month.
• Active Users — the number of active users of the product.
• RR, retention rate — user retention rate.
• CR, churn rate — the churn rate of users.

Product usage metrics

This includes all indicators that demonstrate how people use the product, what scenarios there are.

Examples of metrics:

• ER, engagement rate — engagement rate.
• Sessions per user — number of sessions — how often customers use the product.
• Key user actions per session — the number of key actions per session — how often a specific action that is important to you is performed.

Monetization metrics

These are indicators that make it clear how much you earn on the product.

Examples of metrics:

• CAC, customer acquisition cost — the cost of attracting a customer — how much money on average you need to spend on advertising and promotion to attract one user.
• GMV, gross merchandise volume — the gross volume of trading operations — how much you earned on the sale of the product.
• LTV, lifetime value — how much money the user brings in during the time that he uses the product, or for a certain selected period.
• ARPU is the average profit per user for a certain period of time.
• ARPPU is the average profit from one paying user for a certain period.

Any metrics are just numbers that by themselves do not provide important information. In order to benefit from them, all indicators need to be looked at in dynamics. That is, it is necessary to choose a certain period and analyze how the metric has changed during this time, what happened to other indicators and what may be the reason.

Metrics should depend on business goals. Before choosing which indicators to measure, think about what you want to achieve.

For example: to make a profit, attract investment, achieve user loyalty. In all these cases, you will need different metrics.

It is important not only which metrics to choose, but also how to use the obtained indicators. There are two main approaches that are used in product development:

There are two main approaches that are used in product development:

Data-driven approach

First they get the numbers, and then they make decisions based on them. The team selects metrics and counts metrics. The resulting numbers are the first thing they will look at when deciding where to move on.

Data-informed approach

Metrics only partially influence decision-making. Indicators are important, but not the main thing. You can focus on them in one case and ignore them in the other.

North star metric

What is North Star Metric

The North Star metric (NSM or Polar Star Metric) is the metric that best reflects the core value of a product to users. A well-chosen Polar Star metric can provide a company with stable and sustainable growth in the long term. In other words, NSM is a key success indicator for the product team.

The NSM indicator should include three main parameters:

• profitability — the indicator shows how much the company earns;
• value for users — the indicator reflects the main value of the product for customers;
• measurability — the indicator is easily measurable.

Let’s say your main indicator is “registered users”: the more users have registered in the service, the better. This indicator is easy to measure and it shows how much the company earns. But in fact, this metric does not carry any value for customers. It does not show how often already registered users log into the service and whether they like your product. Such an indicator does not allow us to assess whether the release of a new feature, user onboarding, updated design, etc. was effective, despite the fact that it is measurable and reflects the profitability of the company.

By directing all your efforts to the growth of the correct NSM metric, you will be able to provide the company with revenue growth, you will understand what value users see in your product and correctly measure their actions. With such a metric, you can become a company that knows in advance what it needs to do for improvements, and most importantly, how these improvements will affect further growth.

Why do I need the NorthStar metric?

Polar Star Metric:

• It helps the product team to clearly see what needs to be done to improve NSM performance, and which features can be transferred to the Tasks backlog or canceled altogether;
• Allows you to see the progress of the whole company in numbers and accelerate the implementation of product initiatives;
• Gives an opportunity to see concrete results on the work done

What North Star metrics can different companies have

E-commerce

• The number of users who made the first order per week;
• the cost of daily purchases;
• LTV (Lifetime Customer Value)

Online services

• Number of trial accounts with three or more users in the first week;
• retention percentage after a year;
• MRR (regular monthly revenue);
• MAU (monthly active users).

Media

- Registration and retention;

- number of active users per day;

- total duration of readings;

- the total number of views.

Can the North Star metric change over time

Yes, but remember that the metric is calculated for a long-term period. This is the essence of the strategy. If you often change the main indicator, most likely you choose the wrong metric and cannot achieve tangible results.

A live example of polar metrics in a product:

Hierarchy of Metrics

Who invented it? It’s pointless to argue. A child about 9–14 months old decides that walking straight is cool, and goes. The digital product manager, after about the same time working with analytics, understands that it is necessary to put the metrics in order and does it. This is how its Hierarchy of Metrics appears. It is found in almost all products where the older age of the dashboard is more than a year.

There is a hierarchy of Metrics in almost all products where the older age of the dashboard is more than a year.

What for? The hierarchy of Metrics allows you to

1. Localize the causes of changes in the graphs of important metrics (“investigate anomalies”)
2. Do not roll out changes that can harm the entire company (rule: top-level metrics should not sink, even if the target release metric has grown).
3. Focus on important indicators
4. Identify growth points
5. Prioritize functions

There are many indicators that can and should be measured, there is also a hierarchy in the approach to measurement, and, as in any hierarchical structure, if you do not set up the base, you cannot hope to reach heights.

When you have an idea for a feature, it is useful to “embed” it in the metrics tree. This will help you understand which metric this feature will affect the most. The closer the metric is to the “top” of the metric tree (to the key metric of your product), the more likely it is that the function will be successful.

Algorithm of operation:

It may be useful to visualize the algorithm using MindMap (for example, MindMeister, Mind42, Miro) or on a sheet/flipchart.

So, the algorithm:

1. Determine the main indicator for your company (for example, “Revenue”).

2. Determine which key metric of the product corresponds to the main metric of your company (let’s assume that it is also “Revenue”).

3. Determine which 3–5 main metrics most affect the key metric of the product (for example, it will be “Average purchase price”, “Frequency of purchases”, “Number of users” and “% of paid users”) — these will be the metrics of the first level.

4. For each metric of the first level, ask the question: “Which sub-metric indicators affect this metric the most?” (for example, the “Average purchase price” is influenced by the “Average cost of goods” and “Average number of goods in a particular order”) — these will be the metrics of the second level.

5. Now we take the metrics of the second level and move on to the metrics of the third level and so on — in general, it is reasonable to dig up to 4–5 levels (depending on the size and complexity of your product) — keep digging until you reach a level that you consider reasonable (and while there is an opportunity to influence the metrics of this level).

As a result, you should get something similar to the hierarchy as in the image below.

What to do with all this next?

Take your backlog and decompose the features from it by metrics — the closer the metric is to the top of the hierarchy, the more likely it is that the function will be successful.

Analyze the metrics of the second and third levels, focus on them and brainstorm. Determine which new features can boost these metrics.

Common mistakes

1. Setting goals: make sure that the metric that you have chosen for the top of the hierarchy is really the most important (hint — check with the company management + you can take the metric “Revenue from sales of products” by default). Some metrics reflect only a part of the company’s indicators, omitting other important elements. Do you really want the “Number of orders” to increase the most, and not “Total revenue” or “The ratio of revenue per user to the cost of attracting”?

2. Metrics should quantify what you really need to know. Make sure that each element of the hierarchy is a measurable metric.

3. Focus on metrics, not features: when you correlate feature ideas and metrics, it’s worth writing feature ideas next to (or in a comment) the name of the metric they affect, but not as a separate element of the hierarchy.

4. Make sure that the number of “vanilla metrics” in your hierarchy does not exceed all the others — otherwise you risk chasing them and subsequently ignoring the key metrics of the product.

* Vanilla metrics are any non-objective indicators, they cannot be used to judge the quality of the result and, in general, the presence of the result as such.

5. It is not recommended to directly associate a submetrica with several “parent” metrics — otherwise it will be difficult for you to make decisions. In extreme cases, designate one of the links as the main one, and mark the rest with a dotted line.

Pyramid of metrics

This, like the Hierarchy of Metrics, is a tool for analyzing product indicators. In order to build a Pyramid correctly, it is necessary not only to determine the dependencies between metrics, but also to classify them by goals.

In companies, the Pyramid and the Hierarchy of Metrics are used in different ways: together or separately from each other. Sometimes they are even defined as one model.

When the Metrics Pyramid is most likely to help you

1. You have an operational or design-oriented digital product that is growing towards the grocery business. For example, you have a delivery service. Everything is on the operating system. You are at the stage of transition to the grocery business, you are starting to consider the unit economy, you have realized the importance of CustDev, you are studying or already implementing Growth processes.
2. You have a team that is disconnected from the business and its goals. You want to realize your place in the company’s processes, integrate with other teams and businesses, and ensure manageable growth. For example, you have a small B2B department inside a large B2C company. No one understands what you do. And, frankly, you are already confused too: what they want from you and how to measure their effectiveness.
3. You have a lot of metrics, dashboards, a rich backlog and a misunderstanding of how to cook it all.

When the Metrics Pyramid most likely won’t help you

1. You have an operational or design-oriented product that is not going to change significantly in the near future.
2. You have an understandable product, a small team, and a small and understandable set of metrics that you focus on.
3. You have a clear connection between the product and the business.
4. You don’t have an analytics crisis in the company: decisions are made based on evidence.
5. You have a crisis of analytics in the company, but you understand that it is easier to change the company.

The hierarchy and Pyramid?of metrics are models that help to organize indicators, determine the dependencies between them and better monitor changes.

The hierarchy of metrics describes their dependence on each other in the direction from high-level KPIs to low-level indicators.

Hierarchy levels can be described as deeply as necessary, down to specific indicators that we measure in products.

It is not necessary to rely on strict formulas, some dependencies known to us can be expressed without formulas, for example, Retention in Skyeng as a whole depends on Retention in individual products of the company.

Thus, classification helps to remove the hyperfocus of the team on one of the many processes. During the audit of metrics and dashboards, we can find out that the team thinks only about changing microeconomic indicators or only about the interface, and also helps to outline areas of responsibility for the growth of indicators, competently select KPIs and assemble dashboards for each team in the company. And now let’s study this topic in more detail.

How to simplify working with metrics: frameworks for research

When working with metrics, as we have already said, many questions arise. What and when to choose, how to measure, how to analyze? And how do you understand that metrics are suitable for the product, and you will not count and compare indicators that do not give any information — or give, but not at all the one that is important to know for the development of the product?

To simplify working with metrics, large companies, for example, Google, use frameworks for conducting research. These are ready-made methods that help to measure and analyze product indicators. Here are some of them.

HEART

The HEART framework (Happiness, Engagement, Adoption, Retention and Task Success) was created at Google and is used most often in the digital sphere. Allows you to track the user’s experience (the designation UX, User Experience is often used for it) by certain categories: happiness, engagement, acceptance, retention, success (completing tasks).

It does not include specific metrics, so they will have to be thought through individually. This is no coincidence, because different indicators are important for different products.

AAARRR funnel

AAARRR is a marketing funnel of the main stages of customer interaction with the product. It is needed to divide the work with the user into stages and track the indicators on each of them. There can be six such stages, if you start with Awareness, informing, or five, if with Acquisition, attracting (depending on this, the abbreviation AAARRR or AARRR is used).

The marketing funnel helps to evaluate the success of customer acquisition and monetization.

• Awareness. Acquaintance of the client with the product: for example, he came to the site.
• Acquisition.?The client became interested, and he left his contacts.
• Activation.?The client understood the value of the product and began to use it.
• Retention.?The client constantly uses the product.
• Referral — Virality. The customer shares information about the product.
• Revenue — Profitability. The customer pays for the product.

Conclusion

Creating a hierarchy of metrics solves two main tasks. Firstly, it helps to determine the metric of the consumer value of the product (NorthStar Metric) and link it to revenue. Secondly, it helps to decompose the upper-level metrics into their component interrelated parts. This helps to make business and product decisions more efficiently

Thank you for your attention???