Build a framework yourself: Breaking Down Every KPI

How did I learn the hard way that breaking down a primary metric into its components is as crucial as segmenting it?? And why a sum-based metric tree isn’t a best practice? Join me as we delve into the best practices to uncover the full KPI iceberg.

?? Prefer reading on Substack? https://open.substack.com/pub/rambenishay/p/understanding-user-behavior-distributions?r=1f4y9t&utm_campaign=post&utm_medium=web

Let's start with a short story. Two years post-launch, our product had accumulated numerous features, content, tasks, and goals, resulting in confusion and frustration among users. Despite frequent complaints during customer interviews, I lacked quantitative evidence. Our Support Page received many visits and was considered quite effective.? I turned to the Customer Support department, who had also observed the phenomenon and provided me with some interesting data:

Analysis revealed that 95% of users who submitted a 'General Question' had already visited the Support Page, indicating that the issue was not due to a lack of awareness of the Support Page. I then interviewed users who submitted CS tickets and found that their main issue was that the Search Bar wasn’t effective in finding the relevant article. It was simpler for users to seek help from a human representative to get the information. This claim was supported by quantitative data showing a large number of queries per page visit.

To address this issue, we worked on a new feature: Support Page V2. We overhauled the search engine, updated the content, added screenshots, and more. We chose usability, measured by monthly total queries, as the leading metric (assuming DAU was stable) and tracked the number of general inquiries to customer support as the lagging metric. After a month of AB Testing, the leading metric hardly moved. It was unclear why.

Then, surprisingly, at the end of the month, Customer Support reported a dramatic decrease in 'General' inquiries. What actually happened here?

We'll revisit this later.

Order of Operations: Multiplication Before Addition, Matric Breakdown Before Segmentation

The following mathematical steps essentially break down each metric into its components, involving all other metrics that might explain changes in the main metric:

At each step, I added a new metric (Y, Z, W) that can explain changes in the main metric X. This process creates a "metric tree." For example, if we break down Daily Revenue, we can explain why the world is divided into Monetization, Growth, and Engagement.

Another well-known way to break down Daily Revenue, especially for B2C products based on Daily Purchases, is:

These mathematical steps allow us to mix different metrics to explain changes in the main metric. For instance, in the product I worked on, we monitored our Daily Revenue almost every hour and day, along with all the above metrics that explain its changes.

In one month,? our revenue decreased without any changes in growth or engagement. Breaking down the metric tree allowed us to quickly understand the root of the problem, generate hypotheses, and test them. Possible reasons included:

1. Currency changes in certain countries lead to decreased USD revenue.
2. A special offer targeted a specific segment for one day accidentally continued throughout the month.
3. A bug in the purchase flow of more expensive IAPs (In-App Purchases).
4. Economic conditions were affecting users' price sensitivity.

By segmenting the metrics that experienced changes—by platform, country, engagement level, etc.—we could pinpoint the problem effectively.?

Funnel Analysis: A Subset of the Metric Tree

We use funnels all day, but they are essentially a way to create a 'metric tree': breaking down the problem into different areas to explain changes in the main metric and identify improvement opportunities.

Back to the Support Page:

Usability among the test group did not increase (measured by Monthly Total Queries), yet the number of Customer Support inquiries decreased significantly. What happened here?

Let's break down the metric:

• We wanted the number of queries per visit to the Support Page to decrease, indicating more effective searches and easier content discovery. Users no longer needed to run multiple queries to find the desired content.
• We would like the number of visits to the Support Page to increase, as they experienced this tool as more effective.

Thus, because we didn't breakdown the main metric into its components, we almost reached a wrong conclusion that could have led to a poor product decision. The feature succeeded in improving one ratio and reducing another, making it seem like there was no overall change. Conclusion: Break down your feature's success metrics at the beginning of the process and share the final tree with the entire team. Beyond explainability, this practice helps with alignment.

The Fourth Dimension: Time in Measurement

We often overlook the dimension of time, even though it is central to our lives. Measuring a metric in different time windows and breaking it down can teach us much about its behavior. The most famous example is

This way, DAU can be explained by time ratios, representing the product's stickiness and usage frequency.

The MECE Principle and Segmentation

It's important to remember that not all relations are multiplicative. In the case of DAU, the metric can also be presented as a sum:

In this article, I focused less on sums because it's usually more convenient and correct to view each group in the equation above as a segment and analyze the various metrics for each segment separately. Additionally, when describing a metric using sums, it is easy to violate the MECE (Mutually Exclusive, Collectively Exhaustive) principle. If you've built a tree using sum, ensure there is no overlap between segments (even in edge cases) and no remainder.

Breaking down a primary metric into its components is just as important as segmenting it. Together, they can reveal the entire iceberg.