Understanding User Behavior Distributions

Why do most companies focus primarily on their High-Value Customers (HVCs)? Why should you always include high-risk/reward features in your roadmap? At Zynga, why don’t we necessarily prioritize ARPU? Why do venture capitalists focus more on product metrics than valuations?

?? Prefer reading on Substack? https://open.substack.com/pub/rambenishay/p/understanding-user-behavior-distributions?r=1f4y9t&utm_campaign=post&utm_medium=web

One of the core responsibilities of a product manager is to analyze various types of data—user data, sales data, log data, economic data, and more. Understanding data distributions, including their shapes and trends, is crucial for deriving meaningful insights and making informed decisions.

The Normal Distribution

I was utterly fascinated when first introduced to the Normal Distribution. Most data points tend to cluster around the center, with fewer at the extremes. This pattern appears in various scenarios: analyzing heights and weights, IQ scores, customer satisfaction, A/B test results, and daily revenues of products.

The Normal Distribution is particularly significant in the Diffusion of Innovations model proposed by Everett Rogers. This model reshapes our understanding of products and marketing. It's invaluable for grasping adoption trends and channel performance. The S-curve, representing the area beneath the normal distribution, shows that most products, growth channels, and market segments eventually reach a saturation point. These two graphs shaped the way product people think about market entry and growth.

But… The Power Law

The Power Law is a prevalent phenomenon. Consider a room of 100 diverse people: their ages might show a normal distribution, but their wealth will likely follow a power law distribution. These distributions peak around zero with a very long tail to the right.

If we take a sample of a thousand random musicians and calculate their average physical height, we will probably get a number that is about one meter seventy, closely aligning with the average height of all nine million creators who have ever uploaded a song to Spotify. However, measuring the average streams these musicians receive on Spotify presents a different scenario. Adding megastars like Taylor Swift or Ed Sheeran to this group could cause the average streams to surge by a thousandfold! Yet, no musician—or even a basketball player—could join and significantly alter the average height; you would need many decimal places to notice any change at all. The distribution of streams on Spotify follows a power law, where a small number of tracks gather the vast majority of streams. This pattern of musician streams on Spotify exemplifies the power law dynamics pervasive in many aspects of our lives.

Individual skills of your employees might follow a normal distribution, whereas team skills as a cohesive unit often align with the Power Law. Studies on team performance emphasize the influence of non-normal distributions, social skills, and collective intelligence on team success [Research into team performance and its influences include studies on nonnormal distributions (Bradley & Aguinis, 2022), the impact of social skills (Weidmann & Deming, 2020), and the effects of social sensitivity on collective intelligence (Meslec, Aggarwal, & Curseu, 2016)].

City sizes also demonstrate the power law: the largest city is typically about twice the size of the second largest, and so on, as seen in the US with cities like NYC, LA, and Chicago.

All these examples are specific cases of a more general law called The Power Law (also known as Zipf’s law), which suggests that many ranked phenomena in life will follow this hyperbolic function:

In cities, b = 2,? meaning the largest city is roughly double the size of the second-largest and the second double that of the third, although other datasets may follow different patterns, such as the first city being three times the size of the second.

The more common form of the power law is the 80/20 rule, as observed by Vilfredo Pareto when he noted that 20% of the population owned 80% of Italy's wealth. This principle reveals that a small fraction of elements frequently accounts for a large portion of the outcomes, though the exact ratio is not necessarily 80/20.

Power Law in Venture Capital

VCs operate on a high-failure-rate model thanks to the power law. Returns in VC do not follow a normal distribution. A tiny number of firms and deals generate the majority of profits in the industry. That’s why investors don’t care much about valuations. Even if the price seems high, finding those winning companies is the goal.

User Behaviour

Power law distribution often applies to user behavior and engagement. A small percentage of power users typically account for a disproportionate amount of product usage or revenue.

Examples:

• If you rank your users by their total spend, you will probably reveal power law distribution.? data disclosed by Apple during the Epic Games lawsuit indicating that a small fraction of users—just 0.5%—accounted for more than half of the App Store's revenue.
• On marketplaces like Amazon, eBay, and Etsy, a small percentage of top sellers generate the majority of sales.
• Engagement, ranked by user activity, also follows a power law. For example, only a few titles on Netflix drive most of the product usage.

Applications for Product Managers

Segmentation and VIPs

Product managers should understand the distribution of their users' engagement and spending to inform decisions. Typically, when users are ranked by total spending or engagement, a power law distribution emerges. In free-to-play games, managing Whales economy is our bread and butter. We put a lot of focus on a handful of extremely profitable users (aka Whales). We have hundreds of price and difficulty segmentations. By slicing our big power distribution into smaller, more proportional groups, we can offer more value to our users.

Focusing on VIPs/HVCs is OKAY. The majority of engagement and revenue come from them, offering long-term business stability. These power users provide insights for product development and opportunities for upselling and cross-selling. While you shouldn't focus solely on VIPs, focusing on them at some level can offer substantial value.

Enhancing Core Features: The Key to Sustainable Growth

The thrill of launching new features is hard to resist, but my experience reveals that the true catalysts for growth often lie in the less glamorous work of refining the core product. Typically, the first features built in the early stages of a product’s life are the most critical. As your product evolves, many new features on the roadmap tend to cater to smaller user segments, like power users, or address secondary use cases. This means that optimizing existing essential features can drive more significant business impact than developing new ones that haven’t yet been deemed a priority.

Consider the power law distribution: when plotting feature importance, it’s evident that a few key features have the most substantial impact. By focusing on enhancing these high-impact features, you can achieve greater value and user engagement compared to the continuous pursuit of new feature development. Ultimately, a strategic emphasis on improving core functionalities, rather than constantly adding new ones, can lead to more sustainable and meaningful growth for your product.

**Although harder to plot, even the features that add the highest increment to the probability of sales success usually follow this same pattern. Despite the Sales department often insisting that developing new, tangential features is the most critical, the real picture might suggest that refining the core, high-impact features yields better results.

Metrics

Stop relying solely on averages for your metrics. If the distribution of your data isn't normal, you should focus on the distribution itself and consider alternative metrics like the median, P.90, or geometric average.

For example, let’s say we AB test a feature and see a 10% increase in engagement. In a normal distribution, this would be fantastic news. However, in a Power Law distribution, the average isn't as meaningful unless we segment our audience properly. As shown in the graph on the right, the control group was actually the winning variant for most users. By only looking at the average, we might have ended up rolling out a suboptimal feature to a specific segment.

Power Law in your roadmap

In product development, a small number of features often drive the majority of lifts. Product managers should prioritize these high-impact features that deliver the most value. At Zynga, we call these Bold Beats—features with the potential to move the needle (‘Home Run’). We aim to ship at least one such feature every quarter. We understand that most will fail, but due to the power law behavior of feature success, we prefer taking higher-risk bets that could end up at the top of the power law distribution, much like the VC model. Allocate resources disproportionately to the most promising opportunities.

To Summarize

Power laws are everywhere. Your users, features, and teams follow power laws. Shift your mindset from normal distribution to power law distribution. Tune, create, prioritize, segment, and analyze features as they follow power laws.