Value vs Vanity – better metrics drive better outcomes

A product team’s highest purpose is to achieve product-market-fit by maximizing customer value. Pursuing this purpose requires product teams to measure value accurately. Product creation is usually based initially on intuition-driven hypothesis. This hypothesis of what will drive value for the customer must be quickly validated with product data to guide future enhancements. Not all product data is equal. Some product data can be used to form value metrics, while other product data is only suitable for operational purposes and if used to determine ‘value’ they turn into vanity metrics2.

Value Vs Vanity Metrics – What’s All the Fuss About?

Value

Value metrics are measures of what is ‘of value’ to a stakeholder of the product. The most important value metrics tell what is of value to the customer since products that do not provide value to the customer will not generate value for the business.

An example of this kind of value metric would be the time it takes to pay for parking. It is valuable to the customer to pay for parking, so they don’t get fines or have their car towed, however, this should take as little time as possible. Therefore, in a parking payment solution, the time needed to complete the transaction is a clear value metric. A value metric is a metric that if optimized, brings additional value to the customer or the business.

Vanity

The opposite of a value metric is a vanity metric. Vanity metrics are very dangerous, since they appear to tell product teams about something of value to the customer without in fact doing so. Vanity metrics can make product teams think that the product is well liked when that is not necessarily true. Optimizing for vanity metrics hurts a product, whereas optimizing for value metrics will help a product. In the parking example, it makes sense to measure the number of transactions per day as an operational metric. This can tell how many customers were served by the system and is useful to know for operational purposes. If this is treated as a value metric and the parking system is tuned to maximize the number of transactions, customers will likely be forced to pay frequently for small increments of time and therefore become very unhappy. In this instance transactions per day is a vanity metric, that is a metric that looks good, but has no value to the customer and therefore no value to the business.

Choosing Value Metrics

To make effective product decisions (stop-go-pivot decisions) product teams must identify the crucial value metrics of the product and measure every new feature relentlessly against these value metrics. Product teams must also avoid vanity metrics if they are to generate true, validated learning about what customers want and need.

There is no one-size-fits-all. Each organization and each product team must identify its own value metrics, based on what aspects of a product are truly indicative of product success. Value metrics must be able to measure value to the customer and to the organization. Measuring value to the customer first is crucial since typically value to the customer converts to value to the organization, but not the other way around. While product teams commonly believe they know what the value metrics of their products are, often when pressed to identify their value metrics they cannot.

Example.com

To clarify what is a value metric, look at example.com, a comparison shopping website with value metrics including clicks per session, conversion and NPS. Example.com also tracks the vanity metrics of page views per session and user sessions. These metrics and their usage are explored below. To understand the metrics, it is important to understand the three actors: the business: example.com and the two customers: merchants and users

Example.com presents multiple merchant product offers to users and derives value (revenue) when a user clicks on a merchant product offers.

Merchants are the source of revenue for example.com by agreeing to pay per click on each of their product offers. Merchants derive value from users who buy something from them.

Users come to example.com to be able to find the products they are looking for and compare prices for similar products between merchants. Users derive value by finding products they want at prices they are willing to pay.

To evaluate features and enhancements, the product team needs value metrics that can tell the impact changes are having on each of these users. The product team also needs to avoid vanity metrics that look compelling, but do not tell them anything about the impact on any of the three actors.

Understanding these metrics, clearly any feature that can improve CPS, Conversion and NPS is one that must be preserved and potentially enhanced. Example.com is a high-maturity agile organization, so they make small, incremental changes to the product, measuring after each change to ensure that the change either has a positive or at least neutral effect on their value metrics. Any changes that have a negative impact on these metrics are likely to be rolled back. At the same time, vanity metrics are monitored to provide operational intelligence, but never as an indicator of value. They are labeled as vanity metrics to ensure they are not used inappropriately in product decisions.

Now you try

Organizations wishing to apply this model to their own product development need to start by identifying the actors, then the product-relevant value metrics that are important to each actor. Next build the means to take the measure of these metrics regularly and easily. In the case of example.com, this is done through extensive logging and highly efficient business intelligence systems. However, with a bit of effort and careful consideration, all organizations can find a way to measure the value they are providing to customers and receiving in exchange.

</end>

Metric Notes

?Clicks per Session (CPS) measures how many times a user found an offer compelling enough to click on the offer, therefore going to the merchant’s website and generating a small amount of revenue for example.com. The more clicks, the more revenue example.com has earned. This is certainly a measure of value to the example.com. It may or may not be a measure of value to the merchant and could be inversely valuable to users since they likely benefit from finding the perfect product in as few clicks as possible.

Conversion is the metric of how many visitors to a merchant’s site convert to paying customers. If none of the shoppers that example.com sends to a merchant buy something from the merchant, then example.com is not adding value for the merchant. Conversely, if every shopper buys something, then the small amount paid to example.com for the leads is well worth it. This is a true measure of value to the merchant and therefore the value to example.com and the shopper as well, since the shopper will only be buying something if it matches what they were looking for.

NPS – Net Promoter Score became a commonly used metric in the late 00’s and has remained a popular indicator of the value sentiment of the customer. The NPS consists of a simple question: “How likely are you to recommend (company | product | service) to a friend or relative?” While a good indicator, since NPS is a self-reported value and not a result of actions the user has taken, it can be affected by demand characteristics and self-selection bias. Better indicators include an opportunity for the user to refer someone to the site, since this is an action taken.

Page views per session are a vanity metric. They are indicator of how much time the user spent interacting with the site. The higher the number of page views, the more time the user spent interacting with the site. Always be mindful of unintended consequences. If the teams make a change that requires a user to go through a few more pages to get to each offer, this metric will go up without creating additional value for the company or users, which is why it is a vanity metric.

User Sessions are a vanity metric. The absolute number of unique user sessions in a day is an important indicator for the overall health of the business. User sessions are created every time a user enters the site and are kept alive for 30 minutes past the last interaction with the site. This means that the number of user sessions is more indicative of the quality of the SEM and SEO effort, plus potentially an indicator of brand affinity rather than the impact of an enhancement or new feature on the site. Changes in the site that would increase user sessions, e.g. by changing the 30 minute to a 15 minute threshold, do not deliver any specific value to any of the three actors.