Product Goals and Metrics: Part 2

Defining key customer outcomes and success metrics for the product

In Part 1 of this series, we talked about defining a vision for the product and developing a strong understanding of the product and the business. In Part 2, we will talk about defining clear metrics for the success of your product so we can measure if we are making progress towards the vision.

Let me explain with an example. Let’s talk about a products search engine and let’s assume the vision of the product is “ be the easiest and most helpful way for customers to find products”. Let’s define the outcomes we want for customers and the corresponding metrics to measure these outcomes:

1. Customers should be able to search however they want and should be able to see highly relevant search results that they find useful. Let’s start with the phrase “however they want”. This indicates the growing need for customers to search via voice, pictures, typing or potential means. We believe this will result in more customers using search more often. We also want to make sure customers see relevant results and show their interest by clicking on them. We could measure these outcomes through a variety of metrics including “number of users”, “number of queries”, and “number of or percent products clicked”. We will dive deeper into these later.
2. Customers should find the product useful enough to add it to their basket. This outcome has a few nuances to it. If it is not a retailer, this is not a relevant metric for the product. Even if it is a retailer in the business of selling everything, say Amazon, customers might simply drop in to check prices or reviews on a product they want to buy. They might not make a purchase in that session or ever. They might find a better deal elsewhere. However, if we believe this is a customer behavior we want to push, it will be useful to monitor a metric like “no of queries resulting in a bought product” or “no of customers who search and then buy” or “total sales coming from customers who search” and so on.
3. Customers should come back and use the product again or in short, we want a higher retention rate for the product. Great products provide enough value that they bring customers back. However, this might be limited by the nature of the product itself. Dating or matrimony sites should or would have low retention rates. People only buy and sell homes a few times in their lives and so online housing purchase marketplaces will have low retention. However, people message their friends and family, and check their calendar almost every day and those apps will have high usage and retention if they are good products. The definition of retention varies depending on the business and product. For a search engine on a retailer site, people might come back a few times every week or month. If it’s groceries and everyday essentials, this might be a weekly activity. If it is Walmart or Amazon, selling anything and everything, people might stop by multiple times a week.

So, how does one decide which one of these metrics is more important that others? Additionally, are we missing certain metrics like page load time or crashes and errors?

On the first question, depending on the business, we could select one or a few metrics to monitor. However, when you set goals for the quarter or year, you might not set goals for all of them. For example, for a search engine on a retailer site, that’s been around for a while, some important metrics to monitor include:

1. percent of site users or sessions who use the search engine (a few users will likely make up most of your site or app sessions); this is an indication of whether people find the search engine useful; we could also think of “percent of sessions that have multiple searches” which could be a good or bad thing. for instance, if a customer types in “hiking boots” and cannot find the one they are looking for, they might rephrase their query and try searching multiple times. this is not a great experience and should not counted towards success.
2. percent of search users or sessions or queries who click on the first search result, (the theory is that a search engine should show the most relevant result on top making it easy for customers to find what they need) or who click on at least a few results (the ‘few’ could vary depending on the category. for instance, people typically buy the same paper towels, but they probably want to see multiple results before buying a perfect gown or an expensive laptop; to keep it simple, you could start with 1)
3. percent of search sessions that lead to a search result added to cart; this metric has other nuances. for instance, a user might search for apples but add oranges to their cart. does that count as success? it depends on how you think about customer value. one could argue the customer was out looking for fruits and ‘apples’ was just a proxy and they did buy a type of fruit.

On the second question “what other metrics are important to track and monitor besides these customer outcomes”, I will cover that in Part 3. 

Before we go there, let me take another example to illustrate defining outcomes and metrics for a different product. Let’s say we are building a photo sharing app which customers can use to upload, share and manage photos. Let’s assume our vision is “help people save and enjoy their memories”. I am keeping the vision broad so we can focus on long term customer goals and fulfill more of their needs in the future. Following are a few key customer outcomes and metrics we should think about:

1. Do we have active users who are using the app to upload photos? This could be measured in the form of daily or weekly active users. It’s common for most people (anyone with a handy device including teens, adults and seniors) to take photos every day. Hence “daily active users” might be a good metric to capture and measure. Activity could be defined as uploading a photo (this might happen automatically based on the device’s settings), searching photos, editing photos, or sharing photos. This metric is important because we want more and more people to use the app. In the early days, growth might be slow and as the app proves it’s value, it will attract more users. 
2. A stronger outcome could be get users to share photos. Sharing could be done via messaging, posting on social media, broadcasting on a TV or more. Sharing is one of the main reasons people take photos — with their memories with their friends and family and to celebrate. Sharing also creates more visibility and buzz for the product as recipients can often see the photo sharing app the sender is using. Hence, easy and seamless sharing can improve both acquisition and retention. This will mean integration with messaging apps, social media apps, and whatever tools users use to share. The metric here could be “total or percent users who use the app to share photos daily”. A higher percent number will likely lead to high user retention. 

We could think of more outcomes around searching or editing, tagging photos and more. If uploading photos is a more intentional activity, we could measure percent of users who upload photos weekly. When measuring the health or success of the product, it is good to have a few metrics but it is important to prioritize the top one or two we care about the most. In this case, it could be “total number of or percent users who use the app to share photos daily”. 

In Part 3, we will go through “internal metrics” that product owners need to move or influence the customer facing or “external” metrics.