How do you measure success of a product ?

?????????????????????? ??????????

• Reiterate the question
• Talk about the structure of your answer
• Ask Clarifying questions (if any) like why we are going to measure the goal of that product.
• Ask whether there is any specific scope e.g. if the problem is to measure success of Facebook stories, ask whether scope is United States or globally. Suppose you have to measure success of FB likes, then ask whether you can keep the scope to posts on newsfeed.

??????????????, ???????? & ??????????

• Describe the product - Clarify what the product does. , why the product exists and how it is aligned with the company’s mission.
• Stakeholders- Who its users are etc – Describe the entire ecosystem and all the stakeholders
• Mention the value it is providing to all stakeholders and the company.

??????????

• What is the product vision? Where do we see the product growing in the next five years? For example, if we’re working at Whatsapp, perhaps our overall goal is to see the platform grow in terms of user retention. We want to ensure that users who visit our platform discover useful features and stay on the platform.Another example - Understand what Youtube wants to do this year, what are the priorities Youtube should focus on. Say we want to increase watch time i.e. no of hours watched on Youtube
• Ask what stage the product is in. Goal of a new company is usually growth whereas for others it can be engagement.
• Now describe what all strategic goals the product can have – like acquisition, engagement, retention , monetization etc. Usually there are mainly 4 types of goals for any product : growth, engagement, retention or revenue.
• Note: Generally, revenue is NOT a recommended goal. Companies who focus solely on revenue fail to consider that user experience is a more important health metric for the overall company.
• Now depending on product stage and the vision, state which one should be the goal and why e.g The goal of stories is to increase overall engagement on the Facebook product by creating a new way for users to engage with others.
• Decide whether we should think of the product market fit or grow the product and increase the product in terms of engagement.
• Also talk about the product objective and how you can measure it e.g. if it is FB dating then you need to measure a successful relationship.

Tips

• High engagement may not always be good e.g. if time spent on a particular product is high it may mean users are taking long time to find something or they have to navigate a lot to find something.
• Retention may not always be good e.g. if users are coming back to a dating site that means they are unable to find what they want.

??????????????

• What actions do we want our users to take that support these goals?
• For instance, which features or actions drive engagement? Maybe we want our users to send at least one message per day, since these users often stay on the platform for several months.?
• Mention actions of all the stakeholders which support the goal.
• To find actions focus on the user journey map?

Example : Suppose you have to measure success of WhatsApp story/status update

There are two types of users : Story Creator and Story Viewer

Metrics

• First you need to select a NSM or North Star Metric – A north star metric is the one metric that captures the core value that your product delivers to customers and provides value to all stakeholders.
• Then you need to choose a set of 4-5 metrics
• Select metric for all set of stakeholders
• Then prioritize the metrics
• You may also choose NSM after prioritizing the metrics

Example

In previous example the metrics can be :

• % of DAU who post a story (Priority = 5)
• Average no of stories posted per user per day (Priority = 5)
• % of users who left the feature (Priority = 3)
• DAU who view a story etc. (Priority = 4)
• Story reply rate (Priority = 4)
• Prioritize all these metrics and you can come up with NSM
• NSM = Average no of stories posted per user per day

Evaluation

Proxy metrics

• Some times we may find it difficult to measure some metric. So we may use proxy metric in its place e.g. It is difficult to measure product impact through one take North Star metric. e.g. if you have to measure success of Facebook Marketplace NSM should be no of transactions.
• But you don't know whether transaction happened or not because it often happens outside FB. So create a proxy metric which is more related with the north star metric. For example, number of conversations between buyer and seller. It is easier to measure this. And if it moves top line metric moves.

Counter metric

• You may be asked about counter metric. Counter metric is a metric which may go down if the Northstar metric goes up.
• A counter metric is something that you measure to ensure that you haven’t over-optimized your north star metric to the detriment of your customers and your business. If your north star metric is increasing at the expense of your counter metric, then you didn’t actually move the north star metric forward in the way that you wanted it to.
• If you only measure the north star metric, you won’t have any visibility into other key areas of the business that you need to maintain. You won’t be able to capture any negative downstream effects that could come with changes to your north star metric.

Types of counter metrics :

• In many cases, products cannibalise other products. For example facebook dating app can cannibalise engagement in news feed. For example, if engagement in Facebook dating app goes up, then engagement in News Feed can go down. So engagement in news feed is the counter metric in case of Facebook dating app.
• # of reported content or # of posts taken down or NPS going down

Other Evaluations

• Of course, as with most PM questions, reflect after you deliver your answer. How might this metric be a false-positive or false-negative?
• What are some cases where your defined metrics may show positive, but cause deleterious effects for the platform?
• For instance, perhaps users are upgrading their accounts to premium, but then, after a few days, refunding their purchase because of dissatisfaction with the premium account.
• These false-positives or false-negatives don’t mean that your answer is wrong — it’s just always worth mentioning the tradeoffs and potential areas of concern with your selected metric.
• ?Talk a little bit about the qualitative analysis e.g. increase in watch time will actually create an addicted user base who will ultimately hate Youtube when they realize how unproductive they have become.