Product Case Question - Launch a feature or not?

When interviewing for a product company, a case question determines whether a candidate can methodically tackle a broad question and make a recommendation for the team to act on.?

Let us dive into a practice case. This question throws light on the type of cases you can expect when interviewing for a product company.?

Interviewer: Facebook is considering showing more ads in place of the “People you may know” feature. Do you think the team should go ahead with this implementation??

Candidate: I would like to start by talking about the overall goals for Facebook and what I think the goals for the features you described are and then go from there.?

Interviewer: Sounds good.

Candidate: Facebook enables people to stay connected with their family and friends, and its goal is to give people a platform to build a community with whom they can share their experiences. From what I know, the “People you may know” feature directly correlates with their goal and enables users to build their existing networks. Furthermore, Facebook’s primary source of revenue is ad sales, some of which come from the ads directly displayed on the news feed, which is the feature we are talking about I assume.?

Interviewer: Yes, you are correct about the news feed assumption and the goals you mentioned are aligned correctly as well.?

Candidate: Great, now I would like to think through the pros and cons of each feature.?

While there are trade-offs for choosing either feature, we should align these decisions to any business objective that Facebook has. Is there a specific short- or long-term goal the team is working toward?

Interviewer: No, not specifically for the scope of this question.?

Candidate: Okay. In that case, I propose running an A/B test to assess the impact of showing more ads to a randomized group of users and compare it against a control group who will see the news feed as is.??

Interviewer: Sounds good. What metrics will you track when you run the A/B test?

Candidate: Let’s see. From the point of view of the two options, I will look at the following:

Ads (metrics related to monetization)

• Average revenue per user
• Average revenue per active user

People who you may know (metrics related to engagement?and retention)

• Likes
• Comments
• Shares
• Clicks?
• Total number of users?
• Total number of active users

Interviewer: Thank you for listing those out. I see that you have listed out average revenue per active user specifically. Is there a reason you have in mind to track that?

Candidate: Yes, absolutely. From what I understand, Facebook users can be grouped into two key segments:?

1. Active Users
2. Inactive Users

Active users engage with the product often and are the first to react to any product changes. Hence, we must track any changes in the product usage of the test group.?

Interviewer:?Okay, let’s say we got the results back from the A/B test run over a two-week period. Based on these results, what is your recommendation?

Candidate: On one hand while I see avg. revenue going up by 1 percent overall and 2 percent for active users, the number of likes and shares have gone down, indicating that the user engagement has taken a hit. While the comments metric has gone up, we must evaluate if these are positive or negative sentiments. People could be negatively commenting about the increase in ads, so looking at this increase in isolation may be deceptive.?

In summary, given that Facebook’s primary goal is to build a platform to connect users, I think a significant impact in the engagement metrics is a sign that something is not right. Facebook needs to find that optimal point where engagement is not impacted but the revenue generated increases. For now, I would recommend to not show ads in place of the people you may know feature.?

Comments: The candidate offered a very structured and thoughtful discussion around the pros and cons of using one feature more over the other. The candidate was able to use the data provided well and was able to convince the interviewer that they have a reasonable plan in place to approach such a problem.?

Questions for practice related to this case:?

• How do you measure the success of Instagram stories?
• How will you forecast user sign up for a new product??
• An online platform suddenly saw an increase in sign ups. What could be the reasons for this?

In this newsletter, I share excerpts and stories from my book,?Be the Outlier. I hope you enjoyed this post — if you like the content and want to connect you can reach me via?LinkedIn. Also, you can also find my book on Amazon?here.