A Data Scientist's Guide to New Product Launch

Why This Article?

This article is for Product Analytics Data Scientists who aspire to help new products/features to grow from 0 to 1 and for product leaders who are interested in adopting a data-driven strategy to launch your new product and iterate at a faster speed.

Over the past 2 years at LinkedIn, I've been involved in three significant new product launches as a Data Scientist, watching their development from a mere idea, to a minimum viable product launch, and ultimately a fully-realized product serving millions of LinkedIn members. Throughout this journey, I didn't find many helpful resources outlining the value data and Data Scientists can add to new product launches so I decided to summarize what I learned to share.

My goal is to share the insights I've gathered, particularly beneficial for companies with an existing customer base looking to expand their value offering with new products. For startups just beginning to scale, data strategies may become more relevant as your customer base grows to a size that can't be handled by a single Excel Spreadsheet :)

What Will I Cover?

I liken the role of data to a compass for pilgrims navigating through the desert. Venturing across the desert without a map can be a daunting experience. While data won't entirely eliminate the challenges of building a product, it does help alleviate anxiety by ensuring you're headed in the right direction.?

I will describe three key product stages where data can add values in building a successful product:

• Product Ideation
• Before Product Launch
• Post-launch Iteration?

Product Ideation?

All products start with two fundamental questions: 1) the user problem - WHO we help to solve what particular challenge they face; 2) WHY us: why is our team better-positioned to tackle this user problem vs. any other company/team. A clearly-defined problem statement rallies the team before we embark on the journey to discover HOW (the solution).?

As exciting as it sounds, this stage often involves lots of ambiguity and debates. User research and focus groups play an important role at this initial stage, testing out ideas at small scale before building anything real. Data typically adds more value in the later stage after we’ve launched the product, but it can help test our hypothesis at this stage; it allows us to gauge the size of our market and observe the user behavior to validate whether the pain point does exist. You might have a great idea but ended up discovering that it appeals to only a very small market segment.

Data can also drive the generation of hypotheses. A famous example is Instagram's pivot, which all started by an observation that a significant number of photographers uploaded photos to the platform.

To give a concrete example, before we launched Job Collection , our hypothesis was that some LinkedIn members are casual job seekers. This means they are not actively seeking but open to the right opportunities. To understand casual seekers on LinkedIn, we conducted an analysis to understand the size of the population and what they do today. The result helped validate that it's a big enough problem to solve as well as inform the type of collections to provide.?

Before Product Launch

Congrats! If you have come to this stage, that means you have a product that is ready for the market test. Most product ideas get killed at the ideation stage. Before the product launch, there are a couple checklist items to ensure data-driven strategies.

• Define Success Measurement- Define success metrics- Tracking- Data ETLs to operationalize the metrics
• Develop Launch Strategy and Experiment Setup- Define target audience and rollout schedule- A/B test setup

Define Success Measurement

Success Metrics

"If you can't measure it, you can't improve it" is a famous quote from the management guru Peter Drucker. It's also true for your new product. If you have defined a clear problem statement, this step is about translating the goal into a measurable set of metrics. Typically, you will need a single true north to measure the overall success of the product as well as a set of signpost metrics to help you diagnose the performance of each sub-component of your product (e.g. which funnel steps members drop).

The true north metric should be able to tie back to the overall success of your business.?

Tracking

Now that we define success metrics, how do we make sure the metrics show up in the dashboards every morning after product launch. Tracking is the foundational step to ensure we have the right data - Data Scientists work with app engineers to identify the data points (e.g, user impressions, actions, conversational data etc) to collect, define the tracking data schemas, and ensure the tracking events are firing correctly.

Data ETLs to operationalize the metrics

Tracking data typically comes in the streaming events that is not easily digestible for reporting tools. To go from raw tracking to clean metrics, we need to add a middle layer to Extract, Transform and Load the data (ETL). In some companies, the step is owned by Data Engineering but the goal is to load the raw tracking data and transform them based on the business logic to calculate business metrics and save in a clean table to connect to downstream applications like a dashboarding tool or experimentation platform.

Develop Launch Strategy and Experiment Setup.

Define target audience and rollout schedule

Ramping a product is exciting but we also need to be careful. To ensure our product meets our quality standards from the outset, it might be prudent to initiate a phased launch, starting with a select group of users. This approach allows us to gather valuable feedback and make necessary adjustments, thereby enhancing user experience as we gradually expand our user base. Some suggestions for designing the ramp strategy.

• Pick the MVP audience

Consider which country, language and platform you are initially ramping your product up to. It should be the audience you believe the feature will work the best and core audience your company is most familiar with.

• Start small and ramp up carefully

We want to ramp the product at a small percentage first to make sure all major bugs are cleared up and it doesn't heavily cannibalize other product features before we continue to ramp up.

A/B test setup

Experimentation (A/B testing) is the gold standard for measuring the impact of a new feature, by randomly assigning some users to receive the features and others to a control condition without it. It sounds simple but there could be lots of complexities involved. To name a few -

• Members come in and out of the experiments. Think about if you are ramping this new product to your paid users, but paid users are joining and churning every day. How do you handle them in the experiment?
• Isolate Go-to-Market and product features' impact. Along with new product launches, companies typically send out GTM to share the news including emails, notification, pop-up screen etc. How do you decouple the impact of GTM and the product features' impact on your product engagement?
• Understand the long-term and cross-ramp impact. One round of A/B tests can help us understand the impact of each launch at one time but what you want to understand is the overall impact of multiple product iterations over time even when the macroeconomic conditions change.?

Post-ramp Launch

Hurray! Your team has come a long way to get the product in front of users. Now what - it's time for the real data power comes in to understand the Product-Market Fit and help fuel the growth of the product. Post-ramp, data are valuable sources to

• Help the team understand the market reaction and user feedback
• Conduct Deep-dive analysis to guide product direction

Understand the Market Reaction

We keep our fingers crossed and wait for the few days of data to come in. In the first few weeks, we typically focus on understanding the adoption of the product and also any abnormal data points that may indicate a bug needing to be fixed. Data Scientists work as a raven to collect all the quantitative feedback and sometimes qualitative feedback, digest it and share the key takeaways from the team. The goal is to answer "Is the product working properly?”, “Do the users like this product?".

Deep-dive analysis to guide product direction

If the initial market reaction is positive and the decision is made to invest more resources, we will conduct more deep-dive analysis to help fuel the growth of the product. I found a few angles helpful:

• Understand the usage funnel from Awareness → Activation → Engagement → Conversion →Retention
• Understand the retention: retention is the key for most of the usage products and it's an indication that people find it valuable and keep coming. Here is a great benchmark on the typical retention numbers by industry.
• Identify and understand your power users: if your product retains users, that means something is enticing them to come back. It might be or might not be the intended user case you designed for them. Further data analysis, combined with UXR can help shed some light on the value drivers that make them stay.?

Of course, this only scratches the surface of how to use data to guide the product direction. I will start another series to delve deeper into this topic.?

In Closing?

In summary, this guide takes you through the journey of launching a new product, focusing on areas data science can add value. If you find this article helpful, follow me on LinkedIn! Last but not least, thanks to Bonnie Barrilleaux for helping to review this article!