Collecting data has no value - If You Don't Do Anything With It

In a world where we breathe and it's converted into data, where does Product Management fit into this landscape? And how can we, as Product Managers, make the most of it?

Types of Data Collection

We cannot separate the "old school" method of data collection: talking directly to customers, from the “modern” data collection approaches which involve using digital products to understand exactly how people interact with our product.

Especially because people don't always do what they say they do, and most of the time they don’t need what they say they need.?

As Akio Morita, co-founder of Sony, once said, "We don’t ask consumers what they want. They don’t know. Instead, we apply our brain power to what they need, and will want, and make sure we’re there, ready."

Data collection can be divided into two main types: Qualitative and Quantitative.

Quantitative Data

Quantitative data is all about numbers and statistics. It involves anything that can be counted, measured, or given a numerical value. Examples include user metrics like page views, click-through rates, and conversion rates. Quantitative data provides measurable and objective insights into user behaviour, usage patterns, and performance metrics.

Qualitative Data

Qualitative data, on the other hand, focuses on understanding user experiences and deeper insights that can't be measured in numbers. This involves methods like user interviews, surveys, beta tester feedback, or any other feedback, for example, reviews shared on the App Store.

Qualitative data offers context for user needs, motivations, and pain points.

Collecting Quantitative Data?

The goal is not to record every single possible metric, in fact, that’s a recipe for disaster. First, we need to make sure we’re capturing the right metrics. To do that, it's important to have clarity on the goals we want to achieve with our product. This isn't about creating an extensive list of goals; three to five goals would be ideal. After that, define the metrics that underpin those goals.

For example, if a product goal is to increase customer retention, we should track the percentage of customers who return after their first purchase and the factors that influence their return, such as customer satisfaction, time between purchases, and interaction with customer support.

Next, we group metrics so that we can spot trends and potential opportunities.?

The most common ways to group metrics are Segmentation, Cohort analysis and Funnels.?

Segmentation: involves dividing a larger user base into smaller, distinct groups based on criteria such as demographics, behaviour, or other attributes. For example grouping users by age, location, device type.

Cohort Analysis is used to understand how user behaviour evolves over time.?

For example, compare customer engagement before and after implementing a tutorial for new users.

Funnels: Understand user behaviour through the product journey.?

For example, in Airbnb, how many people open the app, click on a property, and book an accommodation?

Combining different types of metrics is also possible: going back to the Airbnb example, how many people from Portugal (segmentation) clicked to book an accommodation (funnel) before and after introducing guest reviews (cohort)?

Once you’ve nailed down your metrics, it's time to bring them to life. Tools like Google Analytics, UserGuiding, and DataDog can help you track and analyse your data. For generating clear, impactful reports, consider integrating your data collection into Looker Studio, Tableau, or Power BI.

Collecting Qualitative Data

Before you start collecting qualitative data, check out "The Mom Test" by Rob Fitzpatrick. It’s a must-read in this field.

Qualitative data collection is a big, big topic, and we’ll cover it more thoroughly in a future article. For now, here are some key points to keep in mind:

1. Clear Scope: Like quantitative data, start with a clear scope. Define what you want to learn to avoid collecting data that isn't useful or aligned with your goals.

2. Avoid "Would You" Questions "Would you buy this?" or "Would you use this?" are hypothetical and often unreliable, instead you can use "How do you currently solve this problem?". The main idea is to focus on past and current behaviours.

3. Listen and don't compromise: Focus on actively listening to stakeholders (customers, sales teams, support) and avoid making promises about features you might not deliver—you're there to gather insights, not sell future possibilities. Make the most of these interactions by listening as much as you can, and if possible, record them.

Turning Data into Action

Collecting data is only useful if you take the time to analyse and act on it. So, after you start collecting, here are a few steps to follow:

Integrate Data: Combine qualitative insights with your quantitative findings to get a clearer picture of user behaviour.

Prioritise Insights: Use a prioritisation framework to rank insights and guide your next steps in feature development and decision-making.

Communicate: Ensure everyone on your team knows the next steps for product development or issue resolution.

Iterate: Data collection isn’t a one-time event. Set up a regular schedule to review and adjust your metrics based on your product’s lifecycle. Early on, you might need to review your indicators more often (e.g., monthly or quarterly). As your product matures, you can review things every six months, or even annually for stable products, unless major changes occur.

Remember, the frequency of data reviews depends on your product’s stage, the market, and your team's resources. The goal is to find a review schedule that matches your team’s resources and ensures you can handle data efficiently, from collection to action.

Finding the right balance

The balance between qualitative and quantitative data depends on your product, goals, and development stage. If your company is seeking to base the product mainly on qualitative data, a Strategic Product Manager will be a good profile to include on the team. On the other hand, if your company seeks to base the product mainly on quantitative data a Data Product Manager will be a nice plus. Want to explore these roles further? Check out the article "I Want to Be a Product Manager When I’m Older!" Mastering both data types empowers Product Managers to make more informed decisions that can help guide their products to success.

But remember: data only becomes valuable when gathered, analysed, and transformed into actionable insights.