The Hidden Truth about AI Product Metrics (Part 2)

In this article, we will go deeper into AI product development metrics and AI risks.

In Part 1, ‘AI is eating Product quickly’ I covered the paradigm shift for traditional vs AI product management.

So why should we care?

Almost all software will soon be AI-powered and nobody will talk about this. It's like a cloud-based app. Nobody cares these days, everything is cloud-based.

As software development is only going in one direction, I wanted to share the most common key AI product metrics and ways to handle AI risks to help you succeed as an AI Product Manager.

So let’s dive in…

1- AI Risks and Mitigation

2- Basics of machine learning for PMs

• a- Supervised Learning
• b- Unsupervised Learning
• c- Reinforcement Learning
• d- Free resources to go deeper

3- Learn your key AI product metrics

4- Should I use ML in my product design?

5- Key Takeaways

1- AI Risks and Mitigation

Risks are always involved when it comes to new tech, like AI. It reminds me of the cloud computing wave in the early 2000s. Senior execs are likely asking you for your take on AI risks in your organizations. It's super important to spot these risks early on and expose them to the team as you start exploring and possibly launching AI features.

Let's break down the key areas you need to keep an eye on:

Data Security and Compliance: First things first, let's talk about keeping your data safe and following the rules. Make sure your data is handled properly and any (PII) personal info is removed to protect your customers. Avoid using data that's biased or collected illegally. Make sure you're following all the legal stuff and that your systems can handle the data safely.

• eg: Google's got strict rules for handling data to train its AI models. They make sure they're using top-notch, fair data and have super secure systems to handle it all. Even though, some embarrassing Gemini AI comments made the headlines once again.

Risk Management Gates: Next up, let's talk about putting up some checkpoints along the way. You want to have systems in place to check things as you're building, testing, and launching your AI features. This helps spot any issues early on, like if your feature starts acting wonky or if you're relying too much on outside vendors.

• eg: Facebook's got systems in place to shut down certain AI features if they spot anything weird happening. It's all about keeping things stable.

Feedback Loop: Now, let's talk about staying connected with your users. You want to keep gathering feedback and monitoring how they're using your AI features. This helps you see what's working and what needs tweaking.

• eg: LinkedIn keeps an eye on how people interact with their job recommendation AI. They use this feedback to keep improving the system.

By keeping these tips in mind and making the most of AI in your product management journey, you can handle AI risks like a pro and drive some great AI discussions in your own organization.

2- Basics of machine learning for PMs

Now, we will discuss the difference between Supervised Learning and Unsupervised Learning which are part of the main discussion you will have with your data science or ML engineering teams.

a- Supervised Learning

Imagine you have a bunch of ‘labeled’ data' like pictures of elephants, cows and camels above.

In short, you teach a machine to recognize these animals (Label it). You feed the machine images and tell (label) it which ones are elephants and which ones are cows.

The machine learns to map features (like size, body length, ear shape) to the correct label (elephant or cow).

This process is called supervised learning because the data is labeled. This is the main difference with unsupervised learning.

Classification vs. Regression Under supervised learning, there are two main methods you’ll most likely discuss in your organization.

• In classification, you're drawing a line to separate different groups. For example, if you have data about two types of patients, classification helps you figure out which patient is sick based on their vitals.
• In regression, you're predicting a continuous numeric value. For instance, if you know patients' vitals, you can predict their sickness rate. Metrics like root mean squared error or R-squared measure the difference between our predictions and actual values. This is plotted on a line to show the relationship.

b- Unsupervised Learning

In unsupervised learning, you don't have labeled data. You just feed a lot of data into the system, and it finds patterns or clusters like the example above.

Three clusters are detected. e.g. It might group people based on their shopping habits without being told what those groups are.

?? The product mindset

Ask yourself, would a bad recommendation impact negatively your user experience? Any legal or compliance impact for wrong predictions?

e.g. A wrong article recommendation is no big deal on Amazon while a suggested patient diagnosis for a radiologist may be more critical.

That’s your tradeoff decision.

c- Reinforcement Learning

Reinforcement learning is about teaching a machine to achieve a goal by rewarding it for making good decisions. It learns by doing and getting feedback. Imagine a robot learning to play basketball. It tries different throws and learns the best angle and power to score.

Real-life example: Tesla’s self-driving cars use reinforcement learning to navigate roads safely.

d- How can I learn more?

a- Online Courses: Platforms like Coursera, edX, and Udacity offer courses specifically designed for non-technical learners. eg "AI for Everyone" by Andrew Ng or "Elements of AI".

b- Books: "Artificial Intelligence: A Guide for Thinking Humans" by Melanie Mitchell or "Prediction Machines" by Ajay Agrawal are great starting points.

c- Blogs and Podcasts: Follow industry leaders and AI-focused blogs like Towards Data Science, and listen to podcasts like "AI in Business" to stay updated on trends and applications.

3- Learn your key AI product metrics

a- Understanding Model Metrics for Product Managers

As a product manager, it's crucial to work closely with data or research scientists to define the problem and choose the right machine-learning technique.

These experts or ML researchers have advanced statistical knowledge. Even if you don't create these models yourself, it's important to understand their measurements.

These metrics help you make informed decisions about your product's performance, whether to launch it, or when to make changes.

b- Precision and Recall metrics

Precision measures how many of the predicted positive cases are actually positive, while Recall measures how many of the actual positive cases our model correctly identifies. These metrics help us understand the performance of our model, especially in scenarios where some classes are rare or more critical to detect.

c- Understanding the confusion Matrix

A confusion matrix helps us evaluate our model's precision and recall. Let's dive deeper into our example.

The results are shown in the graph above:

• The target variable has two values: Positive or Negative
• The columns represent the actual values of the target variable
• The rows represent the predicted values of the target variable

Precision assesses the quality of our predictions by looking at the number of true predictions out of all true cases. In our case below, the model predicts 30 out of 60 true cases correctly, making it 50% precise - so just average.

Recall, on the other hand, focuses on the quantity of correct predictions out of all true cases. Our model correctly predicts 30 out of 40 true cases, giving it a recall of 75%. While the precision is ok, the recall isn't that bad.

As a product manager, one common task is to analyze the trade-offs between precision and recall.

• Precision is crucial when False Positives are a bigger concern than False Negatives. It's critical for things like music or article recommendations, e-commerce sites, etc. Incorrect results here can drive customers away and hurt the business.
• On the other hand, Recall is key when missing a positive case (False Negative) is worse than a false alarm (False Positive). This is especially important in medical use cases, where catching every actual positive sick case is critical, even if it means a few false alarms.

Exploring Other Metrics

While accuracy, precision, and recall are key metrics, there are many others like logarithmic loss, area under the ROC curve, and mean absolute error.

It's crucial to collaborate with ML and data science experts to track the metrics that align with your intended product goals.

4- Should I use ML in my product design?

Before diving into a machine learning-based solution, it's crucial to evaluate whether it's the right approach for your problem.

If the answer is YES to all those questions, DO NOT use ML!

1. Can it be solved with classic programming? First, classical programming might be a better fit if the problem can be solved using simple rules. It's less complex and faster to build.
2. Do you need 100% accuracy? Additionally, if you require 100% accuracy or need to interpret results comprehensively, ML might not be the best choice.
3. Do you need to explain the reasoning behind your insights? ML models often lack transparency, making it challenging to understand their decisions for your users. Moreover, if your solution doesn't need to adapt to new data, reconsider using machine learning.
4. Can I only have poor data quality? If your problem fits with ML, check your data quality and sources. You need a large, secure, relevant, and fair dataset to create a good ML model. A common error is using bad data, which gives poor results. Remember, "garbage in, garbage out." Bad data leads to bad results, hurting your model's trustworthiness and user experience.

The bottom line is, when a simpler solution exists, go for it. It saves time and resources in the long run.

Until next time…

Key Takeaways

#AI #GenAI #productmanagement #AIProduct #MLProduct

Jean-Michel VAN is an avid (slow) runner, a foodie (only dessert), and a AI product leader, helping product folks accelerate their careers and leadership with tactical frameworks and candid tips - and ultimately helping them build better products.

He shares his journey at Another PM Day, and is a coach at Product School and Product-Led Alliance.

Connect with JM on LinkedIn