PM-in-the-Loop: A Framework for AI Product Management

Will artificial intelligence systems replace product managers? It’s a question that’s likely been in the minds of product management and executives alike for over a year. My opinion…is no. There will always be a need for product managers to actively oversee the AI development process.

Despite the advances in the field of General Artificial Intelligence (GAI), current AI lacks the human product manager’s intuition and ability to piece together the nuances and subtleties that are integral to the product experience - like cultural context, market trends and user needs.?

These autonomous systems lack the ability to empathize and can easily lose sight of the overall user experience. AI can’t fully grasp the understanding of customer needs. It’s also prone to ignore ethical considerations, and fail to account for real-world constraints.?

The algorithms and data models that power AI applications which allow us to automate certain tasks and analysis will continue to find its way into the practice of product management. The savvy PMs will find ways to enhance their practice with AI tools while actively overseeing the AI development process, providing ongoing guidance and supervision. Hence the PM in the loop.

To stay in the loop, the AI product manager can utilize frameworks to guide the development and management of AI products.

The AI Product Manager

The process of developing and managing AI applications is complex and requires an interdisciplinary approach. An AI product manager focuses on the application of artificial intelligence- its various subfields such as deep learning, machine learning or generative AI to enhance- or develop new products. A few articles describe the AI PM role. Briefly, the AI product manager has a unique blend of technical skills including a strong understanding of data processing and statistics as well as risk management.

The AI product manager plays a central role in realizing the product and company vision because they are adept at understanding and communicating complex technical concepts to the broader business organization. The best AI product managers will know how and when to apply a framework to the AI product management practice.?

The most common framework within data science is the CRISP-DM method. This method provides a structured approach to overcoming the challenges of developing and managing AI/ML models and systems.?

Let’s take a closer look at the application of a framework to product management.?

A Framework for AI Product Management

A good framework provides a structured approach, a set of steps or guidelines to follow throughout the AI product development, implementation, testing and deployment phases. The framework allows the team to learn and iterate in order to achieve the business objectives,??

A framework developed by Yuying Chen-Wynn provides an approach to developing and managing AI products. Her framework is based on the CRISP-DM methodology and includes some components of the business lean canvas.?

I’ve adapted Wynn’s framework with a few additional ideas to incorporate responsible AI:

Any Framework begins with a Product Strategy

A strong product strategy is the foundation to creating any AI product. The fundamentals of product strategy are consistent regardless of the technology being applied. The business lean canvas can be applied to define the strategy, and there are three main components:

1. Start with the Problem. The strategy should focus on addressing a real and significant problem that users have and is a pain that they’re aware and bound to seek a solution for. The innovative solution must generate value to the business and the customer. The AI-PM should ask the right questions to both the business and customer and use various research and customer discovery methods. It’s important to decouple the problem hypothesis and assumptions from any proposed solution and evaluate these independently.?????
2. Identify the Right AI Solution - The next component is to evaluate which AI solution is best suited to solving the problem. Not every problem is an AI problem. GenAI is one of many advanced data science and analytics techniques. The AI-PM will collaborate with their data science and technical teams to identify the right AI approach.
3. The Unique Value Proposition and Key Differentiator - It’s important to identify and evaluate the key advantages of the solution and how they’ll provide an edge in the market. Some of the key differentiators specific to AI include:

• Access to specialized datasets?
• Having a defensible IP that addresses risks such as hallucinations and repeatability.?
• Deep domain expertise in a horizontal use-case
• A channel for effective monetization
• Access to a broad customer base.??

Implementing the Framework

1. Set the Product Direction

Setting the direction of the product ensures that the solution is aligned to both the user needs and the business objectives. Understanding the problem within the business context allows us to define the objectives and desired outcomes.?

Creating milestones and a development plan and ask questions about:

• how good the models should be before implementing them?
• What computation resources should be allocated?
• What human resources should be allocated?

Establishing Success Criteria and Metrics

The criteria for measuring the success of the AI product management strategy should include aspects related to the product performance, its impact on business goals, user satisfaction and adaptability.?

2. Data Planning

We know that AI/ML systems and products are trained and evaluated on a vast amount of data to support the AI model. Data planning needs to happen early in the process. The AI PM applies their analytical skills to understand the data source, quality, infrastructure and characteristics to support the AI model. During data planning:

Identify data sources: The most important thing is to confirm if the data sources are of high quality and if the data are relevant to the problem that is being solved.?

Data Prep and Exploratory Analysis: Make sure that you have a system for gathering, preparing and cleaning the data. Perform any exploratory data analysis to understand the relationships between the target variable and feature variables.?

The team should investigate bias within the dataset such as historical bias, measurement and representation bias.

Infrastructure Plan: Determine what technical resources are needed to store, process and analyze the data depending on the scale of the data and the computational requirements.?

It’s important to consider the organization’s technical strength and limitations during the concept and design phase.

Future Proofing: Good AI is scalable. Plan for how your data needs will grow over time and consider storage, processing and future data sources.?

3. Selecting the Model

Having a good understanding of which Ai solution is best suited to solving the problem will determine which model(s) can be utilized. Remember, that not all solutions require an LLM, and it might require a different data science approach. The model will determine the fundamental capabilities and limitations of the final AI product.?

There are various types of machine learning and AI models. The AI PM works closely with the data science and engineering team to select the appropriate model while considering the following:

Know the Characteristics: understand the scope of the model, scope of the dataset and the performance metrics for key scenarios. Basically how you want the model to behave and perform within the business use-case.?

Involving Business Stakeholders: Selecting the model should also involve the business stakeholders. The AI-PM in the loop is constantly translating the technical concepts to business stakeholders in order to ensure alignment with the defined business outcomes.?

4. Analyze Risk

There are various risk factors inherent to emerging technologies such as AI which can impact the end-user, the business and the organization. A good risk management process intends to? anticipate, assess and mitigate potential risks to the organization and ensures that AI products are developed and employed responsibly. This requires that the AI PM collaborate with legal, compliance, finance and business teams in developing a thorough risk canvas.

A risk canvas may include some of the following:

5. Build & Evaluate

AI systems are continually evolving and learning through their interaction with data and users. That is why AI product development requires an iterative approach where continual testing and system tuning is important to meeting the desired performance benchmarks and creating the right user experience. The process may also involve testing and comparing various models.?

Involving users and customers at this stage helps refine the product based on real-world usage. Collecting user feedback can be achieved through usability testing, A/B experimentation, beta tests and interviews.?

6. Deploy, Observe & Monitor

Operating in a controlled test environment is completely different to the real-world scenarios in production. In production, the data may evolve and the performance of the AI system may drift and even degrade over time. Once the AI system is in production it is important to observe and monitor the systems’ performance, integration and the users’ experience. A system for regular evaluation needs to be in place to ensure that the AI product is performing as expected.?

Establish Protocols: Along with regular evaluation, establish a protocol for model retraining based on the evolving nature of the data. This can be done by having:

• A systematic retraining schedule to keep the model up-to-date with current data patterns
• Establishing performance thresholds and triggers for model retraining.??

7. Operationalize Continued Learning??

The nature of AI products requires continual learning, iteration and improvement. The AI PM’s aim is to gather user insights and data-driven evidence of the AI model’s effectiveness and have a process in place that guarantees safety and responsible use of the AI product. All of this requires a structured approach that includes ongoing monitoring, feedback loops and governance. Here are the criteria to operationalize learning:

Continuous learning and adaptation:? The AI system behavior changes over time because the models can learn from new data and user inputs and interaction. This means that the development cycle needs to be flexible and adapt as the model evolves.?

User Feedback Loops: The Ai product should be designed and instrumented to incorporate user feedback on a regular basis. The mechanism should assess how well the product is meeting the users’ needs and collect recommendations for improvement from the users’ point of view.?

Responsible AI: A common misconception that AI is objective. But an AI system can only be as objective and make decisions based on what humans taught or trained it.??

The system should continuously be monitored and tested for biases, fairness and other ethical issues. Consider implementing a responsible AI framework that, for example, prioritizes:

• Fairness
• Transparency
• Accountability
• Explainability
• Interoperability
• Inclusivity
• Data Security and Privacy

Facilitate Collaboration: Ai requires an interdisciplinary approach. The AI PM facilitates collaborations across data science, engineering, design, legal, compliance and business teams across the organization to ensure a holistic view of the entire product development and management process.?

The ability to implement and operationalize any new framework requires that all stakeholders are prepared for- are brought in and aligned to the new direction. When it comes to AI, effective change management doesn’t just involve the product and engineering team. It requires effective collaboration and alignment with every team in the organization.?

Managing Change

The ability to implement and operationalize any new framework requires that all stakeholders are prepared for- are brought in and aligned to the new direction. When it comes to AI, effective change management doesn’t just involve the product and engineering team. It requires effective collaboration and alignment with every team in the organization.?

The IA PM should consider the following:

Assess the need: Evaluate the impact of the proposed change on people, process and technology. Understand where improvements can be made, which solutions are already being used and identify opportunities.?

Implement a Top-Down Approach: Effective change starts at the top. Begin by understanding the executive leadership rationale for the change. Leadership should clearly communicate and help evangelize the change across the organization, emphasizing the benefits and value to be gained.

Align Stakeholders: Map out your stakeholder groups and identify those who are impacted by the change. Use the ‘What’s In It For Me’ (WIIFM) approach to communicate the value to be gained for the new process or change. Emphasize how it will specifically benefit them.?

Build Champions Across the Org: Identify those individuals who can champion and help promote the change across the company. 1. Empower individuals to be an educator, those who understand the framework very well and can quickly align those who do not understand it.

2. Enable other individuals to be a coordinator, those who understand the vision for the new framework.

3. Activate the peer or team member who is most closely tied to the group that you want to adopt the change.?

Provide Training And Documentation: Support the transition with materials, resources and training guides. These materials should be periodically updated to support education and adoption throughout the org.

Final Thoughts

The demand for AI product managers is increasing because of the surge of AI technology in the mainstream. AI is here to augment and enable product managers in their practice, not to replace them. The PM in the loop is essential to the development, deployment and management of AI systems and products. The AI product management framework provides a path to combine human empathy, creativity, foresight and judgment with the analytics power of AI to create impactful solutions that solve real problems for humankind.?