AI product manager for the Microsoft Azure datascience platform

Background

I am working on the idea of the AI product manager for our #universityofoxford course on #AI #generativeAI and #mlops with Ay?e Mutlu . We are also applying these ideas to the erdos community.

The idea is - we define what is an AI product and show how the role of the AI product is implemented in various platforms. In this case, we are discussing the Microsoft Azure platform.

This is a concise and dense post - but should be a good framework for anyone who wants to understand AI product management in Azure.

What is an AI Product

An AI product is a software or hardware solution that integrates artificial intelligence technologies to automate tasks, support decision-making, and enhance functionalities. Using AI (machine learning or deep learning algorithms), an AI product involves solving specific problems, optimising processes, or providing intelligent insights based on using data to improve the process aspiring to achieve human level ability.

Building an AI Product

Building an AI product involves defining the problem, identifying use cases, and setting success metrics. Data collection and preparation involve cleaning and labeling the data. Feature engineering selects relevant variables, and model selection involves choosing algorithms and fine-tuning pre-trained models. Models are trained, optimized, and validated before deployment, which includes integrating APIs, managing scalability, and ensuring version control. Post-deployment, continuous monitoring, drift detection, and retraining pipelines keep models accurate using MLOps.

The user interface (UI) ensures AI insights are understandable, and end to end security, privacy, and compliance are critical to the product’s success. Iterative improvements through feedback, A/B testing, and documentation ensure alignment with stakeholders and business objectives.

The AI product also needs ongoing support and training for internal teams and customers are crucial for adoption and long-term success throughout its lifecycle.

What does it mean when we say that the process improves with experience?

It's important to understand what we mean by 'improving with experience'.

A machine learning model improves with experience by becoming more accurate in primarily three ways: as it encounters new data, feedback, and iterations.

In traditional batch training, models are retrained periodically with fresh data, while in incremental learning or online learning, they update continuously with new data in real time. Reinforcement learning allows models to learn from their environment through rewards and penalties.

Techniques like active learning help models improve by focusing on uncertain data and seeking human input, while transfer learning enables rapid improvement by fine-tuning pre-trained models for specific tasks. Model optimization, through AutoML or hyperparameter tuning, refines performance over multiple training cycles.

MLOps practices like data versioning, CI/CD pipelines, and continuous metric monitoring help automate and manage these improvements, ensuring models stay relevant and effective over time through ongoing retraining and adjustments.

Together, these methods enable a model to continuously learn and improve with experience.

AI Product Manager

An AI product manager (AI PM) is a cross functional role that oversees the development, launch, and lifecycle of AI products. They combine technical expertise, strategic planning, and a deep understanding of AI’s potential to align the product with customer needs and business goals.

AI PMs focus on identifying valuable use cases, managing data sourcing and preparation, and ensuring the AI models are built effectively. The role involves collaboration with cross-functional teams, such as data scientists, engineers, and legal, is key to delivering user-friendly, compliant, and high-performing products.

AI PMs also manage post-launch optimization, track performance metrics, and ensure the AI product delivers measurable business value.

In large organizations, AI PMs communicate the product’s capabilities across the organization and work with marketing and sales teams to drive adoption.

Low code and full code approaches for Azure machine learning

There are two main ways to implement an AI product using Microsoft Azure AI Machine Learning services: a low-code approach with Power Platform and a full-code approach with MLOps.

Low-Code Approach via Power Platform

The Power Platform includes tools like Power BI, Power Apps, Power Automate, and Power Virtual Agents, which simplify building AI-powered products. Power Apps and AI Builder help users define problems and build models for scenarios such as predictive analytics and sentiment analysis. Data collection and preparation are automated through Power Automate and AI Builder. Feature engineering is done with Power Query in Power BI, and model selection and training use AI Builder’s pre-built models or Azure Machine Learning Designer for custom models. Power BI provides dashboards for model evaluation, while Power Automate enables continuous monitoring. Models are deployed in Power Apps or Azure App Service, with monitoring handled by Power BI dashboards and automated notifications. Power Apps and Power Virtual Agents create the user interface and AI chatbots.

Full-Code Approach via Azure Machine Learning (MLOps)

The full-code approach leverages Azure Machine Learning, Azure DevOps, and other Azure services to build scalable AI products. Problem definition and use case identification are managed through Azure DevOps, with models prototyped in Azure Machine Learning Notebooks. Data collection and preparation are handled by Azure Data Factory, Azure Synapse Analytics, and custom scripts. Feature engineering is done with Azure Databricks, and model training uses Azure Machine Learning Service or AutoML. Models are evaluated using Azure ML Metrics and MLFlow. Deployment is managed via Azure Kubernetes Service (AKS) or Azure IoT Edge, with automated pipelines in Azure DevOps.

Monitoring is done with Azure Monitor and Azure Application Insights, while MLOps pipelines handle automated retraining. User interfaces are built using Azure Web Apps or Power Apps, with security and compliance ensured through Azure Active Directory, Key Vault, and Policy.

Both approaches offer flexibility: low-code for rapid development and full-code for more complex, scalable AI development.

Product market fit using both low code and full code

To achieve product-market fit (PMF) for an AI product is not easy.

Considering low code platforms are becoming increasingly powerful, especially assisted by LLMs, one option is to rapidly prototype in low code and then scale to full code.

For the low-code AI product built with Azure's Power Platform, the focus is on business users and SMEs with limited technical expertise. The PMF strategy should prioritize ease of use, integration with tools like Microsoft 365, and quick deployment. Start by identifying the target audience's pain points and launch MVPs to address specific problems, such as form automation or predictive analytics. Collect feedback, iterate quickly, and scale by showcasing success stories and building strong partnerships. Educational content and smooth onboarding are essential for user adoption and growth.

For the full-code AI product using Azure Machine Learning and MLOps, the target is larger enterprises and technical teams that need customizable, scalable AI solutions. The PMF strategy should emphasize flexibility and control, focusing on advanced use cases like predictive maintenance or fraud detection. Launch an MVP, gather feedback from developers and data scientists, and refine the product for enterprise integration. Promote scalability through MLOps, and drive adoption with case studies, enterprise partnerships, and developer engagement. Flexible pricing and long-term contracts will support enterprise adoption.

Other considerations

There are of course other considerations:

• RAG metrics
• GRAPHRAG metrics
• Bench marking processes in organizations.
• Ensuring you are solving the right problem
• Use of LLMs
• Open source -
• On prem options

Currently, we are exploring AI product management solutions with langchain, AWS, llamaindex,, databricks and openAI for enterprises

If you want to study with us , please see our #universityofoxford course on #AI #generativeAI and #mlops . We are also applying these ideas to the erdos community.

Many thanks for Christoffer Noring for his feedback for our work.