Enhancing Transparency in Machine Learning - Enabled Medical Devices: Key Principles and Best Practices

Health Canada, the FDA, and the UK's MHRA collaborated in 2021 to establish ten foundational principles to promote good machine learning practices (GMLP). These principles are crucial for ensuring the development of AI/ML technologies that are safe, effective, and high-quality, capable of learning from real-world applications to enhance device performance.

The agencies also identified key principles focusing on transparency for machine learning-enabled medical devices (MLMDs). This includes ensuring the performance of human-AI interactions and providing users with essential, clear information. Transparency in this context means communicating the device's intended use, development process, performance, and logic to relevant stakeholders. This concept is vital for conveying risks and outcomes effectively, considering user needs, and employing the best strategies for successful communication.

Human-centered design principles are also integral, focusing on a comprehensive user experience and involving stakeholders throughout the design and development phases. This iterative approach enhances transparency, ensuring that MLMDs provide users with all necessary information.

The Guiding Principles

The guiding principles play a critical role in this rapidly evolving field by fostering collaborative efforts, ensuring proper implementation, and enhancing transparency in MLMDs. They address the complexity and iterative nature of machine learning, supporting the safe and effective use of these technologies in healthcare.

The transparency guidelines for MLMDs take into account several key aspects:

• Who – Target Audiences
• Why – Motivation?
• What – Relevant Information
• Where – Distribution of Information
• When – Timing
• How – Approaches to Foster Transparency

Who – Target Audiences

This principle addresses the importance of transparency among the various stakeholders interacting with a medical device. It includes healthcare professionals who use the device, patients who receive its treatment, and other parties like support staff and regulatory authorities involved in decision-making. Transparency is crucial for these groups as it ensures clear communication and understanding, impacting device users such as healthcare professionals, patients, caregivers, recipients of care, and decision-makers, including support staff, administrators, and regulatory bodies.

Why – Motivation

This principle underscores the importance of transparency in machine-learning medical devices (MLMDs) to support patient-centered care and ensure their safety and effectiveness. Transparency involves offering clear and accessible information about the device, including its intended purpose, potential risks, benefits, and any limitations. This clarity aids in evaluating the device's performance, understanding its role in patient care, and ensuring proper usage.

Transparency is essential for several reasons:

• Safety and Effectiveness: It provides a clear understanding of complex and context-sensitive information about the device, ensuring safe and effective use.
• Risk Management: Accessible information supports the identification and assessment of potential risks and benefits, facilitating informed decision-making and risk mitigation.
• Error Detection and Performance Monitoring: It enables the identification of errors and performance issues, allowing for prompt investigation and correction.
• Promoting Health Equity: Transparency in the device's functionality and development helps reveal biases and assess the system's fairness.
• Trust and Adoption: Providing consistent and transparent information fosters trust and confidence in the technology, encouraging its adoption and ensuring fair access to its benefits.

What – Relevant Information

The type of information appropriate for sharing will differ among various machine learning medical devices (MLMDs), depending on the specific benefits, risks, and user requirements associated with each device. It is important to provide information that helps users understand the device and its intended application. A thorough description of a device typically includes:

• Its medical purpose and functionality.
• The diseases or conditions it is designed to treat.
• The intended users, usage environments, and target populations.

Additionally, it is helpful to explain how the device fits into the healthcare workflow. This involves detailing the expected inputs and outputs and describing how these outputs are intended to affect healthcare decisions or the judgments of healthcare professionals.

Providing details on the device’s performance, benefits, and risks is essential for determining its optimal use. When available and understandable, sharing information about the device’s output or the reasoning behind how the MLMD generates its outputs is valuable. This level of transparency enables a thorough evaluation of the device and its outputs during patient care decision-making.

Best practices also include sharing information on product development and risk management throughout the device’s lifecycle. Relevant details should cover:

• The underlying technology and machine learning techniques.
• The characteristics of training and testing data.
• Summaries of clinical studies.
• Regular updates on:

Communicating clinically relevant limitations, information gaps, or contraindications is also crucial. This may involve:

• Noting any biases or potential failure modes.
• Providing confidence intervals for outputs.
• Identifying gaps in data characterization, such as underrepresented patient populations in training or clinical datasets, which may introduce bias.
• Addressing limitations in model development or performance evaluation.
• Highlighting situations where device inputs may differ from the data used for development and validation.

Lastly, it is advantageous to provide information on maintaining the device’s safety and effectiveness throughout its lifecycle, including:

• Methods for local site-specific acceptance testing or validation.
• Ongoing performance monitoring plans.
• Transparent reporting of both successes and failures.
• Change management strategies.
• Proactive approaches to managing vulnerabilities.

Where – Distribution of Information

Device information is available through the user interface, which includes all components with which users interact, such as visual, auditory, and tactile elements. This interface encompasses training materials, physical controls, display features, packaging, labeling, and alarms. It is crucial to refine the software user interface to ensure that the information provided is responsive and meets the needs of the user. The interface can be customized, adaptive, and interactive, employing various modalities like audio, video, on-screen text, alerts, diagrams, software safeguards, and document libraries to effectively cater to user requirements.

When – Timing

Achieving effective transparency in the use of medical devices requires addressing information needs throughout the entire product lifecycle. Detailed information about the device is essential both for deciding whether to acquire or implement it and for understanding its correct usage. It is also important to provide timely updates about any changes, modifications, or new information. Offering targeted information, such as on-screen instructions or warnings, can be especially useful during specific stages in the workflow, particularly during high-risk procedures or when dealing with particular input or output features. This aspect of the guiding principles highlights the importance of timely communication, helping stakeholders make well-informed decisions about adopting and using the MLMD, and ensuring users have access to relevant information throughout the product’s lifecycle.

How – Approach to Foster Transparency

Effectively communicating information about MLMDs requires a thorough understanding of users, their environments, and workflows, which can be achieved through applying human-centered design principles. This approach focuses on the entire user experience and includes responsive and iterative design, validation, monitoring, and communication. For example, delivering information with the appropriate level of detail for the target audience, organizing content logically, and using either plain language or technical terms as needed can enhance accessibility and usability. Applying human-centered design principles improves transparency by tailoring information to meet the needs and comprehension levels of various users, thereby supporting more informed decision-making and better usability.

Conclusion

Maintaining transparency over time supports ongoing safety and effectiveness, fostering trust and facilitating the efficient use of MLMDs. These guiding principles serve as a foundation for adopting and refining effective transparency practices in the field of MLMDs. Ongoing engagement on this subject is crucial for the collaborative development, implementation, and continuous improvement of transparency standards and practices, especially as this area rapidly evolves.

RookQS Support for MLMDs

Rook Quality Systems can support ML-enabled medical devices by providing expert guidance on regulatory compliance, ensuring that machine learning models meet stringent safety and performance standards. Their services include developing robust validation frameworks and facilitating transparency practices, which help optimize device performance and support informed decision-making throughout the product lifecycle.

References:

Transparency for Machine Learning-Enabled Medical Devices: Guiding Principles | FDA

Good Machine Learning Practice for Medical Device Development: Guiding Principles | FDA