Towards Responsible AI in Healthcare: Insights from the AMIA Panel Discussion

?

The rapid integration of artificial intelligence (AI) in healthcare has created a critical need for responsible AI governance that prioritizes transparency, accountability, and patient involvement. At the recent AMIA Informatics Summit in Boston, I had the privilege of moderating a panel discussion titled "AMIA Panel on AI in Healthcare—Towards Responsible AI in Healthcare." The panel brought together experts from various disciplines to discuss the challenges and opportunities in developing and implementing AI in healthcare.

?

The panel discussed the full range of translational trajectories from basic research through design, development, validation, implementation, and maintenance to discuss actionable next steps for the governance of implementation in AI. This panel session resulted from the "Blueprints for Trust: ?Best Practices and Regulatory Pathways for Ethical AI in Healthcare" Conference co-organized by AMIA and the Division of Clinical Informatics (DCI) at BIDMC, and subsequent working group activities to guide AI in healthcare focused on multi-stakeholder perspectives to ensure AI in healthcare is safe, effective, equitable, and trustworthy.

?

The regulatory climates and policy issues related to AI in healthcare were discussed, focusing on responsible AI governance. Speakers emphasized transparency, accountability, and continuous learning to ensure responsible AI governance. Ethical considerations in AI development were also highlighted, with concerns about the potential consequences of neglecting these considerations.

?

Brian Scarpelli , the executive director of the Connected Health Initiative, provided valuable insights into the current regulatory landscape for AI in healthcare. He highlighted the FDA's collaborative approach in developing guidance for AI-based medical devices, the Centers for Medicare and Medicaid Services (CMS) efforts to incorporate AI support for clinical decision-making, and the Office of the National Coordinator for Health IT's (ONC) transparency requirements for AI in certified EHR technology. Scarpelli also discussed the challenges posed by the Department of Health and Human Services (HHS) Office of Civil Rights proposed rule that places maximum liability on healthcare providers for discriminatory outcomes, emphasizing the need for a shared responsibility framework to address potential biases and risks in AI deployments.

?

One key takeaway from the discussion was the importance of involving a wide range of stakeholders, including healthcare professionals, policymakers, AI developers, patients, and patient advocates, in developing and evaluating AI systems. The DCI Network has intentionally brought a broad range of stakeholders, with patients playing a central role. Dave deBronkart , a patient advocate and panelist, aptly quoted the late Dr. Warner Slack, "The patient is the most underused resource in healthcare." This sentiment was expressed initially by Warner Slack, the DCI-cofounder and a pioneer in medical informatics, in 1976 at the Third Annual "Advances in Patient Care" Conference and published in 1978 ("The basis for our use of the computer in medicine is the thesis that the largest and least utilized provider of health care is the patient."). This view still holds true today and highlights the need for patient-centric solutions in AI governance.

Dave deBronkart, a patient advocate and co-founder of the Society for Participatory Medicine, passionately emphasized the importance of involving patients as active partners in developing and using AI in healthcare. Drawing from personal experience as a stage four kidney cancer survivor, deBronkart highlighted the critical role that patient engagement played in his journey. He cited the Andreessen Horowitz map of AI in healthcare, showcasing the vast potential for AI applications across clinical and non-clinical domains. He argued that patients, as end-users of AI, must be included in the decision-making process to ensure their needs are met. deBronkart shared several compelling use cases, such as a patient using ChatGPT to help diagnose a rare condition, a patient leveraging AI to summarize medical notes and action items, and another using AI to advocate for ambulatory blood pressure monitoring. He stressed that patients have a vital role in helping healthcare achieve its full potential and that AI can empower patients to be better partners with their healthcare providers. Ultimately, deBronkart called for the inclusion of patients in the governance decisions surrounding AI in healthcare to ensure that the technology serves the needs of all stakeholders.

The panel discussion also highlighted the need for educational initiatives to help clinicians and patients navigate the uncertainty of AI-provided medical advice. Amy Price MS, MA, MS, DPhil, from the Dartmouth Institute for Health Policy and Clinical Practice, emphasized the importance of patient empowerment and transparency in healthcare, particularly in developing disclosure labels. She emphasized the importance of building AI policies on values, allowing room for adaptation and unforeseen uses while focusing on the intended goals. She highlighted the need for continuous training and updating AI models based on the input they receive, drawing an analogy to the tiger beetle's ability to adapt to its surroundings for survival advantage.

Price stressed the importance of involving all stakeholders, including patients, educators, and industry, from the beginning of the AI development process to ensure a common focus and shared goals. She also raised concerns about the potential for misunderstandings and miscommunication when perspectives differ, as illustrated by the optical illusions she presented. Additionally, Price advocated for establishing a Health AI Consumer Consortium to promote safe, effective, equitable, and trustworthy AI in healthcare, with patient and public leaders having voting rights, education, and empowerment to shape best practices and governance.

The panelists emphasized the need for transparency in AI development, particularly regarding data usage and algorithmic decision-making. Another critical aspect discussed was the role of informatics professionals in creating solutions that have a sound methodology, ethics, values, and patient-centricity. As AI continues to evolve and integrate into various aspects of healthcare, it is crucial to leverage the expertise of informatics professionals to ensure that AI systems are developed and implemented responsibly.

Eileen Koski , from IBM Research, stressed the importance of understanding the context and nuances of the data used to train AI models. She highlighted the challenges of working with real-world data, including identifying the right questions and the potential for data to be "answer adjacent" rather than directly answering the desired question. Koski also emphasized the importance of characterizing unusual patterns in the data, such as skews towards wealthier individuals or specific age groups and considering factors like sample collection days and seasonality when analyzing data. Additionally, she advocated for more education and training on data analysis and interpretation, particularly in the field of public health, to ensure that stakeholders can effectively leverage complex data for AI development and decision-making.

?

Leon Rozenblit , from the Q.E.D. Institute suggested that there needs to be a standardized way to specify the training data, similar to how genetics has a minimum data set for specifying genetic information. He emphasized the importance of transparency in machine learning, particularly regarding training data. He proposed a risk-based approach to AI policy, considering factors like data sharing and patient impact.

Furthermore, Rozenblit argued that large firms often lobby for regulation to protect their interests, which can lead to regulatory capture and prioritize industry over other stakeholders. He noted that burdensome regulation can increase the risk of regulatory capture, citing examples such as the railroads in the 1800s and the energy industry today. To address these concerns, Rozenblit proposed the establishment of a multi-stakeholder consortium, the Health AI Consumer Consortium, to represent patient interests and provide a mechanism for continuous learning on AI governance.

As a call to action, I invite healthcare professionals, policymakers, AI developers, patients, and patient advocates to join forces and create an alliance of organizations and stakeholders to generate patient-centric solutions to the issues raised in the panel discussion. The Division of Clinical Informatics (DCI) Network and AMIA are committed to addressing these problems in a patient-centric way, and we welcome your involvement and contributions.

?

Together, we can work towards responsible AI governance in healthcare that prioritizes transparency, accountability, and patient involvement. Let us leverage the expertise of informatics professionals and the invaluable insights of patients to create AI systems that truly benefit all stakeholders in healthcare. I invite you to register for DCI Network (at no cost) to participate in our future webinars and view the past recordings of our conferences. We will soon announce our next DCI conferences (June 2024 and September 2024), and AI initiatives, and you will get notified via the DCI Network newsletter for how to join.

We invite you to attend AMIA's 2024 Annual Symposium in San Francisco and join the informatics community.

?

Please share your thoughts, experiences, and ideas in the comments below for your thoughts on how we can make AI safe, effective, equitable and trustworthy. Let's continue this critical conversation and collaborate to shape the future of AI in healthcare.

?

Yuri Quintana, Ph.D.

Chief of the Division of Clinical Informatics, Beth Israel Deaconess Medical Center

and Assistant Professor of Medicine, Harvard Medical School, Boston, MA, USA.