Generative AI in Healthcare: Navigating Global Regulation and Future Perspectives

Introduction??

Artificial Intelligence (AI) is making significant strides in healthcare, creating new avenues for improving patient care, refining diagnostics, and advancing medical research. Since the release of ChatGPT in November 2022, the medical and research communities have shown considerable interest, with over 4,688 articles published in PubMed referencing “ChatGPT” (Figure 1). This trend underscores the growing focus on understanding AI’s role and potential in modern healthcare.

?Figure 1: Search Result in PubMed for the term “ChatGPT”?

Top-tier academic journals like the New England Journal of Medicine (NEJM AI) and JAMA (JAMA + AI) have responded by publishing special issues dedicated to AI and its clinical applications, signifying AI’s pivotal role in advancing healthcare practices (Figure 2). These volumes focus on integrating AI into clinical settings, covering topics such as diagnostics, patient monitoring, and decision support systems.

Figure 2: Special AI Editions in Top Medical Journals (NEJM AI, JAMA + AI)?

?

While rapid innovation drives advancements in healthcare, it also introduces the complex challenge of navigating diverse global regulatory landscapes. In our first edition of this newsletter, AI in Healthcare: Navigating Global Regulatory Definitions and Landscapes, we highlighted that the FDA has authorized over 985 AI/ML-enabled medical devices as of September 2024. However, none of these approvals include devices powered by generative AI models. In this article, we'll explore the regulatory frameworks for generative AI and large language models (LLMs) in healthcare, highlighting key opportunities, and challenges, with a glance to the future.?

Product/Technology Definition ?

Generative AI technologies in healthcare encompass advanced software systems capable of producing new data outputs—whether images, text, recommendations or predictive models—by learning from vast quantities of pre-existing healthcare data. Applications for this technology are increasingly diverse, including enhancing medical imaging analysis, personalizing patient care, and supporting clinical decision-making. Let's look at some of the key terms and how the regulators may perceive them:?

Key Regulatory Definitions and Considerations??

Generative AI: Defined as AI technologies capable of creating new content or insights based on existing data patterns. In healthcare, generative AI is being developed to enhance diagnostics, automate patient data processing, and improve patient engagement.?

?Large Language Models (LLMs): LLMs are a class of deep learning models designed to process, understand, and generate human language based on vast amounts of text data. These models utilize transformer architectures, which allow them to capture contextual relationships between words and generate coherent text. In healthcare, LLMs are used for tasks such as synthesizing medical literature, extracting relevant information from patient records, and providing conversational interfaces for healthcare providers and patients.?

?Machine Learning (ML): A subset of AI involving algorithms that learn from data to make predictions or decisions.??

?Retrieval-Augmented Generation (RAG): A hybrid model combining retrieval mechanisms with generative capabilities, RAG systems retrieve updated data from external sources before generating responses. This combination is invaluable in healthcare settings where information must be up-to-date, such as for recent research findings. RAG also helps reduce errors by ensuring that generated content is based on the latest data, while the historical data used remains locked and controlled to maintain consistency.?

?Locked vs. Open Algorithms: A significant regulatory focus is on the difference between locked and open algorithms. Locked algorithms are fixed and do not change post-deployment; any updates require regulatory re-evaluation. Open algorithms, on the other hand, continue to learn and adapt post-deployment, creating challenges in standard regulatory evaluation. To date, no open algorithms have been approved by the FDA. As LLMs and RAGs might be considered similar to an "open algorithm," they too may struggle to receive approval due to their adaptive nature.?

Regulatory Landscape?

Currently, regulatory authorities like the FDA are working on defining and establishing guidelines for generative AI in healthcare, though there is no definitive regulation yet. To address this, the FDA has released preliminary documents and hosted public discussions aimed at understanding how best to incorporate AI and machine learning in healthcare applications. One critical challenge regulators face is that traditional frameworks don’t align well with AI’s adaptive, data-driven nature, especially for models that evolve over time. This limitation has prompted the FDA to explore adaptable guidelines that account for the variability and continuous improvement of LLMs.?

?FDA: United States?

Recently, the FDA has published an executive summary” Total Product Lifecycle Considerations for Generative AI Enabled Devices” for the upcoming Digital Health Advisory Committee Meeting. This summary emphasizes the need for a Total Product Lifecycle (TPLC) approach (Figure 3) to managing generative AI in healthcare. The TPLC framework recognizes that AI-enabled devices must be continuously monitored, tested, and updated throughout their lifecycle to ensure they remain safe and effective as they evolve.?

Figure 3: FDA's Total Product Lifecycle (TPLC) Framework for Generative AI in Healthcare"?

?

Continuing from this foundational work, the FDA is shaping a future regulatory approach for generative AI in healthcare that focuses on five key areas:?

1. Lifecycle Management (TPLC): The FDA's Total Product Lifecycle (TPLC) framework emphasizes the need for ongoing monitoring and updates to generative AI devices, ensuring that safety and efficacy are maintained as models evolve over time.?
2. Risk-Based and Adaptive Regulation: While maintaining a risk-based approach, the FDA is exploring adaptive regulatory controls specific to generative AI’s unique challenges, like output variability and frequent model updates.?
3. Transparency and Explainability: To build user trust, the FDA may require manufacturers to provide clear documentation on model design, data sources, and limitations. This transparency helps users and regulators better understand how the AI functions and where its outputs might fall short.?
4. Enhanced Postmarket Monitoring: Given the dynamic nature of generative AI, the FDA envisions robust postmarket surveillance to track real-world device performance, promptly address any emerging risks, and ensure continued safe use.?
5. Governance and Feedback Mechanisms: The FDA is also prioritizing governance structures and feedback systems that adapt to regional and demographic differences, promoting a balance between innovation and patient safety.?

These strategies reflect the FDA’s commitment to fostering responsible AI innovation in healthcare by proactively addressing both current and emerging risks associated with generative AI technologies.?

EMA – Europe?

Recently, the EMA published its Guiding Principles on the Use of Large Language Models in Regulatory Science and Medicines Regulatory Activities (Figure 4), outlining key considerations for the safe, ethical, and effective integration of generative AI in healthcare. These principles emphasize the importance of responsible governance, transparency, and proactive risk management, aiming to ensure that large language models (LLMs) support regulatory science securely and competently.?

Figure 4: Guiding Principles on the Use of Large Language Models in Regulatory Science by the European Medicines Agency (EMA) and Heads of Medicines Agencies (HMA)??

?Continuing from this foundational work, the EMA’s approach to regulating generative AI in healthcare focuses on four core areas:?

1. Lifecycle Governance and Risk Management: The EMA stresses that there should be clear guidelines for using AI responsibly throughout its entire lifecycle—from initial use to ongoing updates. This means defining how generative AI models should be used, training staff to handle them responsibly, and consistently monitoring them to prevent potential risks.?
2. Data Protection and Privacy: Protecting personal data is a top priority for the EMA. Since AI models are often trained on large amounts of public data, they may unintentionally include sensitive or private information. The EMA recommends strict data controls and careful design of AI prompts to ensure compliance with privacy laws, especially GDPR, which safeguards personal data in the EU.?
3. Transparency and Ethical Standards: The EMA promotes transparency to build user trust. This means that AI tools should clearly explain how they operate, their sources of information, and any limitations in their outputs. Ethical standards, like preventing bias and misinformation, are essential to ensure AI outputs are reliable and do not mislead users—especially critical in healthcare.?
4. Postmarket Surveillance and Feedback: EMA emphasizes the need for ongoing monitoring of AI tools even after they are launched. Regulatory bodies are encouraged to collect feedback on AI performance, fix any issues quickly, and keep AI tools up to date to ensure they continue to work accurately and safely.?

These principles reflect the EMA’s commitment to ensuring that generative AI in healthcare is used responsibly, remains secure, and always prioritizes patient safety.?

Opportunities and Challenges?

Large Language Models (LLMs) encounter substantial challenges with "hallucinations"—instances where the model generates outputs that appear credible but are factually incorrect or entirely fabricated. This issue is widespread and complex, with significant implications for the reliability and trustworthiness of AI-generated information. Studies indicate that hallucinations are not uncommon; for instance, the rate of hallucinations in models like ChatGPT is estimated to range from 15% to 20%. These hallucinations can be particularly dangerous in healthcare settings, where incorrect information could lead to inappropriate medical advice or decisions, potentially endangering patient safety. This underlines the need for rigorous testing, validation, and human oversight to ensure the reliability of AI-generated outputs.?

Case Study: FDA’s Exploration into Generative AI Applications in Healthcare?

The FDA is actively researching generative AI to understand its potential applications and address associated risks before formal approvals. These exploratory projects are helping the FDA set the groundwork for future regulatory frameworks by testing generative AI models in controlled environments and analyzing their effectiveness. Here are two significant FDA initiatives that illustrate the agency’s forward-thinking approach to integrating generative AI into healthcare research.?

1. FDA’s AnimalGAN Project?

The AnimalGAN Project is an FDA initiative that uses generative AI to create virtual animal models for drug safety testing (Figure 5). By generating simulated animal data, AnimalGAN offers a potential alternative to physical animal testing, aiming to make toxicology research faster, more cost-effective, and more ethical. This project demonstrates how generative AI could reduce reliance on live animal models, enabling the FDA to assess drug safety with fewer ethical concerns and a streamlined process.?

Figure 5: "FDA’s AnimalGAN Project: Virtual Animal Models for Drug Safety Testing"?

2. AskFDALabel Project?

The AskFDALabel Project leverages large language models (LLMs) to improve the way the FDA detects and classifies adverse events (AEs) in drug labeling documents( Figure 6). AEs, associated with over 70,000 deaths annually in the U.S., are critical to monitor, but traditional AE classification is labor-intensive and time-consuming. AskFDALabel applies AI to automate and enhance AE classification, achieving impressive accuracy rates (over 98% F1-score) in detecting toxicity and cardiotoxicity and significantly improving AE profiling with an F1-score of 0.911. This project highlights how generative AI can boost efficiency and accuracy in the FDA’s drug safety monitoring processes.?

Figure 6: AskFDALabel Project: Enhancing Adverse Event Detection with AI?

Together, these initiatives underscore the FDA’s commitment to exploring generative AI’s capabilities and preparing for its safe and responsible integration into healthcare. By proactively investigating these applications, the FDA is building a framework that supports innovation while prioritizing patient safety and ethical standards.?

?

Legal and Ethical Considerations??

The integration of generative AI and Large Language Models (LLMs) in healthcare brings complex legal and ethical responsibilities. Compliance with data privacy regulations such as GDPR in Europe and HIPAA in the United States is essential to protect sensitive patient information and maintain public trust. Ethical considerations, including algorithmic bias, transparency in decision-making, and accountability, are increasingly central to AI development in healthcare. Regulatory bodies emphasize the importance of ethical AI, encouraging developers to implement safeguards that minimize bias, enhance transparency, and ensure that AI systems can clearly explain their decision-making processes. These measures are critical to maintaining trust in AI-powered healthcare and ensuring that innovations are both safe and equitable.?

Conclusion??

Generative AI in healthcare is evolving rapidly, with significant potential to improve patient outcomes and transform medical practice. Large Language Models (LLMs) add another dimension, offering advanced capabilities in data synthesis and patient engagement but also presenting unique regulatory challenges. To succeed, companies must understand and adapt to the specific regulatory demands of each market, ensuring compliance without stifling innovation. The FDA emphasizes a Total Product Lifecycle (TPLC) approach, adaptive regulation, and robust postmarket monitoring to promote safety and transparency as AI technologies evolve in healthcare. In parallel, the EMA focuses on governance and risk management, data protection, and ethical standards, promoting collaborative knowledge-sharing across Europe to support consistent, trustworthy AI applications. The path forward will require collaboration among industry, regulators, and stakeholders to create frameworks that foster both technological advancement and patient safety.?

Stay tuned for upcoming editions of 'Innovation Meets Regulation,' where we explore the intersection of healthcare innovation and regulatory frameworks that shape the industry's future.

?