AI Products Like ChatGPT Can Provide Medics with ‘Harmful’ Advice – Should We Be Worried?

A recent study in The European Journal of Clinical Nutrition warns that while AI technology offers tremendous potential, it could pose significant risks to patient health, especially when managing critical conditions like diabetes and metabolic abnormalities.

Whilst AI is increasingly relied upon in many industries, healthcare professionals must stay alert to the potential dangers of depending solely on these tools. This study emphasises that AI can support personalised education but lacks the precision required for safe medical guidance.

The Rising Role of AI in Healthcare

There is no doubt that AI is revolutionising various sectors, and healthcare is no exception. AI platforms, like OpenAI’s ChatGPT, are being integrated into medical applications to offer insights, advice, and real-time feedback. These tools, designed to assist healthcare professionals, promise to streamline workflows and enhance patient care.

For example, healthcare providers might use AI chatbots to suggest meal plans, manage chronic conditions, or answer patients' frequently asked questions.

The Alarming Findings from the Latest Study

A recent study by a team of researchers led by Professor Farah Naja from the University of Sharjah has raised red flags about AI’s role in offering medical advice. The research analysed ChatGPT’s performance in managing diseases such as Type 2 Diabetes and Metabolic Syndrome. What they found was unsettling: ChatGPT missed critical recommendations, such as weight loss and proper nutrient guidance, essential for effective disease management.

The team fed 63 prompts into ChatGPT’s system, covering three major domains: dietary management, nutrition care processes, and menu planning. Unfortunately, the chatbot’s advice was often incomplete or outright incorrect, lacking specific guidance that aligns with the American Diabetes Association and other medical guidelines.

Why These Findings Matter to Healthcare Providers

Accurate and complete information is crucial for healthcare professionals, particularly those managing chronic conditions like diabetes. Weight loss, balanced diets, and lifestyle changes are critical to managing metabolic conditions, yet ChatGPT failed to provide the necessary guidance.

The study found that:

• Weight loss recommendations - a key component in managing diabetes and metabolic disorders - were missed entirely.
• ChatGPT’s dietary recommendations lacked balance, failing to meet calorie, carbohydrate, and nutrient needs like calcium and Vitamin D.
• Advice on increasing fibre intake or using whole grains—vital for those with diabetes - was often overlooked.

This underscores that while AI can simulate a conversation, it does not have the clinical expertise required to make life-saving recommendations. Healthcare professionals should remain the final authority in providing accurate medical guidance.

The Growing Prevalence of AI in Medical Advice

Interestingly, despite their limitations, ChatGPT and other AI tools have found their way into various health-related applications. Whether integrated into fitness apps or public health campaigns, these AI systems are marketed as accessible, user-friendly ways for people to manage their health. ChatGPT, for instance, is praised for its dynamic conversational abilities, often tailored to personalised education.

However, as the study points out, there is limited research on how effective these AI-driven solutions are, especially regarding clinical nutrition and chronic disease management.

The Pitfalls of Relying Solely on AI in Healthcare

The researchers argue that AI systems should not be seen as substitutes for healthcare professionals. Relying entirely on AI poses significant risks, particularly in complex medical fields where precision and personalisation are critical.

The research led by Professor Naja makes it clear: AI tools like ChatGPT should be used with caution, especially in the management of chronic diseases. Healthcare professionals must remain aware of AI's limitations and integrate their expertise into AI-generated recommendations. This balance is crucial for ensuring patient safety.

Why the Human Element in Healthcare is Irreplaceable

AI chatbots react to prompts in a way that mimics human conversation, but they lack the critical judgment that trained dietitians and medical professionals bring. They cannot factor in nuances, long-term care plans, or complex patient histories as a human can.

Professor Naja stated, "ChatGPT could provide incorrect, incomplete, or harmful advice, jeopardising the quality of medical care and consequently patients' health and safety." This should serve as a stark reminder to healthcare professionals: AI is not a replacement for human expertise.

What This Means for the Future of AI in Medicine

The potential of AI in healthcare is undeniable. From streamlining administrative tasks to providing real-time feedback in specific non-critical scenarios, AI holds a promising future. However, as this study reveals, the technology has a long way to go before it can be fully trusted in clinical advice.

Healthcare professionals must continue to be the primary gatekeepers of patient care. AI should serve as a tool, not a replacement. The dangers of relying too much on AI—particularly when the advice given could directly affect patient outcomes—are far too great to ignore.

The rise of AI tools like ChatGPT in healthcare offers opportunities and challenges. While these platforms can support professionals with personalised education, they are not yet refined enough to handle complex medical conditions without human oversight. The findings from this study should be a wake-up call for healthcare providers: AI has its limitations, and its role in patient care must be carefully managed.

The advice is clear for now: Use AI as a supplement, not a substitute. Healthcare professionals should lead the way in delivering safe, effective treatment.