The problem with ChatGPT in clinical settings

Please allow for a long preamble before I make my point on my concerns with ChatGPT's applications in healthcare and improving the productivity of a doctor.

Clinical Trials and Historical Bias

It is well documented that there is a significant problem of bias in clinical trials, but it is not discussed as widely as it should be. Women face the worst of it along with people of color who are systematically excluded from many clinical trials which has lead to harmful medical treatments for these sub-populations. For example, while women make up 50.8 percent of the U.S. populace, they make up only 41.2 percent of the clinical trial participants. As for people of colour, one study published in the Journal of General Internal Medicine found that Black Americans are significantly underrepresented in clinical trials for cancer drugs, with only 8% of participants being Black, despite Black Americans accounting for 13.4% of the U.S. population. Another study published in the New England Journal of Medicine found that less than 5% of participants in cardiovascular clinical trials are Black, despite Black Americans having a higher risk of developing cardiovascular disease.

Women of color face the worst of it - black women have a 41% higher mortality rate from breast cancer compared with white women but they represent only 5% of clinical trial participants.

A key reason for this is that the inclusion criteria for many trials are based on the characteristics of white men, who are often used as the default population. For example, a clinical trial may require participants to be a certain height or weight, or to have certain levels of certain biomarkers, which may exclude women and people of color who do not meet these criteria. Another reason is that women and people of color are often excluded from clinical trials due to historical injustices and discrimination in healthcare.

A striking example is the infamous Tuskegee syphilis experiment, which took place between 1932 and 1972 where involved withholding treatment from black men with syphilis to study the natural progression of the disease. From the 600 black men in the trial, 399 with syphilis were left untreated and not informed about their condition. The impact of the study on the health of black Americans was devastating. The participants in the study were denied treatment for syphilis, even after effective treatments became available in the 1940s. Many of the men experienced severe health complications, including blindness, paralysis, and painful skin lesions. Some of the men unknowingly passed the disease on to their partners and children, leading to?suffering.?

The problem with ChatGPT

In the near future, we will undoubtedly see ChatGPT and GPT in general being used in streamlining administrative tasks, training doctors, helping with diagnostics and in general making healthcare more efficient.

But caution will be necessary when it comes to clinical applications. GPT is known to generate inaccurate completely false outputs. We should understand that despite studying "ethics" and "morals" since the ancient times, the above injustices during clinical trials are still being carried out - and this is the information GPT is using to base its clinical model on - essentially on the physiology of a white male. So I am not sure if medicines commonly used by women will be taken into consideration when doing a differential diagnosis, for example. Add to that that its love for misinformation as it devours all the information on the internet. It will treat pre-prints on arXiv just as it treats say an article from BMJ. Know that the results in pre-prints haven't been peer reviewed much less duplicated, or misanalysed.

And when you account for "scientific sounding" comments on your favorite subreddit, a 13-year old's' cleverly worded glided comment has a odd chance of defining and fine-tuning your "personalized medicine of the future" - essentially misinformation on cocaine.

With AI systems here to stay, what is needed is some sort of a regulation or a certain mandates which should be adhered to when training an AI system used for clinical purposes. These can be used to offsetting the internet's bias towards misinformation as well as pre-prints or invalidated and old trial results. How that regulation should look like or how it should be implemented I have no idea on. But knowing FDA's slowness to shed light on the grey areas of regulations, especially when it comes to "digital", we would be better served with some self-imposed guardrails by AI companies for now.