AI Meets Policy: What ChatGPT Thinks of the FDA’s LDT-rulemaking?

In the last edition of my newsletter, I introduced a super useful bulk data download tool and shared insights on how to efficiently view all public comments on FDA LDT-rulemaking. For those of us diagnostics regulatory policy geeks, the challenge lies in navigating through the overwhelming 6,707 public comments on regulations.gov. Recognizing the limitations of human capacity, I turned to AI ?to help process this vast amount of text. I'll begin by sharing the output and then delve into some interesting aspects of the process.

AI Output:

I sought to address the following three questions:

One: What forms do the comments take?

Remarkably, about 80% of the comments originate from form letters (pre-written templates). Within these, eight distinct types were identified, with only one (FL-S) supporting the policy, while the remaining seven (FL-A #1~#7) opposed it. These form letters are likely drafted by professional associations and submitted by their members. In the rest 20%, 914 comments were relatively unique (non-form, appearing less than six times with less than 80%-word repetition), representing individual inputs or significantly modified form letters. Another 443 comments, lacking textual content, included attachments that likely presented more detailed opinions or positions. Figure 1 offers a visual breakdown of these comment categories.

Two: What is the overall public opinion towards the policy?

Contrary to what one might expect, the opposition isn't as dominant as anticipated. An astonishing 2,849 supportive comments were derived from a single form letter, showcasing remarkable mobilization within this niche (can I say marginalized?) industry. Figure 2 depicts the balance between opposing and not-opposing opinions. For classification purposes, comments not explicitly opposing the policy were considered not-against. I will explain it more below in the methodology.

Three: Where do individual comments lean?

Excluding form letters and attachments attributed to companies, organizations, professional societies or trade groups, we zoom in on the 914 individual comments. This closer examination, categorized into against and not-against, is visualized in Figure 3.

Methodology:

It took quite a few tries to get to a satisfactory ChatGPT prompt.

First, I tried the most routine analysis for social media comments, known as sentiment analysis. The output was 95% or more positive, with few negatives, obviously far from the truth. Looking under the hood, I realized the problem. Sentiment analysis is defined as the process of analyzing digital text to determine if the emotional tone of the message is positive, negative, or neutral. So, it's not hard to guess that the majority of our commenters are so decent that even when voicing the strongest opposition, their tone is actually quite positive! This approach wasn't going to work. We are too polite.

Then, I asked ChatGPT to perform an opinion analysis for supporting, opposing, and neutral stances toward the policy. The output was still way off because the AI tended to focus on the wording rather than the overall attitude. For example, if the beginning of a comment included wording like "appreciate the FDA’s effort for safeguarding public safety," the comment was likely to be categorized as supportive, even though it continued with "however, I am deeply concerned about the accessibility of the tests…" Again, we are too polite.

Finally, I asked the AI to "summarize each comment, extract key points, identify the main ideas or arguments, and concisely provide them" and then perform the opinion analysis for supportive/against/neutral. This time, the output was closer to my feelings of going through large numbers of comments quickly. But ChatGPT informed me that the task was too burdensome for it; it could only manage a couple of dozen comments for me, suggesting I would need a team to systematically tackle the 6,707 comments. Fortunately, my friend Alex Wan Shanyue (Alex) Wan (thank you, Alex!) lent a hand and used the ChatGPT API to feed it slowly. No more complaints.

Alex's take on duplication removal:

Duplication of public comments can influence the accuracy of analysis results, posing a challenge in obtaining a full picture of public opinion. To address this, we relied on a cosine similarity comparison algorithm to detect similar comments, successfully reducing the total number of data samples from 6,707 to approximately 900. By doing so, the key points retrieved from AI models become more concise and trustworthy.

Disclaimer:

This is just my personal interest using natural language processing AI to analyze public health policy comments. The outcomes and methods are not verified or authenticated. The process was enjoyable (so my desire to share with you), but the conclusions are far from reliable. For serious reporting, check GenomeWeb article here.