Harnessing Generative AI for Automated News Report Generation: Insights from My Master's Thesis (1/3)

For the past 8 months, I've been immersed in the world of Generative AI, specifically working with ChatGPT 3.5. My research aimed to harness its state-of-the-art capabilities for automating news report generation through sophisticated data analysis. Along this journey, I've uncovered several fascinating insights and limitations of ChatGPT that I am thrilled to share with you.

Objective and Limitations of ChatGPT

?? Objective: The primary goal of my thesis was to explore the potential of ChatGPT 3.5 in generating coherent and informative news reports based on automated data analysis. The ambition was to leverage its advanced natural language processing abilities to create accurate, timely, and contextually relevant news articles.

?? Limitations: Despite its groundbreaking abilities, ChatGPT exhibits some limitations. Here are three of the most intriguing patterns we observed:

• Pandemic Effect: ChatGPT tends to incorporate pandemic-related content into its outputs due to its training on a vast amount of data from the pandemic era. For example, when tasked with explaining the rise in vehicle prices in Argentina, the model attributed the cause to the pandemic, supply chain disruptions, and inflation, even without period-specific data.

Example Output:

"This increase represents a significant challenge for consumers who are already struggling with the economic impact of the pandemic. The rise in car prices is a result of several factors, including the devaluation of the Argentine peso, inflation, and the increase in production costs. The automotive industry has been hit hard by the pandemic, with supply chain disruptions and reduced demand leading to..."

• Logical Fallacies: GPT is a statistical NLP model meaning it uses the process of predicting the next word in the sequence given the words that precede it. Such models have their limitation when it comes to simple mathematical operations. We performed explicit data cleaning and pre-processing before providing the dataset to the model. Even when provided with explicit percentage change values, ChatGPT sometimes generated outputs with logical inconsistencies. For instance, in discussing the price increase of a models, the model incorrectly called it a price drop!!

Example Output:

N**** J**** Prices Drop by up to 14.5% with New Release

N**** J**** prices have dropped significantly with the release of the 2023 model, according to the latest data. The new model is priced at an average of 12% lower than the previous model, with some configurations seeing a drop of up to 14.5%. The biggest price difference is seen in the *** model, which has dropped from $17,590 to $20,135, a decrease of 14.5%. …

• Contextual Ambiguity and Anchoring Bias: ChatGPT sometimes misinterprets contextual cues, leading to ambiguous or biased outputs. For example, the phrase “version is being phased out, replaced by newer version, temporarily on sale” was often misinterpreted to mean the product is available at a reduced price, rather than just being available temporarily.

Anchoring bias is a cognitive bias that causes individuals to rely too heavily on the first piece of information they receive (the "anchor") when making decisions. This initial information serves as a reference point and can significantly influence subsequent judgments and decisions, even if the anchor is unrelated or irrelevant to the decision at hand.

We observed that anchoring bias triggered irrational and illogical comments, such as interpreting a price increase of 14.5% as a price drop, due to being anchored by the phrase "Temporary on sale." This phrase led the model to focus on the idea of a sale, causing it to overlook the actual data. Interestingly, when we rephrased the statement to "Temporarily available for purchase," the model corrected itself and produced the accurate output, demonstrating how subtle changes in wording can significantly impact the model's interpretation and response.

In the next article I will share the solution we developed that minimizes the risk of errors and how we can use AI to supervise AI.