Why Might LLMs Produce Unexpected Outputs?

Large Language Models, like other machine learning models, rely on pattern discovery and association learning between patterns and features. This learning hinges on the regularities in the training data. For instance, if two English words, such as "I" and "am", frequently appear consecutively, an LLM forms an association between them, mistaking this correlation for a deeper understanding. This phenomenon can be likened to the issue of causation versus correlation.

Consider the correlation between the number of swimming pool drownings and the sales of barbecue grills. Data might show that as the sales of barbecue grills increase, so do the incidents of swimming pool drownings. However, it's not that buying a barbecue grill directly causes more drownings. Instead, both these events are more common in the summer months when the weather is warm. People are more likely to host outdoor barbecues and go swimming during this time. If we only observe barbecue grill sales and swimming pool drownings, a misleading correlation emerges, suggesting a direct relationship where none truly exists.

This is the reason why prompt engineering and setting the scene or context is important when dealing with LLMs. Giving more information, like the temperature, to the swimming pool drownings and barbecue grill scenario would help models to avoid building false associations. This is also crucial for the fine-tuning and training of LLMs. If training data is incomplete (contains only the data of barbecue grill sales and swimming pool drownings), the resultant LLM would be more prone to hallucinate compared to training data that includes additional context, like temperature data. By providing a more comprehensive dataset, we can guide the model to form more accurate and meaningful associations, reducing the risk of misleading outputs.