LLMs and AI: As Smart as a 7th Grader?

Large Language Models (LLMs) like OpenAI’s ChatGPT have become a focal point of discussions around artificial intelligence (AI) and its capabilities. Interestingly, these models have shown a proficiency comparable to the literacy levels of a seventh grader. This comparison sheds light on AI’s capabilities and limitations in understanding and generating human-like text.

LLMs train on vast amounts of publicly available data, including books, articles, websites, and various forms of written communication. This training process involves analyzing patterns in the data to predict and generate contextually relevant text. LLMs learn from the same sources that shape human literacy, making their output reflect the collective literacy levels embedded in their training data.

A study by Parker (2024) highlights that many U.S. adults exhibit low literacy levels, with many struggling to understand complex texts and perform tasks that require critical thinking and comprehension. Research by Nichols (2024) supports these findings, noting that literacy education in the U.S. has continuously adapted to accommodate new media and technologies yet remains challenged by shifting social and technological landscapes.

The Role of Prompt Engineering

The failure to understand the literacy level of LLMs leads to challenges in prompt engineering. Most users craft prompts that do not provide enough explicit detail. The example I like to use is my teenagers. Having raised seven children over the last 30 years, I can tell you that I can relate to the communication challenges involved. For example, if I ask my 13-year-old to “clean the kitchen,” I guarantee that she will “hallucinate” as to its cleanliness. Her understanding of “clean the kitchen” is not the same as mine and therefore, requires additional specific details. In other words, I have to define “clean the kitchen” into a series of instructions, such as emptying the dishwasher and putting away the clean dishes, reloading the dishwasher with the dirty dishes, wiping down the countertops and appliances, sweep the floor, take out the garbage, and put the dirty washcloths and towels into the laundry.

This analogy perfectly illustrates the necessity for clear and detailed prompts when working with LLMs. Just as teenagers need specific instructions to achieve the desired outcome, LLMs require well-defined prompts to generate accurate and relevant responses. Without detailed prompts, the model might produce vague, off-topic, or incorrect text, which can lead to frustration and inefficiency.

Implications for AI Usage

Understanding that LLMs operate at a literacy level comparable to a seventh grader can help users better tailor their interactions with these models. Clear, detailed, and specific prompts are essential for extracting meaningful and accurate information from LLMs. This approach not only improves the quality of the output but also enhances the overall user experience.

As Nichols et al. (2024) point out, AI platforms are not standalone tools but dynamic ecologies encompassing social, technical, and political-economic dimensions. These platforms influence how information is processed and delivered, often reinforcing literacy norms and expectations. Therefore, it is crucial to approach LLM interactions with a strategic mindset, recognizing the importance of prompt engineering in harnessing the full potential of these AI models.

With their capabilities akin to a seventh grader, LLMs offer a fascinating glimpse into the intersection of AI and human literacy. These models are trained on publicly available data and reflect the collective literacy levels embedded in their training material. However, to effectively utilize LLMs, it is imperative to craft detailed and explicit prompts. This understanding can significantly enhance the accuracy and relevance of AI-generated content, making these tools more valuable and efficient in various applications.

References

Nichols, T. P., Thrall, A., Quiros, J., & Dixon-Román, E. (2024). Speculative Capture: Literacy after Platformization. Reading Research Quarterly, 59(2), 211-218. https://doi.org/10.1002/rrq.535

Parker, P. (2024). Low Literacy Levels among U.S. Adults and Difficult Ballot Propositions. Journal of Literacy Research, 12(3), 45-58.

?