Why ChatGPT Struggles with Shakespeare?

Introduction

In the age of artificial intelligence and advanced natural language processing algorithms, tools like ChatGPT seem omnipotent. However, despite their impressive capabilities, they have limitations. One of these is the inability to generate responses that meet strictly defined conditions, such as returning responses containing exactly 10 syllables per line, which is essential in many forms of poetry, including the works of William Shakespeare.

Limitations of Language Models

Stochastic Nature of Text Generation

ChatGPT, like other neural network-based language models, operates on a probabilistic basis. Each generated word or phrase is chosen based on the likelihood of its occurrence in a given context, making the text generation process inherently stochastic rather than deterministic. This means that even if the model "understands" what 10 syllables are, it cannot guarantee that every line will have exactly this number. Research has shown that while LLMs excel in generating fluent and coherent text, their reliance on probabilistic methods makes it challenging to achieve precise control over text generation, leading to variability in outputs (Lu et al., 2024 ).

Lack of Structural Awareness

Although language models are trained on vast datasets, including poetry, they do not possess true structural awareness. They can mimic style and form but do not have the built-in ability to precisely adhere to specific metrical rules, such as the number of syllables in a line. This lack of precise control means that one cannot rely on ChatGPT to create texts with exact metrical requirements. Studies have indicated that while models like GPT-3 can generate text that appears stylistically similar to specific genres, they often fail to maintain consistent structural elements crucial for tasks like poetry (Bender et al., 2021 ).

Technical and Training Limitations

Models like ChatGPT are trained on large datasets containing diverse texts. However, training does not include specific metrical rules unless they are explicitly marked and widely used in the training data. Otherwise, the model does not "learn" to recognize and apply such rules reliably. The training process focuses on maximizing the likelihood of the next word given the previous context, which does not inherently encode structural constraints like syllable count or rhythmic patterns (Radford et al., 2019 ). Consequently, even advanced LLMs struggle to generate text that conforms to strict, deterministic rules without additional conditioning or post-processing (Brown et al., 2020 ).

Emerging Solutions: Agentic LLMs

One promising approach to addressing these deterministic limitations involves the development of agentic LLMs, which integrate game-theoretic methods and preference optimization to enhance model performance. For instance, the "Consensus Game" framework, developed by researchers at MIT, treats the interaction between generative and discriminative components of an AI system as a game. This approach ensures that the model iteratively adjusts its outputs until it reaches a consensus, significantly improving accuracy and coherence in tasks such as reading comprehension, math problem-solving, and dialogue generation (Jacob, Athul P., et al. ).

Another notable method is Direct Preference Optimization (DPO), which optimizes LLMs by directly learning from human preferences without the need for an explicit reward model. This method has shown to be more efficient than traditional reinforcement learning approaches, allowing models to better align with human values and preferences, thus producing more reliable and consistent outputs (Radford, Alec, et al. ).

But how to solve it for now?

Algorithm Ensuring Deterministic Control of Model Inputs and Outputs

But as for now to ensure deterministic control of model inputs and outputs, an iterative approach has to be applied where multiple samples are generated, and those that meet the specified conditions are selected.

Below is an example of such an algorithm in Python:

import syllapy  # Library for counting syllables
import random

def generate_text_with_constraints(model, prompt, max_attempts=1000):
    attempts = 0
    while attempts < max_attempts:
        generated_text = model.generate(prompt)
        if syllapy.count(generated_text) == 10:
            return generated_text
        attempts += 1
    return None  # Returns None if it fails to generate text meeting the conditions

# Example usage with a mock model
class MockModel:
    def generate(self, prompt):
        # Mock function generating random text with varying syllable counts
        texts = [
            "This is an example line",  # 7 syllables
            "We are trying to find the right line",  # 10 syllables
            "This line has exactly ten syllables"  # 10 syllables
        ]
        return random.choice(texts)

prompt = "Begin a poetic line"
chatgpt_model = MockModel()
line = generate_text_with_constraints(chatgpt_model, prompt)
print(line if line else "Failed to generate appropriate text")        

Advantages and Disadvantages of the Approach

Advantages: This approach allows the use of advanced generative capabilities of the language model while meeting precise metrical requirements.

Disadvantages: This process can be time-consuming and inefficient, especially when generating longer poetic texts, as it requires many iterations before generating a line that meets the requirements.

Conclusion

While ChatGPT and similar language models have revolutionized text generation, they struggle with tasks requiring strict adherence to deterministic rules, such as writing Shakespearean poetry. The stochastic nature and lack of structural awareness in LLMs lead to variability and inaccuracies in outputs. However, innovative approaches like agentic LLMs, game-theoretic frameworks, and preference optimization show promise in addressing these limitations. By understanding these constraints and leveraging emerging solutions, we can better harness the power of LLMs for tasks requiring precision and reliability.

Bibliography

• Bender, Emily M., et al. "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?" Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 2021.
• Brown, Tom B., et al. "Language Models are Few-Shot Learners." arXiv preprint arXiv:2005.14165 (2020).
• Jacob, Athul P., et al. "Using Ideas from Game Theory to Improve the Reliability of Language Models." MIT News, May 14, 2024.
• Kim, Alex G., et al. "Financial Statement Analysis with Large Language Models." Becker Friedman Institute for Economics at the University of Chicago, 2024.
• Liang, Tian, et al. "Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate." arXiv preprint arXiv:2305.19118 (2023).
• Lu, Sidi, et al. "Open-Domain Text Evaluation via Contrastive Distribution Methods." arXiv preprint arXiv:2306.11879v2 (2024).
• Rafailov, Rafael, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. "Direct Preference Optimization: Your Language Model is Secretly a Reward Model." arXiv preprint arXiv:2305.18290v2 (2023).
• Radford, Alec, et al. "Language Models are Unsupervised Multitask Learners." OpenAI Blog 1.8 (2019).

Note: This article has been created using various AI models; however, all outcomes have been validated, and the author, Janusz Marcinkowski, assumes full accountability for the content.