The 5 Blind Spots of Synthetic Responses: Simulating Insights and Tribute Bands

The Epistemic Status of Large Language Models

‘Theory-free science’ refers to the idea that scientific research can be conducted without relying on established theories or concepts. This approach emphasizes using AI and machine learning to analyze large datasets, aiming to discover truths directly from patterns in the data. By minimizing preconceived notions, researchers can potentially reveal unexpected trends, relationships, or phenomena that established theories might overlook, enabling deeper exploration of complex systems and leading to better predictions and scientific progress. The theory-free ideal envisions scientific discovery relying on data-driven methods alone. When trained on large datasets, the superior pattern recognition of modern AI has already been beneficial in areas such as diagnostic imagery for medical applications or the discovery of new drug molecules and the shape of proteins.

The theory-free approach connects closely with the operation of large language models (LLMs). LLMs generate language by predicting the next word in a sequence based solely on training data, embodying the idea that proficiency can emerge without explicit, theory-driven guidance. LLMs depend on artificial neural networks and machine-learning techniques that simulate some aspects of human brain functionality—enabling them to learn from vast amounts of unstructured text data.

However, purely data-driven, machine-learning approaches often face criticism for their epistemic limitations, which can result in what some describe as pseudoscience. The term "epistemic status" refers to the validity and reliability of a method for gaining knowledge. A higher epistemic status is achieved when research conclusions are confirmable, replicable, and comprehensible. Two primary issues underpin the criticisms of theory-free science:

1. Theory-free science is not truly ‘free’. Data is needed to train a neural network. Even in unsupervised learning models, researchers may unconsciously cherry-pick data, leading to biased outcomes without a theoretical framework for thorough scrutiny. While it’s not difficult to generate a large amount of data for tasks like “next word prediction”—where we don’t even need to label the data, as the next word itself serves as the label—this isn't always the case. Without a guiding theory, there is a risk of overfitting models to specific datasets, which may lead to conclusions that are not robust or replicable in different contexts. As a result, it has been argued that this so-called theory-free research is not ‘free’, it just omits the theoretical background needed to produce reliable and meaningful scientific insights. Empirically derived results can be more challenging to interpret without a theoretical lens, potentially leading to misinterpretation or misunderstanding of the data.
2. Struggles with Causal Relationships Their reliance on correlations can limit their ability to explain why certain phenomena occur, leading to the classic "correlation is not causation" problem. Critics argue that operating without a theoretical framework can result in fragmented findings that lack coherence and context, akin to hallucinations, potentially leading to seemingly correct but fundamentally wrong conclusions. These limitations diminish the epistemic value of LLMs, as they cannot be relied upon for accurate, reliable advice and information.

A classic illustration of these limitations is in Ptolemy's geocentric model, which, while reasonably accurate in predicting planetary positions, was based on flawed assumptions. This example highlights the importance of theory-driven understanding over mere predictive accuracy. In many cases, effective decision-making requires insight into the reasoning process—a characteristic that might be regarded as true intelligence. Thus, human oversight and complementary technologies are essential to compensate for these limitations, especially in contexts such as:

·?????? Planning and Execution: While LLMs can suggest structured guidance, their inability to gauge feasibility accurately often results in impractical or na?ve suggestions.

·?????? Handling Complex Situations: LLMs struggle with logical deductions, frequently producing incorrect answers that sound persuasive. They lack a genuine "understanding" of meaning or causality; they predict based on probability distributions in their training data, not real-world knowledge or causal logic.

·?????? Retrieving Precise Information: The quality and breadth of an LLM’s training data directly shape its output. This reliance can lead to inaccuracies, fabricated details, and inconsistencies. LLMs may misunderstand prompts or lack contextual awareness, resulting in responses that are fragmented or irrelevant.

Ultimately, while theory-free science and LLMs offer valuable capabilities in pattern recognition and predictive analysis, they often fall short of delivering a deeper understanding. Moreover, their reliance on statistical correlations can yield insights that may be intriguing but ultimately misleading.

The Risks of Simulating Insights with LLMs

The rise of LLMs is understandably a game-changer for the market research industry. While AI’s ability to streamline research processes and analyze vast amounts of human-generated data brings undeniable advantages, one of its most debated applications is the fabrication or simulation of insights. This involves using LLMs to mimic human responses through the creation of ‘synthetic respondents’ – AI agents, created with specific characteristics in terms of demography, preference, or even personalities, that simulate human input. The result is a new category of data that promises faster, more cost-effective solutions for market research.

However, despite its appeal, this fabricated data comes with significant risks. Over-reliance on synthetic responses can lead to flawed insights and unsound decision-making. These challenges can be distilled into what we call the 5 blind spots of synthetic responses:

• Detached from Cause: Synthetic responses often miss the underlying ’why,’ offering a limited understanding that fails to connect the dots meaningfully.
• No Heart: Synthetic responses appear ‘emotionally flat’, lacking the authentic human sentiment needed to resonate with real human behavior.
• Skewed Representation: Synthetic responses offer a partial view that fails to reflect and capture the full complexity of reality accurately.
• One Size Fits None: Synthetic responses appear rigid and overly standardized, failing to accommodate the diversity of human interactions.
• Flickering Consistency: Synthetic responses can alternate between high-quality outputs and glaring inaccuracies; this lack of consistent performance undermines credibility over time.

Let’s dive deeper into these blind spots.

1. Detached from Cause

LLMs learning process mirrors aspects of human development but with crucial differences; while humans develop general intelligence through varied experiences, AI systems require massive amounts of domain-specific data to achieve competence in ‘narrow’ tasks. Research suggests that children's tendency to ask "why" is linked to their cognitive development, particularly their growing understanding of causality and their desire to make sense of the world. When children don't receive satisfactory answers, they often persist with their questions, demonstrating their determination to uncover underlying truths. Human nature is not comfortable with black box models and seeks an understanding of the underlying mechanisms; this is not the case with LLM learning.

For instance, an LLM might predict that a specific demographic prefers a certain product, but it may miss underlying cultural reasons driving these choices. As said, while LLMs and correlation-based models can make effective predictions, they often cannot explain why something happens. Correlations may miss hidden paths or “backdoors” that might confuse us about cause and effect; the classic ice cream and sunburn correlation illustrates this perfectly - to understand the true effect of eating ice cream on sunburn, you’d need to block the “backdoor” by accounting for sun exposure. That means you’d compare people who eat ice cream and those who don’t, but only when sun exposure is the same. Predictions based on causality are stronger because they rely on an understanding of how one factor directly influences another.

However, the landscape is evolving. Recent iterations of LLMs, such as OpenAI’s Strawberry, aim to overcome these limitations by adopting a "chain of thought" reasoning process. This approach mirrors the way humans solve problems step by step, potentially enabling more accurate and nuanced consumer insights. Yet, as of today, the fundamental challenge persists; without deeper causal training, LLMs still struggle to fully capture the underlying reasons and subtleties of consumer behavior

2. No Heart

Synthetic responses, while logically coherent, often lack the emotional depth inherent in human interaction. This shortcoming creates a significant gap in understanding consumer contexts where empathy, trust-building, and personal connection are essential. These aspects are deeply rooted in human intuition and emotion-driven decision-making processes, as characterized by Kahneman's System 1—the rapid, instinctive cognitive functions that guide much of our behavior.

Early studies, such as "Digital Respondents and their Implications for Market Research" by Michael Patterson and Cole Patterson and "Using Synthetic Data to Solve Client Problems" by Julia Brannigan and Kerry Jones, emphasize that while LLM-generated responses may be “rationally accurate”, they fail to resonate on the emotional level humans instinctively seek and recognize. The core challenge lies not only in the limited availability of System 1 training data for LLMs but also in their fundamental inability to experience emotions firsthand.

This highlights a critical limitation of synthetic responses: while LLMs might fully replicate logical decision-making frameworks at some point, they remain disconnected from the emotional underpinnings that drive human behavior. This disconnect has profound implications for consumer research and behavioral analysis methodologies.

3. Skewed representation

LLMs are highly susceptible to the biases inherent in their training data. Because LLMs learn from vast datasets, they often amplify existing biases. If the training data reflects historical prejudices or imbalanced representations, the insights generated by these models may lead to skewed market research findings and biased recommendations. For instance, if an LLM analyzes customer feedback predominantly from a specific age group or region, its insights may fail to capture the preferences of a broader audience, resulting in flawed market predictions.

A recent study by Yan Tao, Olga Viberg, Ryan S. Baker, and René F. Kizilcec titled “Cultural Bias and Cultural Alignment of Large Language Models” illustrates this issue. The study found that, when not explicitly controlled, LLMs tend to exhibit biases, often answering questions in ways that align with the perspectives of individuals from Northern Europe and Anglo-Saxon countries. In one experiment, ChatGPT was asked to respond to an established methodology for measuring cultural values—the World Values Survey. Its responses closely mirrored those of individuals from these regions when plotted on a cultural map (see below).

When ChatGPT was explicitly prompted to respond as if it were a person born in specific countries, its answers aligned much more closely with the cultural values of individuals from those regions. The study underscores the need for rigorous control of biases in LLMs to ensure more accurate and inclusive outcomes. Good seed data is essential for any form of synthetic data.

4. One Size Fits None

One of the most cited arguments in favor of synthetic data's validity is its seemingly uncanny ability to produce results comparable to human data when measured using centrality metrics like means or averages. However, while median values in synthetic data often closely resemble those of real human responses, the distribution of synthetic responses tends to be much narrower, with significantly less variance.

In statistics, dispersion measures—such as variance or interquartile range—are critical for understanding how data points are distributed around the central tendency. These measures are fundamental for making predictions, testing hypotheses, and assessing the reliability of conclusions. Two datasets with identical means can exhibit vastly different spreads, leading to entirely different interpretations and outcomes.

Synthetic data, while useful, often lacks the richness and diversity of real human data. It may capture broad patterns but miss the subtleties and diversity of perspectives that human data provides. This can result in insights that are recycled, derivative, and homogenous.

Thus, while synthetic data often mirrors the central tendencies of human data, its limitations must not be overlooked. The late Clayton Christensen, -renowned for his "Disruptive Innovation" and "Jobs-To-Be-Done" theories-, reportedly kept a sign in his HBS office that read, “Anomalies Welcome.” This phrase encapsulates the importance of embracing unexpected results and outliers— preserving variance and exploring outlier perspectives is crucial, as these often contain the seeds of novel ideas and insights.

Moreover, focusing solely on tightly constrained synthetic data risks creating dangerous feedback loops, where the lack of variability compromises future insights and conclusions. Over time, this could lead to mediocrity and a lack of distinctiveness in decision-making and strategy.

5. Flickering Consistency

The above-mentioned blind spots can potentially be addressed through inverse engineering and refined prompting, however, there is a persistent concern: everything seems obvious once you know the answer, and the "hit-or-miss" nature of LLM-generated responses undermines long-term credibility.

Consumer decisions are shaped by the interplay of cultural norms, economic conditions, and individual motivations. Similarly, detecting and addressing biases in LLMs is a complex and emergent field, one in which?no definitive breakthroughs?have been achieved. This challenge is compounded by the computational demands of LLM research, which require vast datasets and immense processing power.

Adding to this complexity, synthetic outputs generated by LLMs exhibit a form of epistemic instability. Their reliability fluctuates, oscillating between moments of high-quality insights and glaring inaccuracies. This inconsistency mirrors broader issues in scientific fields like psychology, where the reproducibility crisis has raised concerns about methodological rigor, publication bias, and the reliability of foundational findings.

For synthetic systems, consistency, and reproducibility are critical benchmarks for validating outputs. Just as psychology's epistemic instability calls into question its foundational knowledge, the variability in synthetic outputs raises doubts about the epistemic trustworthiness of these systems. Without reliable performance, the credibility of LLMs diminishes over time, eroding their utility as dependable tools.

A Fit-for-Purpose Approach to Synthetic Responses

The 5 blind spots just described illustrate how the use of synthetic responses to substitute human data is inherently limited; human behavior exhibits unpredictable dynamics. This challenge can be compared to the classical three-body problem in physics, where predicting the motion of three interacting objects of similar mass becomes impossible to solve analytically. Small variations in the initial conditions of the three-body system lead to exponentially divergent outcomes, making precise predictions unfeasible.

Similarly, human responses, shaped by a multitude of interdependent factors such as emotions, environment, and social influences, defy precise modeling. Just as the three-body problem's chaotic nature prevents stable solutions, the chaotic interplay of variables in human decision-making leads to outcomes that are unpredictable in detail.

Likewise, synthetic responses may approximate generalized patterns but fall short of mirroring individual human reactions accurately. Predicting human behavior with synthetic data is akin to solving a chaotic system. While broad trends might be statistically modeled, replicating precise, individualized responses remains out of reach due to human interactions' inherently unpredictable and complex nature.

Like Tribute Bands

Understanding the limitations of synthetic responses in reproducing human data doesn't invalidate their use in market research, though; once we recognize synthetic data as a derivative product—one whose value stems from the quality and performance of a more fundamental source—its utility and potential become much clearer.

Consider tribute bands to better understand the derivative nature of synthetic responses and their proper application. Groups like 'U2 2' or 'The Fab Four' meticulously recreate the sound, style, and appearance of famous artists. While they provide remarkably accurate representations and deliver great performances, there's a clear fit-for-purpose distinction: they won't sell out Madison Square Garden, but they can successfully entertain at your local venue. The Next Rembrandt project is another example: while it achieves an uncanny resemblance to the original master's work, its derivative nature and lack of artistic intent invalidate its value as a true piece of art.

Similarly, synthetic responses have their own valuable yet distinct role in market research, with their greatest potential lying in simulations and the delivery of human data insights. This position helps define both the capabilities and limitations of synthetic responses: they can effectively simulate and represent real data patterns but shouldn't be expected to fully replace authentic human responses.