What You Ask AI Matters: A Case Study, Analyzing the Current State of the 2024 Presidential Election

By Severin Sorensen, Author of the AI Whisperer Series

"What you ask AI matters. With Generative AI like ChatGPT, Gemini, Claude, CoPilot, etc., if you ask a poorly constructed question, you get poorly thought-out answers. As Socrates noted so many years ago, the problem lies in our questions, not the answers. The specificity of the questions we ask AI can significantly impact the depth and quality of the insights we obtain. Here’s an example illustrating how a general question can be too simple, and obtuse and how a better-nuanced question can get you much greater, more accurate, and timely insights."

Introduction

The specificity of the questions we ask AI can significantly impact the depth and quality of the insights we receive. For this demonstration, I chose a topic of deep interest and discord within the US. As we approach the 2024 U.S. Presidential Election between President Joe Biden and former President Donald Trump, it's crucial to understand how the questions we pose shape the information we gather and the conclusions we draw. This article explores how different types of AI prompts and search queries can yield varying levels of detail and accuracy in AI responses, providing valuable lessons for business users seeking to leverage AI effectively.

The image above illustrates the current state of the 2024 U.S. Presidential Election, capturing the deep divide and uncertainty within the nation. With crumpled flags symbolizing the contention between political parties, the image reflects the turbulent political climate of July 2024. As voters seek reliable information, the specificity of questions posed to AI tools like ChatGPT, Grok, and Perplexity becomes crucial in obtaining accurate insights into public sentiment and polling data. Understanding this dynamic can help navigate the complexities of political analysis and decision-making in a highly polarized environment.

The Power of Detailed Inquiry

To illustrate this point, let's consider an A/B test I conducted today (July 17, 2024) comparing two different prompts on the 2024 election. The goal was to demonstrate how the specificity of a prompt influences the resulting analysis.

Simplified Prompt (A):

• "What is the pulse, polls, and sentiment of the country now in terms of the US Presidential Election between President Biden and Former President Trump."

Detailed Prompt (B):

• "Provide a comprehensive analysis of the current state of the 2024 U.S. Presidential Election between President Joe Biden and Former President Donald Trump. Include: The latest national polling averages, including both head-to-head and multi-candidate scenarios. Swing state polling averages and trends in key battleground states. Favorability ratings for both candidates. Key issues influencing voter sentiment, including economic concerns, immigration, and social issues. Significant trends or changes in voter demographics, including age, race, education, and geographic divisions. Any notable third-party or independent candidates and their impact on the race. Voter enthusiasm and satisfaction levels with the candidates. Betting odds or prediction market data, if available. Any discrepancies or notable differences between various polling aggregators (e.g., RealClearPolitics, FiveThirtyEight). Potential factors that could influence polling accuracy or voter turnout."

Analysis of Results from Various LLMs

To further explore the effectiveness of these prompts, I ran them through multiple large language models (LLMs), including Perplexity AI, ChatGPT, Gemini, CoPilot, Claude, and X-Grok. Here’s what I found:

Perplexity AI and ChatGPT:

• Response to Simplified Prompt (A): Both Perplexity AI and ChatGPT provided a general overview, focusing broadly on sentiment and polling data. While the responses included some relevant details, they lacked the depth and specific insights needed to fully understand the current state of the election.
• Response to Detailed Prompt (B): These models produced a comprehensive analysis, drawing from multiple reputable sources such as RealClearPolitics, FiveThirtyEight, Pew Research Center, Emerson College Polling, and The Hill. The detailed prompt resulted in rich, nuanced responses that covered national and swing state polling averages, favorability ratings, key issues, voter demographics, third-party candidates, voter enthusiasm, betting odds, and discrepancies among polling sources.

Gemini and CoPilot:

• Both models encountered issues with the prompts. Gemini initially attempted to respond but was halted by a system guardrail, while CoPilot directed me to use Bing Search instead. These responses highlight limitations in some AI models regarding handling politically sensitive topics.

Claude:

• Claude provided a general response, noting its knowledge cutoff in April 2024 and recommending that users refer to reputable polling aggregators and news sources for the most current information. This limitation underscores the importance of selecting the right AI model for timely and sensitive data.

X-Grok:

• Initial Response: X-Grok provided a response based on older data, likely from cached memory. The data did not reflect the most recent polling information.
• Refined Interaction: After updating X-Grok with the latest data from RealClearPolitics, the model produced a more accurate and current analysis, showing the value of refining AI interactions with updated and specific information.

In the end, only ChatGPT, Grok, and Perplexity answered the question. The mainstream LLMs of Google Gemini and Microsoft Copilot avoided or declined to ponder the question. This situation raises concerns about where people will turn for information if the largest LLMs refuse to explore critical issues. The unintended consequence of these mainstream LLMs abstaining from democratic discussions could be driving the interested electorate to less reliably sourced platforms, potentially rife with misinformation and political bias. However, that's not the focus of today's discussion. Let's return to the importance of why prompts matter.

Why Detailed Prompts Matter

Detailed prompts guide analysts to collect specific, varied data points, ensuring a comprehensive overview of the electoral landscape. They compel the inclusion of various aspects such as national and state polls, favorability ratings, key issues, and voter demographics, leading to more accurate and insightful analysis.

Conversely, broad questions yield general overviews that might miss critical nuances. In political analysis, where voter sentiment can shift based on myriad factors, the depth and specificity of our inquiries are crucial for capturing the full picture.

Insights for the Informed Electorate: Individuals, Businesses, and Communities of Interest

For individuals, business, and communities of interest, and other users seeking accurate, timely, and objective data on any complex topic, including elections, consider using detailed prompts like these:

• What are the latest national polling averages, including head-to-head and multi-candidate scenarios?
• How are key swing states polling, and what trends are emerging in these battlegrounds?
• What are the favorability ratings for each candidate?
• Which key issues are most influencing voter sentiment, and how are different demographics responding?
• Are there notable third-party or independent candidates, and what impact are they having on the race?
• What are the levels of voter enthusiasm and satisfaction with the candidates?
• What do betting odds and prediction markets indicate about the election's outcome?
• Are there discrepancies between various polling aggregators, and what might explain these differences?

Conclusion

The depth and quality of analysis are directly proportional to the specificity of the questions asked. Detailed prompts lead to richer insights, enabling business users to make informed decisions. As we navigate the complexities of the 2024 U.S. Presidential Election, the questions we ask will shape our understanding and, ultimately, the decisions we make in our professional and personal lives.

Remember, the questions you ask AI matter. Whether you're a voter seeking clarity or a business leader looking to provide accurate insights, specificity in your inquiries is key. Stay informed, ask detailed questions, and join us next time as we continue to navigate the intricate world of political analysis. Until then, keep questioning and stay curious.

By focusing on detailed, specific questions, we can ensure that our analyses are robust, insightful, and ultimately more useful for informing decisions. This approach not only benefits those analyzing the data but also empowers business users to make well-informed choices based on comprehensive and accurate information.