Understanding how synthetic panels actually work

In our previous book, "Savvy," Rohini and I recounted how AlphaGo, a technology from Google DeepMind, defeated Lee Sidol, one of the world's top Go players. This event showcased a level of deep insight, strategic foresight, and creativity previously thought impossible for an AI engine. Fast forward six years, and the prospect of Artificial General Intelligence (AGI) becoming a reality within this decade doesn't seem so implausible. AGI refers to an AI engine capable of performing a wide array of non-physical tasks, including metacognitive abilities like learning new skills, at a level comparable to at least the 50th percentile of skilled adults. With this future rapidly approaching, it's a pertinent question to ask:

Will such advancements in AI eliminate certain roles in Marketing?

At the first developer conference held by OpenAI, Sam Altman, who was recently and controversially dismissed by his board, made a striking remark. He suggested that the work his team is currently undertaking behind the scenes will render OpenAI's recent developments modest in comparison within a year. With this perspective, let's delve into one way artificial intelligence might be beginning to reshape marketing. It might seem ordinary in the future, but for now, it may feel blasphemous.

For any senior marketer, overseeing a customer research program comes with inherent challenges. Setting a solid learning agenda, getting the rest of the C-Suite on board, safeguarding the budget, recruiting customer panels, deriving insights, and ensuring these insights influence actual marketing and product decisions are demanding tasks. Yet, with the advent of GPT-4 and other Generative AI models, some of these tasks may have become significantly more manageable. In fact, some agencies, like Pereira O’Dell and Dentsu, are leading the way with persona creation experimentation efforts. Consider the below practical scenario you can implement today, and let’s explore both the risks and the possibilities it presents by looking at some recent research out of the Harvard Business School on the subject.

Creating Synthetic Customer Panels

Recently, I had conversations with two agency executives who shared their innovative approach to customer research. They use social media comments to create synthetic customer panels. Their method involves exporting all comments from a brand’s TikTok account and feeding them into ChatGPT. They instruct ChatGPT to form a panel of four different personas, defined by them, using insights from the uploaded data. ChatGPT then acts as the moderator, posing questions about the brand and its products to the panelists. After observing the interactions for a while, they open the floor to audience questions, injecting some of their own queries and paying close attention to the responses. This process is not only cost-effective and easy to implement, but also yields significant insights.

Now, imagine elevating this approach by uploading actual customer demographic and psychographic data from internal databases or external sources like the US Census data tables, instead of relying on social media comments. Moreover, consider integrating all conversations with sales and customer service representatives over the past year. Think about the potential depth and diversity of the panel, the unique artificial personas created, and the richness of the questions and responses as a result. On a larger scale, this method could dramatically reduce the cost of customer research programs while delivering insights comparable, if not superior, to those obtained through traditional human-centric research methods.

Opportunities with Synthetic Panels

Granted, synthetic panels have their own sets of opportunities and risks. It would be naive to think that these would be as good as human ones. We simply don’t know, or do we? Harvard University Professors set out to answer that exact question when they studied (link takes you directly to their paper) how LLMs performed for researchers and practitioners who aim to understand consumer preferences. Here are some of their learnings summarized below:

1. GPT for research works: Their analysis suggested that GPT and LLMs more broadly, do serve as a powerful tool for uncovering customer preferences. GPT exhibited a number of behaviors consistent with economic theory, including both declining price sensitivity with income and state dependence when prompted as if it were a randomly selected customer. Unlike human beings, it typically has fewer issues with stated preferences versus revealed preferences as it doesn’t try to appease in any way.
2. GPT generated richer results than humans: The study also showed that these results were strong while being realistic and consistent with results that were obtained from existing, human-based research. Furthermore, these results did not suffer from the shortcomings that research which is depended on human subjects typically do. In fact, the results suggested that GPT could also be useful for the development of new products and at the very least as a complement reducing the number of human participants required for a study.
3. GPT serves as a realistic simulator of customer choice: The authors of the paper, conclusively, believe that GPT can serve as a realistic simulator of customer choice with minimal effort beyond the types of prompting. GPT was used effectively for conjoint-type analysis to understand which attributes of a product were most valuable. Similar to my scenario above, the authors believe that by providing various forms of “knowledge” (to use their language), and building in personas, GPT can be more context specific.
4. GPT and other LLMs are only getting stronger: Finally, the authors of the paper believe that LLMs will become increasingly useful to customer research over time, with increases in LLM accuracy and access to more real-time data leading to the LLMs being able to absorb and infer even richer aspects of consumer behavior. This sophistication will probably lead to an eventual revisiting of the historic paradigm that customer research fundamentally (and ideally) requires human beings to be recruited, interviewed and or surveyed.

Risks with Synthetic Panels

And yet, all is not rosy with synthetic panels. There are some clear risks to be aware as you explore using them to complement your traditional research efforts.

Here are some of the risks outlined in the paper:

1. Preferences maybe dated: Because GPT is “pre-trained” without specific data typically provided by customer research and without access to the real-time Internet (this is changing though), some of the preferences revealed maybe static or dated. The learning here is to pay careful attention to what data you’re depending upon to train the LLM for your market research. Keep in mind that there maybe some extreme consumers who don’t post online or may not even have Internet access. GPT will not reflect their thinking unless it’s given extra data that captures their point of view.
2. Wording matters with GPT more than ever: Furthermore, the research also highlighted (though you can test this out yourself), that GPTs have sensitivities to how prompts are worded. While the differences in results are not significant based on the different ways questions can be worded, it is still worth noting as a potential risk. Arguably, this risk applies just as much to traditional research where the survey design is very important.
3. Hallucination is a risk but there’s an upside: LLMs are known to hallucinate and that’s a risk in the context of customer research. Hallucination means confidently returning incorrect information. However, there’s a flip side to this as well, LLMs may hallucinate their way to unpacking insights or potential product features that no human being may ever have thought of.
4. LLMs lack ethical judgement: LLMs like GPT-4 lack the ethical judgement and reasoning capabilities needed to navigate more thorny ethical issues. In addition to that, GPTs can be fraught with bias depending upon the data upon which they have been trained on or enhanced with. Human judgement and oversight will continue to play an important role for some time.
5. Contextual understanding may get lost: Last but not the least, LLMs may miss contextual understanding. As I mentioned at the start of this piece, we are still ways away from Artificial General Intelligence (AGI) and LLMs may miss broader insights and more nuanced context simply by virtue of still being young in their own development.

All in all, just as with human oriented customer research, leveraging artificial intelligence to either complement or supplant existing research methods pose its own opportunities and risks. That is to be expected. What’s clear though is that the AI is sophisticated enough to serve as a meaningful complement to human oriented research and in cases where there’s appropriate oversight and with the right data inputs, it may even adequately supplant existing research methods. That alone is mind-blowing for a marketer. I’ll leave you to decide whether this means human-centered customer research, which is typically a lot more time consuming and costly, will evolve or go extinct at some point in the future. What do you think?

Recent Being Savvy Articles

• Thriving Professionally in the AI Era (November 2023)
• The Dual Horizon of AI's Promise and Peril (October 2023)
• Forbes CMO Summit: Marketing Insights (October 2023)
• Simple Steps to AI Proficiency for Marketers (October 2023)
• AI Readiness: A Checklist for Marketing Leaders (September 2023)

What I’m reading

• How OpenAI so royally screwed up the Sam Altman firing (CNN)
• The State of Generative AI in the Enterprise (Menlo Ventures)
• Improve Your Company’s Use of AI with a Structured Prompts (HBR)
• Microsoft Copilot Studio & new features for Microsoft 365? (Microsoft)
• Luma’s State of Digital Marketing 2023 (Luma Digital Marketing Summit)

Companies I’m looking at

• Bardeen.ai - Automation made as easy as texting a friend. Delegate a person’s tedious work with a few lines of text.
• Bluewillow.ai?- From logos and characters to digital artwork and photos describe the image you want and the tool will create the graphic for your project.?
• Octoml.ai - Delivers production-grade GenAI solutions running on the most efficient compute, empowering builders to launch next-gen AI applications.
• Perplexity.ai?- Powerful chatbot-style search engine that uses AI to gather information and provide accurate answers to users' questions.
• Storybird.ai - Improving storytelling through human <> AI collaboration by creating personalized stories with AI.

I'm continually researching and experimenting with new technologies in the AI space. If you have a startup I should look at, please?email me.

Where I’m going and where I’ve been

I recently returned from a brief trip to Asia, which included a layover in Austin where I spoke to 200 marketers at The Room Brand Summit. The event was exhilarating, providing me with a chance to share my perspective on Work in the AI Era. Additionally, it was a fantastic opportunity to reconnect with former colleagues and friends.

Soon, I'll be navigating the busy Thanksgiving travel with my family as I head to the East Coast to visit my in-laws. And no, I’m not one of those Airport Dads in the Uber advertising though I did find it to be incredibly insightful and well executed creative work. Afterward, I'll return home to the West Coast for a private AI Thinkers dinner and lots more writing. I would welcome the opportunity to speak at your event.?Email me?to discuss further.

What I’m writing about this week

I am currently working on my third book, which focuses on artificial intelligence in business and marketing. This week, I'm progressing to a chapter about the emergence of Generative AI. Interestingly, I'm writing this chapter amidst significant board and executive changes at OpenAI, the creators of GPT-4. Keep an eye on this newsletter for more updates and insights from my upcoming book.

Don't miss an edition by subscribing for free to have the newsletter delivered to your email inbox.