What I've Observed Experimenting with Generative AI Customer Personas

Welcome to the second installment of a three part series exploring a much-touted Generative AI use case—customer persona creation. The ability to rapidly create customer personas with ChatGPT and other Generative-AI tools is receiving widespread attention throughout UX/CX, marketing, and product circles. There is currently no shortage of AI pundits and influencers extolling the virtues of near instant personas.

In Part One of this series titled “Why I Don’t Recommend Generative AI Customer Personas,” I described several fateful risks of using generative AI as a substitute for more time-consuming methods of traditional customer research. You can catch-up on the Part One discussion at the following link:

https://www.dhirubhai.net/feed/update/urn:li:activity:7115388318992748544/

In this installment, I put my assertions about ChatGPT to the test, comparing Gen-AI personas against those developed with actual customer research. In Part Three coming soon, I'll explore ways to minimize Gen-AI research risks.

Will My Beliefs Hold Water?

ID 14507271 ? Akinshin |Dreamstime.com

As a Vanderbilt University Medical Center -trained audiologist, evidence-based practice is fundamental to my professional beliefs. (For this, I tip my headphones to audiology luminaries James Hall and Fred Bess). Having an evidentiary basis for best practices is sound medicine. So, while I intuitively believe that swapping actual customer research for Gen-AI proxies is a bad idea, I want to substantiate my belief. What if, now or in the future, AI-generated personas are shown to be substantially equivalent to those created with primary customer research?? What if they're even better due to Gen-AI's vast data sets and superior pattern recognition capabilities? If that’s what we find with high confidence, my views will need to change.

Let the Experiments Begin!

ID269544630 ? Wisconsinart |Dreamstime.com

Using the latest ChatGPT4.0 beta version that browses the internet for up-to-date information along with an innovative prompt created by nxtting Chief Customer Officer Daniel Roundy , I've conducted a series of experiments. I've prompted ChatGPT to create multiple Gen-AI customer personas. All I had to do was specify a goal, target audience, and up to six customer research areas of interest. You can check-out the complete details of Daniel's approach at the following link: https://www.dhirubhai.net/feed/update/urn:li:activity:7112631668628185088/

?

Next, I compared these AI-generated personas with a convenience sample I had on hand. The sample consisted of research-based personas that I’d either created or used in past product, marketing, or CX projects. They are mainly an aggregate of customer interview findings but are also informed with customer feedback and a mix of product and operational data.

Let me say upfront that what you're about to encounter is best described as a casual exercise and not a laboratory-controlled experiment. I look forward to learning from others who will publish on the topic. But for now, I’m assessing Gen-AI persona quality for myself and sharing my findings with you to draw whatever conclusions you will. I hope you’ll similarly experiment and share your findings with me and our community.

Pro Tip: To get the most accurate view of Gen-AI's persona-creation potential, I urge you to delve into the full ChatGPT conversation rather than simply skimming the summary screenshots below. Doing so takes minimal time, and the increased detail you'll receive offers a more realistic view of Gen-AI's persona-creating capabilities. Links to the full ChatGPT conversations are included with the examples below.

Comparing AI-Generated Customer Personas with the Real Deal

Case #1 Research-Based Persona--Megan the Pediatric Medical Assistant:

Below is an example of a user persona I created for a pre-market medical device. The device's intended users are medical assistants working in pediatricians' offices. The persona has two pages--page one appears below and offers an overview of pediatric medical assistant attributes based on interviews with representative customers and non-customers. For confidentiality reasons, I'm unable to share the second page which consists of persona insights specific to our product concept.

?

Case #1 Chat GPT Version--Pediatric Medical Assistant Persona Summary:

Compare the persona developed from traditional customer research above with ChatGPT's persona summary below. Again, I strongly recommend a review of the complete ChatGPT conversation at the following link:

https://chat.openai.com/c/74b1b34d-1065-46cf-8816-6a39c2a632d3

Case #1 Discussion:

In full transparency, I expected page one of ChatGPT's pediatric medical assistant persona to be spot on. I figured there would be plenty of online content about both medical assistants and the pediatric primary care environment to piece together a realistic picture of our persona's overall characteristics and concerns. I predicted that page two which featured our persona's views on a product category we sought to enter would fall short due to a lack of sufficiently-relevant online data.

Based on the full chat conversation, in many cases ChatGPT's insights were consistent with our primary customer research findings--at times better than I expected. Despite this, ChatGPT failed to surface crucial details. For example, the majority of medical assistants have children living in the home, substantially impacting our persona's attitudes, needs, and priorities. I saw no evidence that ChatGPT took this crucial factor into account. In much the same way, ChatGPT failed to discern the overwhelming stress and burn-out medical assistants described. And while both our research and ChatGPT found excellent patient care to be a goal of pediatric medical assistants, ChatGPT elevated additional goals such as professional growth and keeping up with the latest developments in healthcare and technology. These weren't our subjects' highest aims. The medical assistants I interviewed were far more concerned with just surviving the day--keeping-up with their task loads, staying on schedule, trying to leave at a decent hour to care for their families. ChatGPT was silent on these concerns. In addition, many I interviewed directly contradicted ChatGPT's assertion that trust in senior medical staff was a fundamental medical assistant attitude. Instead, strained relationships with superiors and feeling unappreciated and disrespected were consistent sentiments. While getting a lot of the basics right, ChatGPT missed much of what really mattered to the women serving in this role.

Case #2 Research-Based Persona--Geraldo the Home Chef with a Taste for the World:

Next is a user persona I created for a personal project when I was a student in CareerFoundry 's Voice User Interface Design Program. I was developing an 亚马逊 #Alexa app that offers recipes and assists home chefs with hands-free dinner preparation. I conducted interviews with individuals who wanted to experience dinner prep and meal time as a way to unwind at the end of a busy work day.

(Please keep in mind that I created this persona for a personal project, accounting for its oddly specific family archetype that mirrors unique characteristics of individuals within my social circle??).

A screen shot of ChatGPT's persona summary is below. The full interactive persona workshop conversation can be found at the following link:

https://chat.openai.com/c/78a0fdf2-e0a8-4125-ba80-e1ce96068f3f

Case #2 Chat GPT Version--Home Chef Persona Summary (Screenshot):

Case #2 Discussion:

Taking into account the entire chat conversation in addition to the persona summary above, ChatGPT did a decent job representing the attributes of home chefs seeking a pleasurable escape from hectic days. It didn't get anything terribly wrong; it just wasn't as detailed as the persona created with real life home chefs (no mention of dirtying too many dishes or difficulty juggling cooking times when preparing multiple recipes, etc.) However, our AI subject experts offered a number of test-worthy suggestions for delighting our persona. These ideas included social sharing and recipe swapping, surprise recipe suggestions based on past favorites, gamification with badges cooking badges achievements, and cooking masterclass special events. These are great ideas I'd love to test-out with real subjects!

Case#3--Client Examples:

To preserve confidentiality, I won't be posting any client personas. However, I asked ChatGPT to re-create multiple examples from past projects so I'd have a larger sample for comparison. Each time, I entered the project goal, target audience, categories of interest, and the titles of internal subject matter experts we tapped prior to initiating customer interviews. I'll patch together any notable finding and include them in the remaining sections.

ID 68611452 ? Alexey Osadchiiev |Dreamstime.com

Observations

Photo 89377609 ? Dmytro Zinkevych | Dreamstime.com

Below are some of my key findings listed in the order observed--three positive and three negative. There are plenty more worthy of discussion. I hope you'll add some you spot or have perhaps encountered yourself in Comments.

Generative AI Positives:

1. Ease of Use. I was pleasantly surprised by how easily I could refine my ChatGPT persona throughout the development process. Based on my understanding of the persona-creation prompt, I incorrectly believed I was limited to the prompt's proposed number and type of persona categories as well as the number of subject matter experts. Nope! All of this was completely modifiable. In much the same way, it was easy to provide additional context throughout the process. For example, when reproducing the product-focused page of my Megan the Pediatric Medical Assistant persona, ChatGPT initially listed affordability as a chief product expectation. Knowing that medical assistants are users but not buyers, I was able to supply this additional bit of context and within seconds revise ChatGPT's output so it bore a much greater resemblance to reality.
2. Efficiency. Never before have I been able to create over half a dozen customers personas in a single afternoon. While I typically spent ten to fifteen minutes per persona rather than three, the speed at which these personas came together was seriously impressive.
3. Quality and Detail. In many areas, Chat GPT's personas aligned well with research-based ones. While not highly detailed, ChatGPT did a respectable job characterizing target audience demographics, activities, and influences. If you're a "glass is half full" type, for the most part our virtual SMEs got the remaining categories partially right (wants/needs, pain points, goals, what success looks like). Moreover, ChatGPT proposed a few additional considerations that hadn't arisen during our customer conversations. A clear instance of this was seen with the 'Geraldo Home Chef' persona. Our Gen-AI experts raised difficulty sourcing foreign ingredients as a significant issue. This is a concern worthy of exploring with target users to ensure their satisfaction with the recipe selections our voice-controlled cooking assistant offers.

Generative AI Negatives:

1. Excluding the Customer. In all cases, ChatGPT failed to include an actual representative of our target audience on its panel of subject matter experts. Fortunately, Daniel's interactive prompt offered the opportunity to modify the SME line-up, so I added a customer representative each time.
2. Missing Crucial Drivers. As noted, ChatGPT missed the boat on real life medical assistants' needs, pain points, feelings, goals, and priorities. Similarly, failing to discern emotional impacts and mistaking priorities was a consistent finding throughout the numerous client personas I evaluated. One notable example involved an information technology administrator persona whose job was to keep his large organization's cloud computer systems running and provide technical assistance to the company's employees. Even with copious re-prompting, ChatGPT's responses were superficial. ChatGPT overlooked critical fears and challenges that IT administrators face like the immense pressure they experience when systems fail and the anger and despair that ensues when their supplier's support response fails to reflect that urgency. Across personas, our AI subject matter experts' contributions often struck me as pollyannaish, tone deaf, or oblivious to our audiences' real life goals, needs, and pains. Rather than providing insights based on our personas' perspectives as directed, our AI experts repeatedly interjected what appeared to be their own viewpoints instead. This mirrors a common problem encountered with real-life SMEs who, instead of representing the customer, inadvertently volunteer their own perspectives. So, while our Gen-AI personas got a lot right, they consistently missed the good stuff--the priorities, emotional drivers, goals, and behavior motivators that make personas actionable.
3. Lacking Employee Insight. Within my convenience sample, there were a small number of employee personas. (By this, I mean personas that characterize a key employee group within an organization as opposed to its external customers). For my sample, ChatGPT-generated employee personas were noticeably inferior to the ones it produced for customers. For example, ChatGPT showed virtually no understanding of the perspectives, obstacles, and requirements of a specific HR specialist group whose role centered around enterprise employee development. As a rule, I expect Gen-AI employee personas will be less reliable and offer substantially less insight compared to its customer personas due to a scarcity of data on internal viewpoints on the internet. While this bears further assessment, this finding makes sense to me. From what I've observed, employees tend to share fewer details about specific workplace experiences online, likely due to project confidentiality reasons and fear of potential backlash.

Conclusions

My foray into Gen-AI persona creation was filled with surprises. At times, I was impressed by how well ChatGPT personas aligned with real life examples. While ChatGPT demonstrated remarkable efficiency and ease of use, there were, however, significant trade-offs in depth and accuracy. In my opinion, ChatGPT's greatest weaknesses throughout this test were its limited ability to accurately surface human emotion and identify what mattered most to real people.

Based on these tests and observations, I'm still every bit as concerned with the risks of Generative AI personas I outlined in Part One of this series. I still favor a persona-making approach rooted in customer research--a mix of interviews, structured and unstructured customer data, and subject matter expert input. I still strenuously caution against replacing research-based personas with Gen-AI proxies...

...But I recognize Potential.

While it begs further assessment, for organizations seeking a holistic understanding of their target audiences, a blend of both methodologies may offer a balance of speed and depth. That said, I'm not so convinced that personas created with generative-AI sources like ChatGPT, Bard, and the like will prove to be valuable. It's essential that we preserve the human dimensions that account for the genuine value customer personas bring to business. Since customer validation is necessary, it remains to be seen whether a hybrid approach will translate into appreciable time savings for persona creators. And while expedient, in my opinion, it's not enough to present our AI-generated personas to our target audience, asking them whether our conclusions accurately reflect their views. In my experience, a great many people will nod their heads and agree. Agreeing takes the least cognitive effort and is more comfortable for criticism-averse individuals. In the end, I'm still not supportive of personas in minutes with Gen-AI tools like ChatGPT, but I am incredibly excited about the potential of internal, secure AI systems to find insights within confidential company data, using this to produce personas with real customer data in a fraction of the time. Exploring the possibilities here makes far more sense to me than personas in minutes with ChatGPT.

Thank you for taking the time to explore this important and timely topic. I hope you'll join me again for the final installment where we'll take the next step, discussing ways we might integrate Generative AI with customer research to maximize benefit and mitigate the risks. Stay tuned!

Photo 166573571 ? Viktor Kozlov | Dreamstime.com