Bias is the new oil

Imagine how boring Thanksgiving dinner would be if people weren’t biased. The same turkey, Stove Top stuffing, sweet potatoes and Libby’s pumpkin pie every year. The same positive, uplifting discussions about how well the Federal government is run and 100% agreement on parenting styles year after year.? But people aren’t like that and neither are the massive corpuses that the big chatbots are trained on.

Using LLM bias to measure your market

Data scientists spent 50 years striving to reduce bias in AI models. For decades the challenge was to develop accurate models that weren’t too biased toward their training data. Now, all of a sudden, large language model bias is a first-class resource to manage, manipulate and exploit. Here’s an illustrative example. I asked three popular chatbots to simulate asking 1,000 U.S. adults to name a car brand, sum the results and calculate percentages. Then I compared those percentages to actual market share by units sold last year. The results are pretty good.

Gemini: 84% correlation, 1.7% average absolute error
ChatGPT: 81% and 2.6%
Claude:  74% and 2.2%        

Then I tried calling GPT-4’s API 300 times with temperature 2 (the maximum randomness), asking it to name a car brand, and compared those percentages to actual market share:?

68% correlation and 3.4% average absolute error

Still quite good. The same repeated API-call approach also works well for city populations (73% correlation) and pop music artists (90%+ correlation). Now the fun part… LLMs aren’t just one level of probabilities, they’re millions covering niches upon nichies upon niches.?

LLM personas

Large language models can “adopt” most any persona so I told GPT-4:

Prompt: ?You are a 35-year old living in West Texas. You like shotguns, rodeo, and dogs. Name a car brand.??

87% Ford, 10% Chevrolet and 2% Dodge. Definitely directly accurate.?

Next prompt: ?You are a 45-year old soccer mom living in Boulder, Colorado.? You like dogs, wine, and antiquing. Name a car brand.??

48% Volvo, 19% Subaru, 13% Mercedes-Benz and 8% Toyota.? Dead on.

One more: You are a wealthy stockbroker living in Miami, Florida. Name a car brand.

26% Ferrari, 19% Porsche, 17% Lamborghini and 10% Mercedes-Benz.

My point is that this can be done for many products, many regions, and many personas in seconds for pennies. No large panel survey technique has ever had this kind of capability and price point.

So what?

Twenty years ago businesses learned that internet natural search rank is a key to success and developed techniques known as search engine optimization (SEO) to improve it. SEO became the highest ROI of all B2C marketing tactics. Today your customers are slowly discovering a 1,000X more powerful information resource in the chatbots. At Touchpoint Strategies we’ve developed metrics to measure how businesses perform with this new resource. An example is a chatbot buzz score. It’s a business’s share of chatbot voice (as described above) divided by the business’s actual market share, minus 1 and times 100. Zero means that your bot share of voice is the same as your market share, 100 means your bot share of voice is twice your market share, and -50 means it’s half your market share. Here are some examples for brands you might know:?

Phones: Apple 52, Samsung -16

Cars: Subaru -86, Kia -82, Ford -12, Volkswagen 24, Ferrari 2,500, and Lamborghini 3,500

Music artists: Kanye West -68, Billie Eilish 0, and Taylor Swift 21?

I used July 2024 U.S. units in use for phones, 2023 U.S. units sold for cars, and Spotify songs streamed from January through June 2024 for music artists. Of course, precisely which market share metric is used makes a big difference. Lamborghini’s chatbot buzz score would be much lower, for example, if we used revenue rather than units.

Now what?

At Touchpoint Strategies, we’ve developed a methodology that uses multiple AIs to very rapidly assess your market situation and recommend actions to drive growth. If you’re a mid-market B2B or B2B2C company looking to accelerate growth, contact us at TouchpointStrategies.com.?

#genai #LLMs #marketresearch?