Why sentiment scores don’t tell the whole story

A customer visits a U.S. Bank branch to ask about a promotional offer of $400 for opening a checking account online. The personal banker assures them they can receive the bonus even if they open the account at the branch.

The customer fulfills their eligibility requirements, yet the $400 never arrives.?

After several interactions with U.S. Bank’s complaint line and the branch, the customer is told they’re ineligible for the bonus because the account wasn’t opened online. Fed up after numerous attempts to negotiate with the bank across channels, the customer takes their feedback to the Consumer Financial Protection Bureau (CFPB) — and shares the following complaint.

Would you categorize that customer’s narrative as positive, neutral, or negative?

Inside the complex world of scoring sentiment

Financial services institutions are testing and adopting Large Language Models (LLMs).?There are different methods for building algorithms that detect sentiment in the market; and the approach, the data sources used, the desired outcome definition, and the application of the algorithm determine its accuracy, relevance, and effectiveness.

Inside the PositivityTech? platform, we run multiple algorithms on customer complaint data to reveal potential risks and to drive preemptive management actions.

• The Sentiment Score ranks the sentiment of a complaint narrative from very negative to very positive, on a scale from -1.0 to +1.0, with +1.0 being the highest level of satisfaction. Using an open source algorithm and lexicon, this score was developed from multiple industry data sources.
• The Severity Score ranks the level of customer frustration from very negative to very positive on a scale from from -1.0 to +1.0, with -1.0 being the highest level of frustration. Designed to identify future risk, PositivityTech’s proprietary?Severity Score uses a domain-specific algorithm and lexicon and was developed specifically on financial services data.

Applying these scores to the frustrated U.S. Bank customer narrative above reveals opposite results:

• Sentiment Score: +0.9977 (Very satisfied)
• Severity Score: ?0.9709 (Very frustrated)

These two scores identify unique variances in customer narratives and used concurrently, enable specific and targeted actions to remove friction, prevent mismatched expectations, and increase customer engagement.

Together, these algorithms can and should inform U.S. Bank’s response.?

The growth opportunity of conflicting score outcomes?

Of all the CFPB complaints in the last 12 months, 46% of complaints with the highest level of frustration (PositivityTech’s Severity Score) are also scored with the highest level of satisfaction (Sentiment Score).

Four steps to gain value from these different score results:?

1. Identify why the score results differ.
2. Understand and prevent the risk of misclassification.
3. Avoid non-relevant messaging and actions by using the Sentiment Score alone.
4. Implement targeted actions and benefit from the insights derived from each score.

Results vary by product, customer issue, and the topics within customer narratives.?

Sentiment scores don’t tell the whole story

Institutions must rely on the accuracy of their tools to take management action.?

If you’re leaning on LLMs without introducing complementary capabilities, and human or domain expertise, you risk misclassification. This can lead to further friction or regulatory impacts, and also cost you a lot of money.

In a recent LinkedIn post, Microsoft’s Satya Nadella stated, “As AI becomes more capable and agentic, models themselves become more of a commodity, and all value gets created by how you steer, ground, and finetune these models with your business data and workflow…”

Entrepreneur turned Investor Tony E. Kula responded, “Those who understand how to LEVERAGE HUMAN INTERACTION will be able to build capital efficient and profitable businesses like never before. The others will be taken over by co-pilot… Time to double down on human to human.”

Be ready to act upon your customers’ issues with PositivityTech’s Algorithms — and get ahead of risks before it’s too late.?

Explore how your customers really feel by reaching out to me at marcia.tal@positivitytech.com. I look forward to helping you transform negatives to positives.?