?? Is Generative AI Just Cool, or Does It Really Have an Impact? That's the big debate in tech circles these days.
A study led by researchers from Stanford University, MIT, and the National Bureau of Economic Research (NBER) sheds light on this question by examining the real-world impact of deploying generative AI in a customer support environment. Their analysis offers empirical evidence on how AI tools, specifically those based on OpenAI's GPT models, are transforming customer service operations at a Fortune 500 software company.
The researchers employed a mix of methodologies: a randomized control trial (RCT) and a staggered rollout, encompassing around 5,000 agents over several months. By analyzing 3 million customer-agent interactions, the study assessed metrics such as resolutions per hour, handle time, resolution rates, and customer satisfaction (Net Promoter Score). To understand the AI's impact over time, dynamic difference-in-differences regression models were used.
Here is what they found:
1. ?????????????????????? ?????????? ???? ????????????????????????: The AI tool led to a 13.8% increase in the number of customer queries resolved per hour, particularly benefiting less experienced agents.
2. ?????????????????? ?????? ?????????????????????? ??????: AI tools accelerated the learning curve for newer agents, allowing them to reach the performance levels of seasoned employees more quickly.
3. ???????????????? ???????????????? ????????????????????????: The AI deployment resulted in higher customer satisfaction scores (as shown by improved Net Promoter Scores) while maintaining stable employee sentiment.
4. ?????????? ?????????????????? ??????????: Interestingly, the AI support led to reduced attrition rates, especially among new hires with less than six months of experience.
5. ?????????????????? ??????????????????: The AI system reduced the need for escalations to managers, improving vertical efficiency. However, its impact on horizontal workflows, like transfers between agents, showed mixed results, suggesting more refinement is needed in AI integration.
6. ???????????????????? ???? ??????????????: The software wasn’t off-the-shelf; it was a custom-built solution tailored to the company’s needs using the GPT family of language models. This emphasizes the importance of context-specific AI applications for effective outcomes.
For leaders, managers, and AI practitioners, these insights are invaluable—highlighting not just the potential of AI, but also the nuanced ways it reshapes workflows, impacts employee dynamics, and transforms customer experiences.So, does generative AI really make a difference? According to this study, the answer is a resounding yes—but it depends on how thoughtfully it is deployed.
Link ?? to the paper: https://lnkd.in/ejhUfufz