GenAI lessons from the frontlines: How Bolt automated customer service

Bolt is one of the few bold companies that adopted and innovated with Generative AI (GenAI) early and quickly. For those unfamiliar with the company, Bolt's services focus on the needs of city dwellers. It offers ride-hailing services, scooter and bike rentals, and food and grocery delivery services across Europe, Latin America, Asia, and Africa. Domino's Kjell Carlsson, Ph.D. was fortunate to sit down with Mikhail Korolev (Listen to the podcast now), Bolt Food's data science team leader, to discuss his experiences with GenAI technology. And while most companies are still testing GenAI on internal use cases, Bolt uses generative models with customers. Most importantly, the company achieved 80% cost savings while improving customer experience. Let's see how!?

Bolt's customer service opportunity

Like many rapid-growth companies, Bolt is under pressure to maintain high levels of customer service. Hold times to speak with agents can stretch and frustrate customers. Worse for Bolt, the cost of growing its customer support team can significantly negatively impact the startup's bottom line. Unsurprisingly, Bolt's team identified customer support automation as an improvement opportunity.?

Operationalizing a transformative GenAI application

Bolt turned to OpenAI's GPT API as the basis of its generative solution. The company established a pipeline with the following components:

1. Support case category classification: Bolt created a predictive model that picks from 300 possible request categories. This model passes all request categories to the generative model except for safety-related issues. Bolt chose to use this internal predictive model for the classification step as it requires deep knowledge of its support case resolution history.
2. Resolution logic: The application then passes the customer request to OpenAI's API. The goal is to identify possible resolutions to the request based on Bolt's guidelines. API requests include information about the customer's order. Details include time, location, services, and other related information, excluding personal data. Along with the request, Bolt passes the API a JSON schema containing a closed set of response options to choose from. For example, the API can respond with a discount coupon or ask the customer for additional information. The schema helps guide the API's decision-making logic regarding possible actions.?
3. Finalize response: Bolt passes the API's response to an internal model to make the final decision.?
4. Human response: The decision is finally passed to OpenAI's API again to wrap the response in human-friendly language.

What customers think of Bolt’s GenAI-driven customer service

While Bolt did not specifically poll its customers, its automated support tools received superior satisfaction feedback than the company's human support agents. There may be several reasons for this:

• Human support agents try to offer resolutions they perceive as more compliant and 'safer' with Bolt's guidelines.?
• Humans respond using formal language, while the GenAI solution uses a friendlier, casual approach.?
• And while GenAI offers more variety in its responses, it remains more consistent in its case resolution logic than a human.?

Helping the bottom line and customer experience

Bolt has several thousand support agents via its contact center partners. GenAI, though, is helping Bolt save money and advance efforts to cut support headcount by half. The share of requests handled by automation is increasing, a stark improvement compared to Bolt's previous, non-GenAI approach.?

The success is encouraging Bolt's team to expand its use of GenAI technology to image generation. The restaurants that use Bolt for delivery do not always provide images of the dishes they have on their menus. Bolt can now generate those images for them. Restaurant owners can approve whether to use the generated images or not. The idea is to help restaurants avoid food photography expenses and, ultimately, help sales.??

Building trust in a GenAI workflow

But how do you build trust with Bolt’s leadership team that GenAI can work without harming the company’s reputation? That's not an easy task. Many view machine learning models as black boxes that are difficult to understand. LLMs are even more opaque due to their sheer size and complexity, and there is always some randomness in OpenAI's API responses. Such variability occurs even when using the API's controls for creativity. Further, Bolt found out, the model’s attention is limited. If developers add as few as two new lines of text to prompts, the model can behave differently, “forgetting” information earlier in the prompt.?

So, how do you build that trust? Bolt focuses on extensive testing using a validation dataset. The dataset contains two thousand support cases with responses that comply with Bolt's rules and have a positive customer outcome. Whenever Bolt changes its prompts, it tests its pipeline's accuracy and compliance with its guidelines using this validation dataset.?

Best Practices

So, in summary, Mikhail offered us hard-earned GenAI best practices:

1. Make sure you have a validation dataset.
2. Test generative AI responses extensively before using them with customers.
3. Pay time and attention to prompt engineering. Be prepared for a lot of trial and error.?
4. Have a contingency plan ready in case the API you rely on is no longer available or stops behaving in the expected fashion.

For all the excitement surrounding GenAI, real-world examples of GenAI being put into production are still few and far between. Lessons like Bolt's are invaluable for the AI community and all future efforts to unlock the potential of GenAI. We are grateful to Mikhail for sitting down to speak with us and for sharing such valuable insights. If you would like to share your story and experience with GenAI technology and how you use it in production, please reach out to Kjell Carlsson, Ph.D. or Yuval Zukerman . In the meantime, please visit Domino’s GenAI page to learn more about how we enable our customers to harness the technology. If you want to get started with GenAI, please visit our developer-focused page, Sprint Zero.