We Screwed Up and We Didn’t Lose a Single Customer

Within one month of a specific software release – one that didn’t go well – almost 60% of our customer base was identified as being at risk. This release was so bad that we eventually called it Voldemort – “he who should not be named.” When I see a colleague from that time, all I have to do is say the name of that release and we both shudder.?

But we got through it. We learned from it! And you can too.

Background:

We had an on-premise product with quarterly releases. We needed to make some significant data structure changes and had been planning this release for a while. We deployed the release and made it available for installation. We released it on a Tuesday and that Wednesday, our support volume was 50% higher than forecast.

There were two problems with the release:

1. We didn’t adequately prepare customers for the database changes. They were significant and our lack of preparation created change management issues for the customers.
2. We made some pretty significant errors in the database joins which caused data integrity issues. As one specific example, the email addresses for Grey Watson were on Molly Jones’ record and vice versa.

As you can imagine, customers were outraged…and vocal. It was the biggest social media hit we had ever experienced. Industry analysts asked our CEO about the issue at a public event. And competitors leveraged the issue as an opportunity to attack – they created targeted campaigns highlighting our failures.

It could have been catastrophic…and it almost was. Instead, we approached it as an opportunity to benefit from the service recovery paradox where “customers can often be more loyal to your business after they have experienced a service failure, than if it had never happened in the first place.”? Here’s what we learned.

First and foremost, we addressed the issues.

We corrected the database join issues and we created resources for the customers to understand and deploy the changes including manuals, videos, and knowledgebase tips.

The key is how we addressed the issues. We approached it with humility, transparency, and accountability. This approach was key in regaining trust.

1. We owned up to the mistakes we made. We didn’t sugarcoat it or spin it. We didn’t hide behind phrases like “undocumented features”. We screwed up and we were going to fix it.
2. We shared our plans to address the issues with proactive messages and a status page that we committed to updating in real time, with at least weekly updates.
3. We hired temps to help with the support volume.
4. We proactively called customers and updated them on the status and our plans. We staffed this by partnering with our colleagues in other departments, i.e. we commandeered their teams. And when we called those customers, we tracked their sentiment.
5. We armed our teams with everything they needed to talk to customers about the challenges including resources, scripts, and status updates. We also gave them simple escalation paths to get help when needed.
6. We didn’t blame, ever. Which teams caused the issues didn’t matter. It never does matter to customers. We are the company to the customer; we are accountable as a single team.
7. We learned. While we didn’t focus on blame, we did take time to understand how this happened, what went wrong. And we put measures in place to apply those learnings.
8. We were accessible, more than accessible. We were out front and center with our customers. We had fireside chats. We went on Facebook and Twitter and responded to every post. I shared my direct email address and phone number and responded to every call and email.
9. Oh…and we apologized. This is important. Sometimes we forget this important step. Or we are too nervous to apologize because we think it will open us up to liability.
10. I couldn’t just stop at 9, so I will use this as a reminder. There is no recovery if you do not first fix the issue. Resolving the problem using the steps above was critical – it laid the foundation so customers would be open to listening to us when we asked for forgiveness.

Rebuild Trust

The key behind the service recovery paradox, the reason it creates increased loyalty is because it demonstrates to your customers how you handle an emergency. It gives them faith that when something goes wrong in the future, you will take care of it. Most times, addressing the issue quickly and fully will earn their trust. However, in the case of Voldemort, we recognized that we needed to do something to prevent a major attrition event. We had to make up for the frustration, lost time, and possibly even lost revenue we caused our customers. And if we did it correctly, we would regain their trust!

To rebuild the trust, we created a customer loyalty program with 4 perspectives: Escalation Management, Escalation Prevention, Customer/Site Retention, and User Satisfaction/Retention.

For each perspective, we identified the following items:

• Impact: # of Customers/Users impact; include high-level segmentations that are critical to the company: ARR, risk level, industry, etc.
• Plan: Plan to address the need
• Financial Impact: Estimated cost
• Status: Current status

Focus on the User

Let’s dive into User Satisfaction/Retention. Many companies think about the first 3 perspectives. But I have rarely seen programs that account for the user. How were they impacted? And if they were your champion, what happened to their credibility? What happened to their view of your company? Are they still your champion?

In this case, we determined there were over a thousand users that were personally impacted. Over a thousand users who felt unseen, who felt like we didn’t care. Our solution was to truly see them, to recognize that they were impacted.

For each user, we sent a box of local cookies and a handwritten note apologizing for the issue and thanking them for the opportunity to regain their trust. And while you can never underestimate the power of a cookie, the handwritten note was what resonated most with our users. The leadership team partnered with us in writing the notes. They each wrote about 30-40 notes; using a script we provided as the basis while adding a personal touch. Many of them received emails from the users thanking them for the note.

Direct acknowledgment from a vice president at the company, not just a generic email, made all the difference in the world! We see you.

Did it work?

At 12 months and 18 months, we reviewed the account status of the customers who were impacted by this release. We determined two things.

1. The retention rate for that set of customers was higher than the average retention rate.
2. For impacted customers who churned, Voldemort was not mentioned as one of the reasons for the non-renewal.

So while my colleagues and I may have PTSD thinking about Voldemort, and our customers certainly wouldn’t want to go through that again, we proved the service recovery paradox.

And that cookies always make things better.