The App Update That Broke Everything
Paul Tekle-Tsadik ??????♂?
Head of Product | Solutions Architect | Software Delivery | Building financial services that are accessible and inclusive
Imagine this: you and the team spent months working on a software update, refining features, optimising performance, etc. The deployment is signed-off but within hours, your inbox floods with complaints. Core functionalities are broken and to make matters worse, there’s no rollback plan. This isn’t a hypothetical scenario, it’s exactly what happened at SONOS.
Here’s what they went through, what they learned, and how they're turning a setback into a comeback.
ROOT CAUSE
The update in was designed to enhance UX, introduce some new features, and improve overall performance. However, shortly after deployment, it became clear that the update had unintended consequences.
Core functionalities like device connectivity, voice control, and multi-room audio were disrupted for a subset of users. For a company like Sonos, whose reputation is built on seamless, high-quality audio experiences, this was a significant blow of the app's core functionality.
The immediate challenge was the lack of a rollback plan. A rollback plan is a set of steps that outline how to revert a system back to a previous stable version in case a new deployment encounters issues or fails. A fail-safe. In this case, the absence of that option meant the team were unable to quickly mitigate the impact on our users. This delay to get things back to normal frustrated users and eroded their trust.
To our users: thank you for your patience and understanding as we’ve worked through this challenging time. Your feedback has been invaluable, and it has inspired us to raise the bar for what it means to be a customer-centric company. We know that trust is earned, and we’re committed to earning yours back, one update at a time.
I’ve always believed that the true test of a product leader isn’t just in launching successful features or driving growth or revenue but how we handle the moment things go wrong.
The failed deployment was so significant, and it led to the departure of their Head of Product, among other internal changes.
FALLOUT IN LEADERSHIP
This decision (Head of Product's departure) was not made lightly, but it underscored a fundamental principle of leadership: accountability. Some mistakes cannot be undone apparently. When things go wrong, especially at this scale, leaders must take responsibility. This isn’t about pining the blame but owning the outcome and ensuring that the organisation learns from the experience.
This experience is a humbling reminder that even the most well-intentioned teams can miss the mark.
And in the case of Sonos, an opportunity for reflection, learning, and growth.
Leadership in product management (or any other department for that matter) isn’t just about celebrating wins; it’s about navigating challenges with integrity and transparency. It is also about willing to make tough decisions, even when they’re personal.
According to different sources, the departure of our Head of Product was a sobering moment for the team, but it also reinforced their commitment to doing better for their customers, fellow colleagues, and the future of Sonos.
领英推荐
LESSONS
Personally, reading about this incident has left residual haunting. And alternatively a powerful learning experience. Here are some of the key takeaways that I believe are relevant not just for myself, but for any product team:
No matter how confident you are in a deployment, having a rollback plan is non-negotiable. It’s a critical fail-safe that ensures you can quickly address issues and minimise disruption for users.
At the heart of this incident was a disconnect between user needs and the actual impact on users. Were the new features more important and did the users ask for it? Ensure that every decision is grounded in a deep understanding of your users’ needs and expectations.
In the early stages of the incident, communication with users was not as timely or transparent as it should have been. I've learnt that being upfront about what’s happening and what is being done to fix it is essential to maintain trust. Transparency is a must-have.
Moments like these test the resilience of our teams. They force us to confront our shortcomings, adapt to new challenges, and reinforce commitment (and trust) amongst team members.
THE SONOS RETURN
The company's main focus is on rebuilding trust with their users and ensuring that they never repeat the same mistakes. This is their approach:
FINAL THOUGHTS
As a product leader, I’ve always believed that the true measure of success isn’t just in the products being shipped, but in how we handle the moments when things don’t go as planned.
This incident at Sonos be a powerful reminder because it can happen at any time to any team. As well as nudge product managers to put their egos or fears aside in relentless focus on the customer.
Let this be a case study in the importance of planning for the unexpected, listening to your users, and leading with integrity.