My 5 learnings from the CrowdStrike update

In light of the issues with the CrowdStrike Falcon update, I am reminded of a time about 5 years ago at Bose Corporation .

At the time, there were about 300,000 devices regularly polling the Bose Cloud for new updates. These updates were tested internally to ensure that the risk of the worst-case scenario of locking up or bricking customers’ devices would be minimized.

The only way for a customer to recover from a bad update was to reset the device manually.

Carbon Black used a similar Cloud/agent-based architecture as CrowdStrike , and at CarbonBlack new updates were released routinely using a staged release strategy. That is, instead of updating all customers at once, each stage was a subset of the install base and updates continued to more customers only after validating that things worked.

If there is an issue with a software update and it is caught in the staged release process, the benefit is that the impact of the issue is minimized.?

At Bose, it was not easy to convince the leaders to enhance the release process:

• We’ve never had a problem with updates, it’s not worth the effort to move to staged releases
• If a customer imparting problem happens, we can issue a statement on the website with? reset instructions

I got buy-in from an engineer for a proof of concept. After enhancing the update process metrics, we ran a test to prove that staged release process worked. The enhanced metrics helped us validate that updates had been applied successfully.

With this data, I was able to get support for implementing staged releases, which minimized the risks associated with new updates.

Here are my 5 learnings from the CrowdStrike update:

1. Relentless discovery: Get to the bottom of all assumptions (e.g. “We’ve never had a problem with updates”) and drive “relentless discovery” of systems end to end to understand how to minimize systemic risks (something I learned in Professor Steven Spear’s excellent Creating High Velocity Organizations course).
2. Leverage prior experiences: Use your prior experience and training to question status-quo to drive proven solutions, such as the example in this post.?
3. Identify and mitigate worst-case scenarios: Ask “Imagine we meet on the day after launch to discuss why the launch failed, what happened, and why?” This is an effective way of getting teams to identify risks that need to be mitigated before launch.
4. Don’t take no for an answer. Seek alternative supporters for ideas, and surround yourself with trusted and driven colleagues. Using smaller-scale experiments can help drive buy-in.
5. Learn from failures: While it’s important to celebrate success, it’s just as important to learn from failures and build safeguards to minimize risk.?Retrospectives are helpful for this.

Disclaimer: This post is only intended to share learnings based on my own experience. I have the utmost respect and admiration for all the companies referenced.