My (personal) Thoughts on The CrowdStrike Debacle!

I woke up early on Friday morning for my 6:15 EST itinerary from JAX to ORD to SFO on United. Even before I left the hotel, I had alerts from both CNN, our CEO and others and figured it would be a rough day. Well, 2 days and 12 hrs, an Uber from JAX to ORD, 2 extra hotel nights, 2 cancelled, 2 delay, 2 missed connections and dozens of back up reservations, I finally made it home, tired, frustrated and sick!

Now with time to reflect and having read both technical explanations of the failure, a myriad of the CrowdStrike posts and apologies, and much thought, I have at least calmed my annoyance and have observations that I hope are worth sharing, as a) an impacted consumer, b) a technology marketer and c)someone with 20 yrs in and around the cybersecurity industry. So FWIW, here they are

As a Impacted Consumer

Man, what a nightmare. Where was the help, where was the information, why can't you get me home! Frustration and annoyance. Knowing what I know, I saw the dreaded "blue screen of death" and knew it was going to be a long day (never imaged a long 3 days!). This is no easy reboot I knew. At about 10am we started seeing people on ladders updating screens at gates...hmmm. Not only that, Starbucks was closed! No #TripleVenteDryVanillaSpiceLattes, where are the riots! Seriously though, helplessness is what it felt like, my United App was wrong, until it wasn't, the counter people knew nothing, and my travel agent chat staff didn't really understand what was happening for 2 hrs, and by then the system was totally fffd.

I will say, the Lake Nona Wave Hotel with the adjacent Lake Nona Performance Center (a gym, not a concert hall) made for quite a nice day of being stranded in Orlando!

Final tallies:

• Out of pocket travel costs incurred probably at >$1000 between ground, meals and 2 nights hotel.
• Personal productivity and days lost - ~47 hrs (52 delay hours - 5 nice hours at the LNPC)
• About 10 booked and cancelled alternative flights

Question: Where can I join the class action lawsuit, and who can we sue?

B - As a Tech Marketer

I have a lot of empathy for the team of marketers at CrowdStrike. What a crisis to manage. What a brand nightmare. What a challenge. Carilu Dietrich did a very nice job of evaluating the crisis management done by the CrowdStrike team. I won't rehash her v. positive review but you can read it here - https://www.carilu.com/p/crowdstrike-crisis-comms-case-study

Now maybe it's because I was personally impacted by this, I do see it a bit different. I reviewed many posts and videos by the CrowdStrike execs, and I found one thing dramatically missing; true empathy and ownership of the impact of the outage NOT on the CrowdStrike customers, but on the end consumers; the millions stranded in airports, the thousands of cancelled surgeries, and even those who couldn't get their beloved #TripleVenteDryVanillaSpiceLattes!!!

In all seriousness, I suspect the CrowdStrike lawyers cautioned taking any real ownership of downstream impacts so that they are better ready to defend the inevitable lawsuits from both their customers and downstream consumers. To me, that's a big brand miss. Time will tell how the Brand recovers, but me personally would be looking for alternatives if I were a decision maker, and NO, my company Aryaka DOES NOT COMPETE with CrowdStrike, I have no business pony here, but it feels like CYA legal trumped brand here a bit! It will be interesting to watch this play out, CrowdStrike is a leader, but has many credible and sizable competitors; how will brand and response impact share??? The market, as always, will be the real judge, as customers vote with their $$s!

C - As Someone with 20 Yrs in The Cybersecurity Industry

Overall, this makes me sad. I worked with CrowdStrike founder Dmitri Alperovitch at Secure Computing and then McAfee where he met co-founder and now CEO George Kurtz . And I have known many who worked there and have greatly admired their company's success, and they have done IMHO some of the best technology marketing of the last several decades. BUT, our mission as Cybersecurity vendors is to keep our customers safe, but that is irrelevant if they are not "in business". If this were a hack, it would be even bigger news, but the damage is the same.

First - Regardless of how well they respected CrowdStrike is/was, this is a failure of an incredible magnitude. Now, I am neither a DevOps or SecDevOps or ProductOps or Release or QA Engineer or manager. But I am still flabbergasted that this one line code error, which caused such a "hard crash" got released. Hopefully we will get a full accounting of this from CrowdStrike so we can ALL learn, but even so, customers will and should be more cautious of new releases. This is BAD for everyone. How do we restore this lost confidence?

Second - Taking a customer of CrowdStrike perspective, and here I must admit I don't know if you can, but I would certainly be turning off auto-update, this creates some security risk (I think small), but until I had confidence restored, I think I'd feel compelled to do so. Also, should endpoints be tiered as trivial, back office, and operational, and should I also update in that order. If United had updated their back office systems, found this crash, and not their operational ones, I might have got home in time. I am also not an ITOps expert, but I'd think a lot of endpoint managers are asking these very questions right now!

Third- This is a nightmare for any vendor. We live in a highly interdependent, extremely powerful yet fragile technology matrix. Outages like this hurt not just vendors, not just their customers, but society. We all could be the next CrowdStrike, and every vendor I know whose products are "inline" should be checking dependencies, building contingencies plans, shoring up QA testing and release processes., and from what I hear, that is happening in many companies as we speak, I know it is at mine. We owe it to not just our customers, but to everyone.

In the last 5 days I have been frustrated, tired, angry, hangry, disappointed and sad. But let's keep it in perspective, it was an adventure, and just another chapter in my #SummerOfHardThings :). I wish you all safe travels, safe software releases, and safe updates!!! - Ken