CrowdStrike's Global Outage: Lessons learned from the Update Failure and the Agile vs. Waterfall Debate

Dr. Imran Khan

Introduction

On Friday, a routine update to CrowdStrike's widely used cybersecurity software caused computer systems to crash globally. Security experts suggest that the latest version of the Falcon Sensor software did not undergo adequate quality checks before deployment, leading to significant disruptions across multiple sectors.

The Incident

The update, intended to enhance security against hacking by updating the threats it defends against, instead introduced faulty code. This resulted in one of the most widespread tech outages in recent years for companies using Microsoft's Windows operating system. Impacted sectors include global banks, airlines, hospitals, and government offices.

CrowdStrike released information to fix affected systems, but experts warn that manually weeding out the flawed code will take time.

Expert Analysis

Steve Cobb, Chief Security Officer at Security Scorecard, remarked, "Potentially, the vetting or sandboxing they do when they look at code, maybe somehow this file was not included or slipped through." Cobb's own organization also faced system impacts due to the issue.

Problems emerged quickly after the update, with users posting pictures of "blue screens of death" on social media. These blue screens displayed error messages, highlighting the severity of the issue.

Security researcher Patrick Wardle identified the faulty code as being in a file containing configuration information or signatures, which detect specific types of malicious code or malware. "It's very common that security products update their signatures, like once a day... because they're continually monitoring for new malware and because they want to make sure that their customers are protected from the latest threats," Wardle explained.

He further noted that the frequency of updates might have led to insufficient testing. "The update's problem was in a file that contains either configuration information or signatures," Wardle said. "Such signatures are code that detects specific types of malicious code or malware."

John Hammond, Principal Security Researcher at Huntress Labs, suggested a safer approach: "Ideally, this would have been rolled out to a limited pool first. That is a safer approach to avoid a big mess like this."

Agile vs. Waterfall: A Critical Perspective

This incident brings into focus a significant question in the software development and deployment world: Agile or Waterfall?

Agile methodologies, known for their iterative approach and quick releases, can sometimes lead to inadequate testing if not managed properly. The high frequency of updates, as seen with CrowdStrike, while aimed at providing the latest protections, may have contributed to the oversight in quality checks. However, pushing (deploying) code to production on the very first day (a one-day sprint ??) by the CrowdStrike team doesn't reflect true Agile practices. It might be a startup approach, often leading to failure as they prioritize fast failure over steady progress.

On the other hand, the traditional Waterfall model emphasizes thorough testing and review phases before any deployment. While this approach might seem slower and less adaptable to rapid changes, it ensures that each step is meticulously completed, potentially preventing such large-scale issues.

The CrowdStrike incident highlights the need for a balanced approach. Perhaps an Agile method, which blends the rapid response capabilities of Agile with the rigorous testing protocols of Waterfall, could serve as a solution. This hybrid model would allow for swift updates while ensuring each release undergoes comprehensive quality assurance.

Broader Implications

Other security companies have had similar episodes in the past. McAfee's buggy antivirus update in 2010 stalled hundreds of thousands of computers. However, the global impact of this outage reflects CrowdStrike's dominance in the cybersecurity market. Over half of Fortune 500 companies and many government bodies, including the top U.S. cybersecurity agency, the Cybersecurity and Infrastructure Security Agency, use CrowdStrike's software.

Conclusion

This incident underscores the critical importance of thorough testing and quality checks in software updates, especially in the cybersecurity domain. As companies and organizations continue to rely heavily on digital infrastructure, the need for robust and reliable security solutions becomes increasingly vital. Moving forward, adopting a hybrid approach that incorporates both Agile and Waterfall methodologies could help mitigate the risk of widespread disruptions while ensuring timely updates.

#CyberSecurity #TechOutage #CrowdStrike #SystemCrash #SoftwareUpdate #DataProtection #Infosec #TechNews #Agile #Waterfall #SoftwareDevelopment

About the Author

Dr. Khan is a dedicated educator and advocate for sustainable business practices, with extensive experience in guiding startups and nurturing talent in the tech community. Connect with Dr. Khan for insights on creating resilient startups and fostering a thriving programming community.

With over twenty-five years of experience in industry and academia, Dr. Khan is an international award-winning researcher and the founder of five companies. He earned his Ph.D. in Computer Science, specializing in developing a framework with simulation for executable system architecture.

Currently, Dr. Khan serves as an Assistant Professor at the School of Mathematics & Computer Science (SMCS) and the Center for Entrepreneurial Development at IBA Karachi. He is also the Program Lead for the International Entrepreneurship Summer School at IBA. He played a pivotal role in establishing Pakistan's startup ecosystem through initiatives like INVENT, the Entrepreneurship Development Program, the Women Tech Entrepreneurship Program, Kids Entrepreneurship, the Gender Equity Program, and Family Business case writing and consulting.

During the global COVID-19 lockdown, Dr. Khan hosted a two-day Entrepreneurial Leadership online Bootcamp that attracted participants from 27 countries, receiving widespread acclaim and appreciation. His current research projects include the Digital Twin of Heart Patients, where he developed a complete framework for predictive analysis of cardiovascular disease (CVD) risk.