Ensuring System Reliability through Traditional Testing & Quality Engineering: Lessons from the CrowdStrike Outage

The recent CrowdStrike outage on July 19, 2024, disrupted operations across major industries, revealing critical vulnerabilities in our digital infrastructure. This incident underscores the necessity of traditional testing methods and highlights the shared responsibility of major tech companies, including Microsoft, in maintaining system stability.

CrowdStrike, a leader in cybersecurity, provides the Falcon Security agent widely used across various industries. On July 19, 2024, an untested update to the Falcon Sensor led to widespread connectivity issues and system reboots on Windows systems. This update also disrupted Microsoft’s Azure cloud platform and other cloud platforms like AWS, Google Cloud, IBM Cloud, and Oracle Cloud, which have a significant percentage of their virtual machines running Windows-based systems, further exacerbating the situation.

The faulty update impacted millions of Windows systems across the globe, highlighting Microsoft's role in ensuring compatibility and stability with third-party updates. This incident underscores the shared responsibility between CrowdStrike, Microsoft, and other cloud platforms. The outage disrupted operations in sectors such as airlines, banking, and media, highlighting the broader implications for global business operations and consumer trust. Both CrowdStrike and Microsoft faced financial and reputational damage. Businesses worldwide experienced operational challenges and downtime, underscoring the need for robust preventive measures.

Financial and Operational Impact

The outage resulted in significant financial losses and operational disruptions:

• IT Impact: Millions of Windows computers—personal computers, workstations, high-end servers, virtual machines, and cloud-based server systems—were affected globally.
• People Impact: The outage affected millions of users who rely on Windows-based systems protected by CrowdStrike, not counting the hundreds of millions whose every day life was disrupted severely.
• Global Impact: The issue had a worldwide impact, affecting multiple countries across different continents.
• Impact to Industries: Key sectors including airlines, banking, healthcare, and media experienced significant disruptions.
• Financial Impact: Exact figures are not available yet, but the overall financial impact is substantial. CrowdStrike’s share price dropped by 11.1%, closing at $304.96.

Preventive Measures and Lessons Learned

This event will go down in the history of Information Technology as one of the biggest incidents that could have potentially been avoided with a more conventional and traditional approach to software testing. Rigorous testing, better collaboration, and communication could have prevented this. Investing in high-quality standards in software development is crucial. The long-term benefits of prioritizing quality over speed far outweigh the costs.

Traditional Testing and Quality Engineering Methods

Traditional testing includes manual testing, regression testing, pre-production testing, and real-world scenario testing. These methods are critical for identifying and mitigating bugs that automated testing might miss. Reliability testing and other conventional methods ensure that updates do not introduce new bugs or conflicts. They also validate that updates do not disrupt existing security measures and user impact. Thorough testing prevents widespread disruptions by validating updates before release.

Neglecting traditional testing and quality engineering contributes to accumulating technical debt, which compromises system stability and security. Addressing technical debt is essential for preventing costly disruptions.

AI and Governance

While it's not clear how or if CrowdStrike leveraged AI in this instance, the reliance on AI-based systems, including automated coding, testing and decisioning , while beneficial in many aspects, may not always yield the best outcomes. The CrowdStrike incident emphasizes the need for robust IT and AI governance to ensure that automated systems are subject to rigorous oversight and traditional testing methodologies. Ensuring the governance of AI systems involves setting clear guidelines, conducting regular audits, and maintaining transparency in AI decision-making processes.

Conclusion

The CrowdStrike outage highlights the importance of traditional testing and quality engineering methods and the shared responsibility of tech giants like Microsoft in maintaining system stability. It is imperative for organizations to invest in robust testing frameworks and prioritize comprehensive quality engineering in their software development lifecycle to ensure reliability and prevent future incidents. Additionally, there should be a stronger focus on governance, particularly around AI, to mitigate risks associated with automated decision-making processes.

Manik Gupta

Nikhil Joshi