3rd Party Risk - Can it impact your brand and revenue genrating activities?

In 2024, several major outages were caused by third-party providers, highlighting the risks associated with relying on external partners for critical services.

CrowdStrike Outage (July 2024): A software update from CrowdStrike, a leading cybersecurity firm, led to a massive global outage affecting around 8.5 million Windows devices. The faulty update caused a system failure, leaving critical services like hospitals and 911 centers unable to operate properly, and disrupting banking and airline operations worldwide. The issue arose from inadequate testing of the software patch, showcasing the vulnerability of relying on third-party providers for essential infrastructure

AT&T Third-Party Outage: In 2024, AT&T experienced a significant service disruption due to issues related to a third-party provider. This outage affected communication services across various sectors, further demonstrating the risks tied to outsourcing key operational components to external vendors.

These incidents emphasize the importance of robust third-party risk management, especially as organizations increasingly depend on outsourced solutions for critical operations. As regulatory frameworks like the EU's Digital Operational Resilience Act (DORA) push for stricter oversight of third-party providers, businesses must be proactive in testing and securing their vendor relationships to avoid such disruptions

Incident Details:

- Date: October 2023

- Cause: A third-party fiber provider that supports AT&T’s network experienced a major fiber cut. This caused a cascading outage that impacted AT&T’s services across various regions in the U.S.

- Impact:

? - Widespread Disruptions: The outage affected business communications, enterprise applications, and emergency services that depended on AT&T’s network for connectivity.

? - 911 Emergency Services: In some regions, customers were unable to make 911 calls, raising significant safety concerns.

? - Business and Consumer Impact: Many companies experienced a halt in operations, as AT&T’s services are used by a wide range of industries, from finance to retail. For individuals, the loss of internet and mobile services caused communication blackouts.

- Resolution Time: AT&T worked to resolve the issue by coordinating with the third-party fiber provider, and services were gradually restored. Full service restoration took several hours, causing a ripple effect on productivity and business continuity.

?Risks Associated with Third-Party Providers Highlighted by the AT&T Outage:

1. Service Disruption and Dependency

?? - Risk: As seen in this outage, organizations relying on third-party providers (in this case, AT&T relying on a third-party fiber company) are vulnerable to disruptions outside their direct control. This underscores the risk of dependency on vendors for critical operations.

?? - Mitigation: Companies should evaluate backup providers and build redundancy into their infrastructure, such as secondary failover networks.

2. Operational Downtime

?? - Risk: Prolonged outages, such as the AT&T incident, can lead to substantial operational downtime. Businesses depending on AT&T’s services experienced delayed projects, communication blackouts, and loss of customer trust.

?? - Mitigation: Ensuring that contracts with third-party providers include stringent SLAs, and having a detailed incident response plan in place can help mitigate operational impacts.

3. Emergency Service Vulnerabilities

?? - Risk: One of the most serious concerns during the AT&T outage was the impact on 911 emergency services, raising questions about the vulnerability of critical infrastructure when dependent on third parties.

?? - Mitigation: Developing alternative routing mechanisms for emergency services and ensuring more resilient infrastructure for critical services is key to avoiding catastrophic failures.

4. Customer Dissatisfaction and Reputation Damage

?? - Risk: AT&T faced reputational damage as a result of the outage, with affected businesses and consumers expressing frustration over the lack of communication and transparency. Such incidents can erode customer trust.

?? - Mitigation: Proactive communication during outages and ensuring that customers are kept informed with real-time updates can help manage dissatisfaction and maintain trust.

5. Third-Party Risk Management

?? - Risk: This outage exposed the inherent risks of third-party service providers, especially when they have a direct impact on core business operations. In AT&T’s case, relying on a third-party fiber provider created a single point of failure.

?? - Mitigation: It’s crucial for businesses to conduct regular risk assessments of their third-party vendors and ensure adequate contingency plans are in place to handle potential disruptions.

?Lessons Learned:

The outages at Microsoft (CrowdStrike) and AT&T in 2024 highlight several critical lessons in third-party risk management and operational resilience:

1. Testing and Quality Assurance

• Lesson from CrowdStrike: The July 2024 CrowdStrike outage was caused by a software update that had not undergone sufficient testing. This faulty update led to widespread device shutdowns globally. Lesson: Comprehensive testing is crucial for updates, especially for critical systems. Organizations need to enforce stricter quality control on third-party patches and updates

2. Redundancy and Failover Systems

• Lesson from AT&T: AT&T’s reliance on a third-party provider for its fiber network led to significant service disruptions. Without robust redundancy in place, their operations were critically affected. Lesson: Businesses must ensure backup systems and alternative providers are available to mitigate the risk of single points of failure, especially in mission-critical services

3. Third-Party Vendor Oversight

• Lesson from Both: Both cases emphasize the need for continuous monitoring of third-party vendors. As seen in the CrowdStrike outage, even trusted cybersecurity providers can be sources of risk. Similarly, AT&T’s dependence on a single third-party fiber provider exposed vulnerabilities. Lesson: Organizations need strong vendor risk management programs that include regular audits, contract reviews, and risk assessments

4. Communication and Incident Response

• Lesson from AT&T and CrowdStrike: The speed and clarity of communication during these outages were critical. Delays in addressing the issues and informing users exacerbated the impacts. Lesson: Having a clear incident response and communication plan is vital. Organizations should prepare for immediate, transparent communication with stakeholders during disruptions

5. Regulatory Compliance and Preparedness

• Broader Lesson: With new regulations like DORA (Digital Operational Resilience Act) coming into effect, businesses are now required to have stressed exit plans for third-party providers. This shift encourages companies to be more proactive in managing third-party risks, ensuring compliance and preparedness for future outages

These incidents underscore the importance of resilient infrastructure, thorough vendor management, and proactive planning to mitigate third-party risks.