Unraveling the ChatGPT Outage: Insights from OpenAI's Post-Mortem
This post-mortem analyzes the root cause of the widespread service disruption that affected all OpenAI services on December 11, 2024.
Incident Overview
All OpenAI services experienced a significant outage on December 11, 2024, beginning at 3:16 PM PST and lasting until 7:38 PM PST.
OpenAI operates hundreds of Kubernetes clusters globally. Kubernetes has a control plane responsible for cluster administration and a data plane to serve workloads.?
The outage was attributed to an internal change related to the rollout of a new telemetry service aimed at improving observability across OpenAI's Kubernetes clusters. This deployment inadvertently overwhelmed the Kubernetes control plane, leading to cascading failures across critical systems. At 3:12 PM PST, new telemetry service was deployed to collect detailed Kubernetes control plane metrics.
Root Causes
1. The configuration of the new telemetry service triggered simultaneous, resource-heavy Kubernetes API operations across thousands of nodes. This unexpected surge in demand overloaded the Kubernetes API servers, causing the control plane to fail in most large clusters.
2. The change was tested in a staging cluster without any issues. However, the impact only occurred in clusters beyond a certain size, which the testing environment did not adequately replicate.
3. DNS caching on each node delayed the detection of failures, allowing the rollout to proceed before the issue's full extent was recognized. Once the 20-minute DNS cache expired, services began to fail due to their dependence on real-time DNS resolution.
Remediation Efforts
Upon identifying the issue, OpenAI initiated multiple workstreams to restore service quickly. These included scaling down cluster sizes to reduce API load, blocking network access to Kubernetes admin APIs to prevent further requests, and scaling up API servers to handle the load. These efforts allowed for the removal of the problematic telemetry service, leading to a gradual recovery of the affected clusters.
Preventive Measures
To prevent similar incidents in the future, OpenAI is implementing several measures-
1.? Robust Phased Rollouts - Improved monitoring and phased rollout processes for infrastructure changes to detect failures early.
2.? Fault Injection Testing: Running tests to ensure the Kubernetes data plane can function independently of the control plane and to identify bad changes.
3.? Emergency Access Protocols: Establishing mechanisms to ensure access to the API server during high-pressure situations.
4.? Decoupling Data and Control Planes: Investing in systems to reduce the dependency on the control plane for service discovery.
5.? Faster Recovery Strategies: Implementing caching and dynamic rate limiters to facilitate quicker recovery process.