Containing the Networking 'Blast Radius'

As the world relies more and more on cloud computing for its day-to-day functioning and real-time communications, the availability & consistency of ‘distributed cloud computing’ play a very vital role. Lately, there has been a spate of large-scale catastrophic outages in North America, Hyperscalers, Asia, Europe, and very recently Oceania with far-reaching impacts on society and commerce. Everything that can go wrong went wrong. Due to complex interactions among multiple faults (e.g. TE controller removing routes/going offline, incorrect switch native IGP configs, delayed controller response) the faults snowballed and cascaded uncontrollably. Even after fixing the original root cause, aftereffects continued. The majority of the aforementioned outages can be traced back to networking, with many parallels with removal/misconfigurations in route filters which allowed route leaks to overwhelm internal routers, wrong circuit breaker settings (Maximum-Prefix), and the loss of Control Plane triggering spikes in IGP/BGP routes advertisements. Today there is a lot of industry focus on containing the Blast radius (extent of the impact from a fault), as it is nearly impossible to eliminate all fault scenarios.

?As can be seen on Fig.1 below, the core tenants of Blast Radius minimization in Distributed Cloud Computing are achieved by way of containment. The regions are geographically isolated and maintain a ‘share nothing’ philosophy. Each region is then subdivided into independent availability zones (AZ), which are built in a certain proximity to each other for latency and synchronous replications. Furthermore, cell-based architectures and shuffle shading techniques are used to further contain fault propagation and reduce blast radius. Given the homogeneity in the DCs, implementing these ‘Blast Radius’ containment techniques has been relatively easy and effective. However, the same level of design rigor and operational discipline has not yet been implemented in WAN underlay networks, which are often a collection of heterogeneous networks from multiple service providers and Hyperscalers

Below are some of the WAN initiatives that are being adopted to help contain Blast radius

?WAN ‘Blast Radius’ Containment Initiatives

• ?Traffic Engineering (TE) control plane isolation: Cloud computing networks moved to centralized global control plane for better network efficiencies and throughput. To contain Blast Radius, the TE domains are now being sliced to sub domains with their own independent sub-controllers (instead of controller replication). Microsoft was one of the early adopters of this strategy with their BlastShield implementation in the WAN and now claims 60% reduction in loss of traffic from controller failures.
• Network Element (NE) isolation: Core WAN routers now begin to operate at 50-200T fabric capacities, requiring the highest level of availability. WAN router misconfigurations and the loss of control plane have been one of the main causes of recent widescale outages. Adding further optical layer functional loads to the same routers can increase the blast radius exponentially. Although there could be some cost savings from IP-Optical convergence, operators need to strike the right balance between Blast Radius and cost savings.
• Operations Rigor: There is an increased need for rigorous operations discipline that includes code reviews, testing, staggered deployments, root cause analysis, and also the use of digital twins to stress test deployment procedures and fault scenarios. Furthermore, operational automation will also be key to minimizing human errors.

?The world today runs on distributed cloud computing and the need for high availability under network partition tolerance is paramount (in addition to Scaling, Security, and Sovereignty).

?As we’ve witnessed recently, an outage in ‘cloud computing’ going to not just inconvenience you from access to your favorite social media sites, it can disrupt your basic phone lines, paralyze the transport and location services, and disrupt the communications to the emergency services. As outlined above, WAN underlays that inter-connect distributed cloud computing need more improvements in terms of design and architecture in containing the Blast Radius of ‘distributed cloud computing.’

?Ryan Perera

Opinions expressed in this article are the author's own