#21 - Google went down

12th November, 2018,

ThousandEyes seen issues interfacing with G Suite.

Evaluating ThousandEyes Endpoint Agent details, his was affecting all clients at the ThousandEyes office. The blackout influenced G Suite, as well as Google Search and also Google Analytics. The most interesting thing was that activity to Google was getting dropped at China Telecom. For what reason would movement from a San Francisco office crossing to Google go the distance to China? It was detected that a Russian ISP in the rush hour gridlock way, which certainly started a few concerns.

A few ThousandEyes vantage focuses far and wide were revealing comparative surprising movement steering, all ending at China Telecom.

ThousandEyes BGP Route Visualization painted a fascinating picture. Activity from Paris to www.google.com made plans to 216.58.204.132. While Google reports numerous/24 prefixes to cover its IP address go, this location was not secured by a/24 prefix. Rather, it was secured by a/19 prefix. Suspicious declaration for 216.58.192.0/19 show up after about 12:45 pm PST with a convoluted AS way that included TransTelecom (AS 20485) in Russia, China Telecom (AS 4809) in China and MainOne (AS 37282), a little ISP in Nigeria.

The traffic paths mirrored the BGP AS Path, except all the traffic slammed into the great firewall, terminating at China Telecom edge router. 

This incident at a minimum caused a massive denial of service to G Suite and Google Search. However, this also put valuable Google traffic in the hands of ISPs in countries with a long history of Internet surveillance. Overall ThousandEyes detected over 180 prefixes affected by this route leak, which covers a vast scope of Google services. 

The origin of this leak was the BGP peering relationship between MainOne, the Nigerian provider, and China Telecom. MainOne has a peering relationship with Google via IXPN in Lagos and has direct routes to Google, which leaked into China Telecom. These leaked routes propagated from China Telecom, via TransTelecom to NTT and other transit ISPs. 

This leak was primarily propagated by business-grade transit providers and did not impact consumer ISP networks as much.

Today, MainOne tweeted out that the root cause of the problem was in fact due to a configuration error.

This episode additionally underscores one of the principal shortcomings in the texture of the Internet. BGP was intended to be a chain of trust between benevolent ISPs and colleges that aimlessly trust the data they get. It hasn't advanced to mirror the perplexing business and geopolitical connections that exist among ISPs and countries today. While confirmation techniques like ROA exist, few ISPs utilize them. Indeed, even organizations like Google with enormous assets available to them are not insusceptible from this kind of BGP spill or pernicious captures. MainOne took 74 minutes to either see or be advised of the issue and fix it, and it took around seventy five percent of a hour more for administrations to return up.. Most undertakings who don't have Google's range and assets will be unable to determine the issue as fast, which can altogether affect business.

BGP-related occurrences have been on the ascent. In April 2018 saw a baldfaced digital money heist including the capture of a whole DNS supplier (Route 53). Only a year prior to that, in April 2017, saw the Rostelecom BGP course spill which influenced countless business and budgetary administrations sites. While the ISP people group perceives the extent of the issue, and arrangements, for example, ROA and IRR separating exist, none of them are silver slugs and executing them dangers breaking the Internet. 

Without ensures, endeavors need to ceaselessly screen their BGP routes and recognize such occurrences rapidly with the end goal to relieve any administration effects to their business.