A network configuration error made Google unavailable to up to 5 percent of the internet

Nov 7, 2012 14:15 GMT  ·  By

While it may not look like it, the internet is quite vulnerable. That's because there's no central authority, there's no one controlling it, it's just a network of networks that relies on everything working as it should and everyone behaving as they should.

This actually works, most of the time. And the decentralized system is the internet's biggest threat, while any single error from one player can affect the entire web, not a single country or organization can effectively bring it down or control it.

Case in point is an error at an Indonesian ISP which managed to bring down Google, well technically bring down connections to Google for a short while in the region.

In total, between 3 to 5 percent of the internet's population was affected by the error all around Southeast Asia but not only.

In fact, CloudFlare, which noticed the problem and tracked it down, is located in California a short distance from Google's data center there.

What happened was that Moratel, an Indonesian ISP was routing traffic bound for Google through its servers and sending it to a dead end.

One CloudFlare engineer noticed that connections to Google were not working, not even DNS connections to Google's own public DNS service.

He then saw that his attempts at reaching Google were directed via Indonesia, which should not happen since he was in California.

All of this was pinned down to a hardware failure that led to a software failure. While the ISP was aware of the hardware failure and working on it, it was not aware that erroneous routing configurations "leaked" out of its network and into the global internet.

The ISPs upstream provider, PCCW, started relying on the erroneous routes provided by Moratel and started spreading those to other routers. This is how the internet is supposed to work, but this relies on the data provided being accurate, which was not the case here. The problem was quickly solved, with no involvement from Google, but the larger issue remains.