Fastly internet outage caused by bug with single customer

Yesterday, we saw many top ranking websites taken offline briefly due to an issue with Fastly, one of the biggest Content Delivery Networks (CDN) on the Internet. The issue was quickly fixed, although we weren't told exactly what went wrong. That changed today and as it turns out, one customer update managed to bring the whole ship down. 

Speaking with The Guardian, Nick Rockwell, head of infrastructure and engineering at Fastly, explained that the issue actually dates back to the 12th of May, when a software update introduced a bug on a specific customer's website.

“On May 12, we began a software deployment that introduced a bug that could be triggered by a specific customer configuration under specific circumstances. Early June 8, a customer pushed a valid configuration change that included the specific circumstances that triggered the bug, which caused 85% of our network to return errors.”

So the bug itself was introduced a month ago, laying dormant until yesterday morning, when it took effect and had a wide reaching impact across many sites, including the likes of Twitter, Twitch, Reddit as well as many news sites and even the UK government web page.

Discuss on our Facebook page, HERE.

KitGuru Says: The downtime didn't last too long, with most websites returning to normal within an hour of going offline. Still, it is interesting to see the domino effect that a CDN issue can have. 

Become a Patron!