On July 18th, 2020, caused an outage for about 27 minutes affecting businesses worldwide. a configuration error in Cloudflare’s backbone network The error configuration sent all traffic across the backbone to Cloudflare’s Atlanta node and unfortunately “overwhelmed” the router. Cloudflare later mentioned they saw traffic drop by about 50% across the network. Although not entirely catastrophic thanks to the architecture design, but many locations with heavy internet usage were affected including San Jose, Seattle, Los Angeles, Chicago, Washington, DC, London, Amsterdam, Frankfurt, Paris, Stockholm, Moscow, São Paulo, and the list goes on. How our system handled it Cloudflare is one of the many CDNs mlytics offer via Power-Ups (CDN marketplace) on the platform. When the outage occurred, the AI Load Balancing swapped out Cloudflare with the next best-performing CDN (varies in different regions) for requests. The chart below is the AI Load Balancing optimization chart, it shows which CDNs were used (swapped over) during a certain timeframe. In this case, we’re using a demo site with Akamai and Cloudflare installed. As shown, Akamai had several query spikes in the states due to the Cloudflare performance drop – the system is actively swapping over to Akamai. Frequent swaps were made from Cloudflare to Akamai in the US This had successfully helped many of our customers mitigated the outage, and we received no complaints over the course. On the chart, we’re also seeing a latency spike and availability drop for Cloudflare in the same time frame. This helps illustrate a clear picture of what happened when aligning this with the chart above. Pulse It shows a latency spike and availability drop for Cloudflare on the Pulse chart We believe all the CDN providers are doing their best to deliver the best performance possible, but unfortunately, things do happen. And our job at mlytics is to help our customers minimize the exposure of such risk and maximize the uptime at all times. Again, outages DO happen This is a wake-up call reminding all businesses will eventually have to deal with outages if offering any form of web services. Thus, a solid cloud redundancy and disaster recovery plan in place to prevent any event from taking down your service is imperative. Imagine losing money and customers hate you… horrible right? But stopping users from doing what they were doing (especially if they enjoy it so much) is your express train ticket to PR hell. Small or medium businesses might have room for mistakes, but some of the bigger companies affected by the outage such as Discord, Feedly, Politico, Shopify, and League of Legends might not be so lucky. These are some of the most used online services today, and imagine millions of users got disconnected without being informed. We've talked about in the past, and there are just so many factors that can bring a website off the grid. A website without a backup plan is exactly like sailing a cruise without liferafts – you’re betting you won’t need it. website downtime causes and solutions Previously published at https://mlytics.com/blog/how-we-helped-our-users-mitigated-the-cloudflare-outage/

Helping Our Users Mitigate The 2020 Cloudflare Outage - How We Did It

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Exploring the World of Distributed Systems: A Beginner's Guide to the CAP Theorem

How To Prevent Server Overload In Go

How To Set Custom Metrics for Autoscaling in AWS

Protect Your Availability During The Holiday Season The Smart Way

The Impact of Big Data In Business, Past and Future

Achieving Optimal Service Reliability: Insights Into Service Level Objectives (SLOs)

Exploring the World of Distributed Systems: A Beginner's Guide to the CAP Theorem

How To Prevent Server Overload In Go

How To Set Custom Metrics for Autoscaling in AWS

Protect Your Availability During The Holiday Season The Smart Way

The Impact of Big Data In Business, Past and Future

Achieving Optimal Service Reliability: Insights Into Service Level Objectives (SLOs)

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps