You’ve heard the joke, sometimes in haiku form:
It’s not DNS
There’s no way it’s DNS
It was DNS
But why is it always DNS? There are a few reasons.
DNS is a simple concept to understand but extremely difficult to put into practice. DNS is often to blame for massive internet outages because even the most savvy companies and educated minds make mistakes while DNS misconfigurations are unforgiving.
At the same time, DNS issues often don’t manifest themselves as DNS issues. Based on the symptom, DNS is often the last thing you’d expect it to be—this explains the denial stage of the haiku.
There was a comment on Reddit recently that sums this up perfectly:
It takes a lot of DNS know-how to have the ability to look at a non-DNS issue and come to the conclusion that it’s a DNS problem before exhausting every other possible option, which is usually the case.
A “glitch” in the internal network caused major outages in Amazon’s US-EAST region (Virginia) after a scaling for more capacity event was triggered. Increased latency and errors for services communicating between these networks, resulted in persistent congestion and performance issues on the devices. There are unverified rumors that the glitch was caused by an orchestrated DNS attack of unknown origins.
A bad BGP router configuration caused a global cascade of outages removing all Meta/Facebook properties from the internet for nearly six hours. To the rest of the internet, Facebook’s infrastructure IPs were unreachable and DNS names stopped resolving. Essentially a black hole where Facebook used to live.
The network experienced a global issue relating to edge DNS impacting many internet resources caused by a "software configuration update [which] triggered a bug in the DNS system”. Fidelity, the US Securities and Exchange Commission's document search site, Airbnb, British Airways and others were affected. Most of the sites were back up and running in under an hour.
A bad router configuration on the Cloudflare global backbone in Atlanta started announcing bad routes, rerouting massive traffic volumes to the geolocation overwhelming the hardware. The incident only lasted 27 minutes but affected San Jose, Dallas, Seattle, Los Angeles, Chicago, Washington, DC, Richmond, Newark, Atlanta, London, Amsterdam, Frankfurt, Paris, Stockholm, Moscow, St. Petersburg, São Paulo, Curitiba, and Porto Alegre.
A software deployment introduced a bug that could be triggered by a specific customer configuration under specific circumstances. On June 8th, the specific configuration and circumstances collided to produce a “broad and severe” Fastly CDN outage.
This outage was not caused by DNS so why do I mention it? Because Fastly and the affected web properties used the magic of DNS to route around the disruption in less than an hour. DNS for the win! Tell your friends.
Masters of DNS are a rare breed and DNSFilter was founded by one and currently has several on staff (I am not one of them, I’d call myself a DNS dabbler).
When we hear "it's always DNS" it's true, not because the technology is bad. It’s because the practitioners are, no offense, inadequate. DNS is so ubiquitous, and it means people are thrown into it without really knowing what they’re doing. It’s not their fault, DNS touches more than they realize.
But once you realize how intertwined DNS is with the entirety of your IT infrastructure, you start to have more respect for it. And you’ll recognize you need to trust the masters of DNS with your DNS security.
______________________________________________
4 https://twitter.com/Akamai/status/1418271515192270850?s=20
5 https://blog.cloudflare.com/cloudflare-outage-on-july-17-2020/
6 https://www.fastly.com/blog/summary-of-june-8-outage