Share this
An Update on Today’s DNS Outage: A Message From Our CEO
by Ken Carnesi on Jan 25, 2022 12:00:00 AM
As a recursive DNS resolver, we take any incident or perceived outage very seriously as a company. We know that you rely on us to be highly available as your DNS provider and DNS security. We recognize DNS is a critical aspect of your infrastructure, and for our MSP partners, that of your customers as well.
On Thursday January 13th at 20:08pm UTC and again on Tuesday, January 25th at 14:33 UTC, we began to receive tickets from users in the Northeastern United States stating they were experiencing DNS timeouts. Based on our monitoring, it seems that just under 1% of our network was impacted. Our network monitoring also showed a BGP route peering change which was shifting a heavy volume of DNS traffic to our nodes in Eastern Europe. The initial form of notifications to our customers was posted on our status page.
Our engineers discovered the root cause was a provider that made an upstream peering change. In both incidents, we immediately stopped advertising to that node and restored traffic right away. Service was completely restored by 20:27 UTC and 14:47 UTC, respectively.
When we select service providers and begin the process of deploying new anycast nodes, we tune our BGP advertisements and blocks based on the network paths which give our customers the best experience. In this particular case, our provider added a new peering connection that introduced routing changes that negatively impacted some of our customers. In response, we removed that node from service until we could fully understand the change and tune our BGP advertisements. We are actively improving our observability to detect and respond to these types of changes more quickly, ahead of any customer impact.
We will further vet this new provider to ensure they’re truly able to meet our SLAs. We hold our providers to a very stringent process—any providers unable to meet our SLAs will be eliminated from the DNSFilter network.
I sincerely apologize to our customers for this inconvenience, especially all those impacted. We remain highly committed to operating a reliable and innovative platform that you can trust.
Sincerely,
Ken Carnesi
Founder & CEO
Share this
Categories
- Featured (260)
- Protective DNS (21)
- IT (15)
- IndyCar (9)
- Content Filtering (7)
- Cybersecurity Brief (7)
- IT Challenges (7)
- AI (6)
- Deep Dive (6)
- Public Wi-Fi (6)
- Roaming Client (4)
- Team (4)
- Compare (3)
- Malware (3)
- Tech (3)
- Anycast (2)
- Events (2)
- MSP (2)
- Machine Learning (2)
- Phishing (2)
- Ransomware (2)
- Tech Stack (2)
- Secure Web Gateway (1)
"Green IT" isn't just a buzzword; it's a transformative approach reshaping how we manage technology. As IT professionals, embracing Green IT means integrating sustainability into every facet of technology management. This isn't merely about being eco-friendly; it's about crafting IT environments that are both cost-efficient and future-proof.
Roblox, one of the world’s largest online platforms for children, has recently been making headlines—and not for the right reasons. While it's a playground for millions of kids, it's also become a target for online predators. A recentBloomberg article revealed disturbing details about predators exploiting Roblox’s ecosystem, raising urgent concerns about the platform’s safety.
Is your company's Wi-Fi network inadvertently opening doors to cyber threats? Are employees and visitors putting company data at risk every time they connect their personal devices? With over 60% of data breaches linked to unsecured network access, understanding your Wi-Fi setup has never been more critical.