RCA Review: Addressing Latency in the NY Region Tuesday, July 25

by Dave Raphael on Jul 25, 2023 5:48:19 PM

Today we had an incident that lasted for approximately one hour in the NYC region. I wanted to take the time to address this incident and review what happened, why it happened, what we did to resolve it, and what we’re doing to prevent future issues like this.

Impact and cause of incident

The incident began at 11:00 a.m. ET when some customers in the New York City region experienced high latency over our DNS1 Anycast network.

The cause of this incident stemmed from an issue with one of our hosting providers in which a circuit was inadvertently turned off. This resulted in DNS traffic to the Washington, D.C. region to be rerouted to servers in New York City. While this traffic shift is what we expect BGP to do in instances like this, the load was too great to resolve some requests in a timely manner. This resulted in the latency issue experienced by some of our customers.

Prior to receiving our first ticket, our team had already recognized the issue was with that particular upstream provider. We had visibility into the traffic moving from DC to NYC, and we were able to fully diagnose and solve the issues by 12:10 p.m. ET—24 minutes after our first support ticket was received.

But we’re committed to doing better. We’ve already had plans to increase our load capacity, including the NYC region, which will allow us to have greater resilience in the case of unbalanced loads resulting from unplanned traffic shifts. Our goal is to serve our customers and provide the best possible experience.

A while ago, I promised transparency, action, and accountability. I’m here to keep that promise.

Platform, Fires, and You: Navigating the Fine Line Between Operations and Development

The Old-School Operations Role: Backbone or Bottleneck?

In the early days of IT, the operations team was the unsung hero—the silent, and often siloed, force that kept everything running. They were responsible for the infrastructure: Servers, databases, and networks that powered the business. They managed deployments, monitored systems, and ensured uptime. If it was working, no one noticed them. If it wasn't? Well, then the questions started: "Wha...

The Hidden Risks of Refreshing Old Threats

When Vintage Goes Viral (In All the Wrong Ways)

Remember that time you found your old Tamagotchi and thought, "Hey, this could be fun again"? Well, cybercriminals are having a similar nostalgic moment, but instead of resurrecting digital pets, they're breathing new life into outdated malware and long-forgotten data breaches. Welcome to the world of recycled cyber threats, where everything old is terrifyingly new again.

Trends of Cybersquatting, Typosquatting, and Other Malicious Domains

The Growing Threat of Malicious Domains in Cybersecurity

As cybercriminals continue to evolve their tactics, domain-based attacks like cybersquatting, typosquatting, and other malicious domains have become a significant threat to businesses and individuals alike. These attacks are designed to exploit trust, impersonate brands, and mislead users into handing over sensitive information—often resulting in financial losses, data breaches, and reputat...