A Response To Tom Lawrence’s DNS Malware Filtering Comparison

This blog post was written by Peter Lowe, Security Researcher at DNSFilter, in collaboration with Serena Raymond.

Last week Tom Lawrence of Lawrence Systems released a video comparing the malware filtering capabilities of four DNS filtering solutions (in the order of how well they performed in Tom’s test):

1. Quad9 (a free service)
2. Cloudflare
3. DNSFilter (that’s us!)
4. OpenDNS/Cisco Umbrella

After the video was initially released, I posted a comment on Tom’s forum sharing my thoughts, which I’ll expand on here. The focus of this post is to clarify why we appeared to do poorly in Tom Lawrence’s test and what that particular comparison test actually shows.

What is wrong with this DNS comparison test?

The short answer is that the threat feeds used to compile the list of domains for this particular comparison are not actually good sources for running a test like this.

Before the initial test started, the threat list Tom chose to use had roughly 2,300 domains. But only 141 of those domains actually resolved. That means 94% of the list was unusable. This is not only an issue given the sample size of the data, but it also indicates that the quality of the list itself is poor.

As we dug into each of these feeds from SANS Internet Storm Center, we noticed a few interesting things.

The domain list titled “Domain Blocklist From Malwaredomains” was primarily made up of domains added between 2014-2018. Though there are entries between 2019-2020, that particular feed seems to have been effectively abandoned as of February, 2019. The most recent post on their main website was in 2018.

The domain list “Malware Domain List.com,” when opened in a Google Sheet shows that the last updates were from 2017:

Threat Experts, another list (which appears blank) referenced by Internet Storm Center, last updated their website in 2009. And based on the threat lists we could find, the feed went dormant sometime between 2017 and 2018.

One particular list used contained domains related to the ZeuS Botnet. This bot first appeared in 2009 and sold to another hacker in 2010. This successor was taken down by the FBI in 2014 through a “multinational effort”. This particular list was actually discontinued because of the absence of the ZeuS threat in 2019.

Because of the age of these lists, it’s no wonder DNSFilter only blocked 15% of the 141 domains on the list in Tom’s test. We regularly retire domains from old threat lists at DNSFilter.

Why do we retire threat feeds?

Finding good threat feeds is honestly hard. There are not a lot out there, both paid and free feeds. And a lot of them are not updated anymore, as evidenced by the feeds we investigated above.

When we do ingest threat feeds, we’re very careful about it. We review the list and spot check the domains. Just because a list is called “threat feed” doesn’t mean we take it at its word. Additionally, once an entry from a feed has been ingested, it will expire as a blocked domain after a certain amount of time (generally a period of a few months) that’s dependent on the type of threat it is, recency of the threat, and the quality of the threat feed it came from.

We retire expired domains for a few reasons:

• Malware is usually only viable for a short period of time. Afterwards, sites will not resolve or hacked sites will be reclaimed by the rightful owner.
• To avoid false positives in our DNS filtering. Continuing to block “malicious” sites from 2014 might be blocking websites people want to access in 2020.

Not pruning your threat list gives people a false sense of security. Just because you’re blocking 500 threat feed lists (good luck finding that many) doesn’t mean you’re really protected from 500 lists worth of threats. It doesn’t account for new and unknown threats. As a threat list can only be compiled after a malicious domain is found and shared.

Part of the reason we believe Quad9 did so well in Tom Lawrence’s test is because as a free service, it likely ingested a lot of publicly available threat feeds without setting expiration dates. There is nothing wrong with Quad9’s decision to do that as a free service, though it does open users up to false positives.

Which brings us to the final point that the results of the test only show how well these DNS filtering services stack up to a very small list of outdated threats, meaning a majority of the companies who did well on this test are actually registering false positives.

[Edit June 4]: Before publishing this blog, I had reached out to ISC SANS regarding their suspicious domain lists and pointed out that the sources are out of date, much as I did above. This morning on June 4, they responded to me and took down the list.

Johannes B. Ullrich posted the following response on the ISC SANS website:

“Yesterday, Peter from DNSFilter send us a message noting that many of the domains in our “Suspicious Domain” feed no longer resolved, and some of the feeds we used as input were no longer maintained. After investigating, I have to agree with him. The remaining feeds don’t make a valuable service at this point. The idea of the “Suspicious Domain” list was to aggregate different lists, but with essentially only 1 or 2 lists left, that doesn’t make sense and I decided to no longer maintain the feed until we find new inputs.”

We appreciate Johannes’ responsiveness on this matter and willingness to suspend the feed until it is replaced with a better one.

What do we look for when we check our sources?

Research is what I do all day. I check both false positives and false negatives, and I use a variety of tools to aid me in my work. Two of these tools are VirusTotal and URLVoid.

Checking to see that a site shows up on these lists isn’t enough. We also consider the number of different lists that domain shows up on.

I randomly pulled a domain from one of the lists to help illustrate what we look for: http://888758.com/

This domain was not blocked by DNSFilter.

When we look at the domain in VirusTotal and URLVoid, only three current lists have this site as a threat. 2/35 on URLVoid and 3/80 on VirusTotal. The low number can be indicative that this site is not a threat. But we also look at the quality of the detection engines that mark a site as deceptive or not deceptive.

On both VirusTotal and URLVoid, we’re given the name of the source that has marked that domain as deceptive.

We won’t name them here, but there are a few sources that are less trustworthy than others. We know this because we deal with these threats daily, so over time we’ve become accustomed to a certain type of quality based on the source of the threat list.

In this instance, I decided to do a little more digging. When I land on this website, it appears to be an unregistered domain from China. The company named at the bottom in Chinese characters is eName Technology Co., Ltd. A very large domain name marketplace in China that was founded in 2005.

This is what the investigated domain looks like:

The icon in the upper-right corner points to the domain for eName’s Chinese website.

In going to eName’s website, I looked for another unregistered domain to verify that this type of page is still used by the brand to rule out that the above was a fake. I found another similar site where the icon in the upper-right corner also takes visitors to eName’s Chinese website:

Additionally when researching whether a site is malicious or not, I’ll look at current and historical Whois records, the history of domain (VirusTotal and URLVoid help with this, as does the WayBack machine), IP addresses, geographic location, links from other websites to the site in question, subdomains, similar domains (like the example above), mentions about the site on forums, and much more.

I won’t use all of these techniques on every questionable site—only when the foundational methods fail to turn up conclusive results.

Based on the research I’ve done around this domain, it is no longer a malicious domain.

But at DNSFilter, it’s not just me doing all the legwork manually. Our Webshrinker, our AI-based categorization, is trained to spot suspicious domains in real-time. We detail how our website categorization works here.

Why didn’t we give a malware domain list to Tom?

Allow me to quote myself:

“I would provide a decent public source you could use for domains to use in this test, but I don’t want there to be any suggestion of bias. However, we’d love to work with [Tom] to improve the test, because honestly we want the same thing: decent protection from online threats.”

We could absolutely provide a list of domains to Tom, but admittedly we would ace that follow-up test. There’s no right answer in determining a good list of malware domains to use in a test like this. Though, I have reached out to Tom as I’d love to work with him on figuring it out in a non-biased way.

What do we like about this video?

We appreciate that Tom took the time to do this test and was completely transparent about his methodology. The structure of the test is sound. Our main issue is the source of the domains.

We’re also happy that Tom opened the forum and asked for engagement from the community, and he was very receptive to it. Which shows in the follow-up video that he released.

In the video, he acknowledges that the list of malware domains used in his test are not the best domains to use. He decided to rerun the test for the Cisco Umbrella MSP portal with the same domain list since he used OpenDNS originally. The Cisco Umbrella score was over 50% the second time. However, as you can see from what we’ve outlined above, a good score on this test only demonstrates that at some point these companies added old lists to their software.

If you're interested in learning more about how our AI categorization works, watch our on-demand webinar Advanced Threat Identification.