Geoff Huston’s recent post about the rise of DNS amplification attacks offers excellent perspective on the issue. Major incidents like the Spamhaus attack Geoff mentions at the beginning of his post make headlines, but even small attacks create noticeable floods of traffic. These attacks are easy to launch and effective even with relatively modest resources and we see evidence they’re occurring regularly. Although DNS servers are not usually the target of these attacks the increase in traffic and larger response sizes typically stress DNS infrastructure and require attention from operation teams.

Amplification attacks can be launched in a couple of different ways, but we’ve mostly seen attacks using DNS resolvers, deployed by providers as part of their service (some attacks rely on sending queries directly to authoritative servers). The attacks we’re seeing are especially insidious because of the way they work. As Geoff points out researchers have detected tens of millions of “open resolvers” on the Internet; the overwhelming majority of them are home gateways with open DNS proxieswhich attackers can easily induct into their exploits. Not only do attackers get access to a substantial pool of devices but bouncing their attack queries off home gateways also conveniently circumvents protections network operators may have in place such as source address validation (to prevent address spoofing from within their network) and restricting access to resolvers they deploy (“closing” the resolver to off-network queries). What this means is even providers doing the “right” things can still be complicit in amplification attacks—through no fault of their own.

Throughout this year we’ve seen DNS resolvers at large ISPs around the world report much higher than normal volumes of DNS queries. In some cases network infrastructure such as load balancers has been stressed to failure from traffic spikes, causing shifts in load to other DNS servers. Fortunately in the cases we’ve seen DNS servers have been properly dimensioned and highly resilient, remaining available even with substantially increased load. Visible events like this are obvious indications of attacks and always cause operations to investigate.

Detecting amplification attacks requires proper tools to identify unusual traffic and determine root causes. Most DNS servers can capture query data but some server releases are substantially taxed when logging is turned on so the extra CPU cycles need to be factored in to infrastructure sizing to ensure performance metrics will be achieved and subscribers have the best possible experience (low latency responses). In this case operations had ready access to DNS query data, which was captured as a standard practice since it had minimal impact on their servers.

Access to data is always necessary but rarely sufficient in troubleshooting a problem; exposing vital details to identify an attack requires navigating through large data sets and visualizing query data. In these cases the Ops teams had access to additional tools which allowed them to easily show Top Domains queried. This revealed names they hadn’t seen among the Top Domains before. Further navigation and visualization allowed them to drill down into the data to determine the types of queries being made. Query counts associated with each query type were also available so it was easy to gauge magnitude.

Operations staff immediately recognized a spike in ANY queries, which are widely known to be evidence of an attack. ANY queries have virtually no legitimate uses; they’re used for amplification attacks because they return all the DNS records for a domain, turning very small questions into extremely large responses. Depending on the name queried with ANY amplification can be as much as 80 times. With readily available data and tools telltale signs of an amplification attack were revealed: atypical domain names were among the Top Domains and ANY queries were being used for those domains. With these insights in hand the Ops team was able to take steps to mitigate the attack. This will be discussed in the next post.

Because relatively familiar techniques were used in these attacks identifying them was much easier. As attackers change their tactics, using less well known names that aren’t easily associated with attacks for instance, or distributing their attacks widely, identifying attacks will get harder. But no matter what ready access to DNS data and analytical tools will be essential. Subsequent posts will discuss additional methods for detecting attacks as they evolve.