For some time now we’ve been tracking the progress of the deployment of DNSSEC in the Internet. Its been a story of an evolution of the measurement technique, starting with a technique that attempted to guess at the behaviour of resolvers, through to techniques that explicitly pose novel DNS names to clients so as to negate aspects of resolver caching that otherwise complicate the measurement technique.

In the process we’ve learned perhaps more than we had wanted to about the behaviour of Flash engines, Apache web servers and FreeBSD system tuning, and also learned much more than we had anticipated about the finer details of Google’s online ad presentation behaviour. But one thing we did not see in all of this was any large scale jumps in the level of client use of DNSSEC validation over this period at the start of the year.

This apparent slowness in the adoption of DNSSEC a source of some frustration. We have heard many times that some of the more insidious threats to the security and integrity of the Internet’s service environment start with attacks on the DNS. Such attacks exploit a real weakness in the behaviour of many users: You type in a URL, and you see a familiar screen in response and you think you are now connected to the service you specified. But there is the risk that you are not, and this risk is there irrespective of whether the service is “secure” or not. There is a risk from various forms of so-called “man-in-the-middle” attacks that you have been mislead. Most such attacks pass largely unremarked, if not unnoticed. But from time to time the issue generates a high level of public prominence, such as the attacks that resulted from Diginotar attack on a Domain Name Certification Authority back in 2011 and the subsequent exploitation of that attack in a consequent structured attack on Gmail users located in Iran.

The best defence against these forms of attacks that we’ve been able to devise is to secure the DNS, so that a system making a query of the DNS can be assured that the answer that they are given in response to their query is precisely the same information that was entered into the DNS by the authorized zone administrator. This, in turn, allows the embedding of information about domain name certification into the DNS in a secure manner (as described in RFC6394), which is a relatively effective response to the form of man-in-the-middle attack we saw as a consequence of the compromise of Diginotar’s CA services. The combination of these two measures, signing the domain name using DNSSEC, and placing certificate credentials into the DNS as signed data, would allow a user’s browser to reliably avoid using a compromised CA to validate a fake Domain Name Certificate.

For some years attention has been focussed to the effort to deploy DNSSEC in the domain name system. We have seen an extensive effort to get to a DNSSEC-signed root, and now that this has been achieved, we are now seeing this effort focus on the signing of all top level domain names (TLDs).

But signing domain names is only one half of the story. Clients’ DNS resolvers also need to retrieve this information and validate the responses they receive from the DNS. So, at the other end of the spectrum, in the realm of DNS resolvers, we have seen increasing use of DNSSEC validation, but little in the way of structured measurement of progress in this area. At APNIC Labs we have been looking at ways to measure this, and are trying to answer the basic question: How many of the Internet’s user population exclusively use DNS resolvers that perform DNSSEC validation?

In late 2012 we saw some 1.6% of clients exclusively use DNSSEC-validating resolvers, using a relatively imprecise measurement methodology.

At the start of 2013 we revised the experimental technique, and saw some 3% of users appear to exclusively use DNSSEC validating resolvers. Appropriately, these users were unable to resolve a DNS name when its DNSSEC signature was invalid. In the same experiment, we also measured a further 2% of clients who use a mix of DNSSEC-validating and non-validating resolvers, so that when a DNSSEC validating resolver correctly responds with SERVFAIL, the client then queries another resolver who does not perform DNSSEC validation, and therefore dutifully returns the original address record.

A couple of weeks after we conducted this experiment Google announced the inclusion of DNSSEC validation to its public DNS service. Earlier configurations of Google’s public DNS service required the client to set the DNSSEC OK (DO) flags on its queries in order to trigger a DNSSEC validation operation, but in late March Google switched this behaviour to perform a DNSSEC validation for all queries, except for those that explicitly requested no validation via the setting of the Checking Disabled (CD) flag in the DNS query.

What did we see in May when we again measured DNSSEC use by the Internet’s end user population?

Measuring DNSSEC Use

We ran an experiment across the period of the 9th May through to the 26th May, and ran a DNSSEC capability test across 2,746,777 clients, selected using an online advertisement placement method. Of these clients we saw 2,595,672 complete the experiment’s tests and submit results to our server.

As with previous DNSSEC experiments, we presented the client with three URLs, all using IPv4. URL A used a domain name that was validly DNSSEC-signed domain name. URL B had no DNSSEC signature. URL C had a DNSSEC signature that was corrupted so that DNSSEC validation would fail. All three URLs used a label part in their DNS name that was relative. Through this measure we were trying the minimize any measurement “noise” as a result of DNS caching, ensuring that all queries were passed to the authoritative name server for both queries for the original resource records and queries for the DNSSEC DS and DNSKEY resource.

To explain “relative uniqueness” a little more, we wanted to present each client with a unique DNSSEC signed domain name. The most obvious way to achieve this would be to use a large signed zone, but there are a number of major operational considerations in maintaining a signed DNS zone file with upward of 10 million entries that would be needed to support an experiment of this scale. Instead, we opted to use a smaller pool of some 500,000 unique names, and set a one hour TTL on the DNS data. We were cycling through this name space in around 20 hours, which was considered to be comfortably in excess of the one hour TTL.

The various combination of URLs that were logged as being fetched from the associated web server produced the outcomes as shown in Table 1. This table also includes results from the earlier run of this experiment in February of this year.

Table 1 – DNSSEC Fetch Results (Web)

If a client supports DNSSEC validation outcomes then it would fetch URLs A and B, but not URL C (row 4 in Table 1). If it does not support DNSSEC validation then it should fetch all three URLs, as in all other respects the three URLs are functionally identical (row 8 in Table 1). However, some 3% of clients fetch various combinations of URLs other than these two anticipated combinations, showing that there is some variability in the precise mode of execution of the experiment in certain client scenarios.

The web server logs of this experiment show that, within an error bound of ±1%, some 8% of clients appear to be performing some kind of DNSSEC validation, based on the combinations of URLs that they are able to fetch. This represents an increase of some 5% since February, which we could surmise as being attributable to the change in behavior of the Google Public DNS resolvers.

A detailed examination of the logs of the DNS authoritative server, when coupled with the Web log data can provide a better insight into these numbers. We are looking for clients whose DNS resolvers query for both the A resource record and the DNSKEY and DS resource records, and where the client fetches the A and B URLs, but not the C URL. These clients, we assume, are using DNS resolvers that perform DNSSEC validation, and pass the client a SERVFAIL response to the DNS query associated with the C URL. Furthermore, we can assume that where a client’s DNS resolvers make no query for URL A or C’s DS or DNSKEY resource records, then we can assume that the client’s DNS resolvers perform no DNSSEC. The other case is where we see the retrieval of DNSSEC resource records, but see fetching of the C URL. We assume that in this case the client’s DNS resolvers are a mix of validating and non-validating resolvers, and in the case of the C URL the response of SERVFAIL from a DNSSEC-validating resolver causes the client to ask the same query from the next resolver in its local resolver set, or causes its DNS resolver to ask the next resolver in its DNS forwarding set without the client’s explicit knowledge.

We can combine both the DNS query profile and the web server logs to produce the results shown in Table 2.

Table 2 – DNSSEC Fetch Results (DNS)

This DNS data shows that the proportion of clients who exclusively use DNS resolvers that perform DNSSEC validation has risen by a little under 5% of the total client population, from 3.1% in February 2013 to 8.3% in May 2013.

We have also seen a significant rise in the number of clients who use a mix of DNS resolvers, some of which were seen to perform DNSSEC validation. This has risen by some 1.8%, from 2.6% in February 2013 to 4.4% in May 2013.

To what extent is this increase in the population attributable to Google’s Public DNS servers?

Of the 2.34M unique IP addresses of clients who ran this experiment, we saw 174,082 clients use Google Public DNS servers, or 7.4% of all tested clients. Of these, we saw some 128,359, or 5.5% of all clients exclusively use Google’s Public DNS servers, with the remaining 45,723 (or 2%) clients use a mix of Google DNS servers and other servers.

More specifically, where are these DNSSEC-validating clients, and what resolvers do they use?

Of those countries with more than 200 samples, those countries with the highest proportion of DNSSEC-validating clients are as follows:

Table 3 – DNSSEC Validation by Country

There are not many country tables where the Faroe Islands features in the top 20, but of the 242 tests we ran against clients located in the Faroe Islands we found one quarter of them performed a full DNSSEC validation.

A complete list of all countries and their counts of clients who perform DNSSEC validation can be found here (csv file).

To what extent can these results be attributed to Google’s Public DNS service? Table 4 shows the same 20 countries, but with additional columns added, namely the proportion of use of Google’s Public DNS service, and also looking at just the subset of end clients who are observed to perform DNSSEC validation, and the proportion of these clients who use Google’s Public DNS Service.

Table 4 – DNSSEC Validation and use of Google’s Public DNS by Country (Click to Enlarge)

The relative number of clients who exclusively use Google’s Public DNS service is prominent in Vietnam, Jamaica and Afghanistan, while the relative number of clients who use Google’s DNS service in conjunction with other servers is prominent in Angola, Indonesia and Turkey.

If we look at the subset of clients who are seen to be performing DNSSEC validation, then there is a very strong correlation between the use of Google’s Public DNS resolvers and DNSSEC validation in Vietnam, Jamaica and Indonesia. In other countries in this list it would appear that there are other resolvers used by these clients that also are performing DNSSEC validation.

What we find for the entire data set gathered in May 2013, is that of the 240,635 end clients who are performing DNSSEC validation, some 47% of these clients use Google’s Public DNS service exclusively, another 47% of these clients do not appear to use Google’s Public DNS service, and the remaining 6% use a mix of Google’s and other DNS resolvers.

What this shows is that some 4% of the Internet’s user base exclusively use Google’s Public DNS service and now are having their DNS names validated by this public DNS service. A further 4% of users also use DNS resolvers that perform DNSSEC validation, but do not use Google’s public DNS service to do so.

But doesn’t the final line of table 4 indicate that 5.6% of the tested clients exclusively use Google’s Public DNS? Does this mean that 1.6%, or 45,000 clients, who are exclusively using Google’s Public DNS are not performing DNSSEC validation? It would appear so. Part of the issue with measuring DNS is that an authoritative server cannot unravel the manner by which a DNS query is passed from a client through a DNS forwarder chain before it reaches an authoritative name server. If a DNS Forwarder passes all its queries to Google’s Public DNS service, but marks all its queries with DNSSEC Checking Disabled, then we would see queries from Google’s public DNS resolvers that are not apparently performing DNSSEC validation. This would appear to be the case here, and this is the most likely explanation for these “missing” 45,000 DNSSEC validating clients.

Table 4 displayed a country-by-country view of DNSSEC deployment. What is the view of DNSSEC when looking at service provider networks? This is shown in Table 5.

Table 5 – DNSSEC Validation and use of Google’s Public DNS by Origin AS (Click to Enlarge)

Table 5 shows a similar analysis of the DNSSEC data, but this time using the originating network, or Autonomous System, rather than country. Of the top 20 DNSSEC validating networks with more than 200 users seen in this experiment, only three networks, Ambra (AS 34630) in Romania, Linkem (AS 198471) in Italy, and Olo Peru (AS 52400) in Peru, appear to have directed all their customers’ DNS queries to Google’s DNS systems. The other 17 networks in this list all appear to be using local DNS resolvers that have been configured to perform DNSSEC validation.

A full list of all networks, and their counts of clients who perform DNSSEC validation, can be found here (csv file).

Google’s Public DNS Service

Since March 2013 we’ve seen the proportion of end users who use DNSSEC resolvers that perform DNSSEC validation rise from 3.3% to 8.1%, or a rise of some 4.7%.

Most, but not all of this rise, can be attributed to Google’s Public DNS service, which is used exclusively by some 5.6% of all clients across the entire Internet. When Google turned on DNSSEC validation on their resolvers, the majority of these clients were then performing DNSSEC validation even though they had not changed any part of their local DNS configuration. Just over one half of all clients who are seen to be performing DNSSEC validation, use Google’s Public DNS Servers.

However, that’s not the total population of users who avail themselves of Google’s DNS services. A further 3.7% of users also use Google’s Public DNS service, but they use it in conjunction with other DNS resolvers. The most common case here is that the other DNS resolvers used by the client, or by the client’s local DNS resolver, do not use DNSSEC validation, so the SERVFAIL response from the query to Google’s service prompts additional DNS queries to be made to other configured DNS resolvers. A total of 9.4% of the clients we saw in this experiment made use of Google’s Public DNS service in one way or another.

Performing DNS resolution for almost 10% of the Internet is a very significant undertaking by Google. There are some interesting issues this figure raises in terms of the breadth and volume of information about user behaviors that is exposed in DNS queries. In light of the current concerns over the use of so-called Internet meta-data by a number of national intelligence operations, and broader concerns relating to individual privacy and the use of various forms of cloud services and other public services, this aspect of today’s DNS operations raises some further questions about extent to which certain aspects of user behaviors are visible to this individual DNS operator though the use of their DNS systems by some 10% of the Internet’s user population. However, that is perhaps heading down the path of a new topic, and is straying from this study of the level of deployment of DNSSEC validation in today’s DNS.

On the topic of DNSSEC deployment, we’ll continue to track the state of DNSSEC deployment in the coming months, to see how and where the story of DNSSEC deployment evolves.