March 22, 2019, saw the completion of the final important step in the Key Signing Key (KSK) rollover—a process which began about a year and half ago. What may be less well known is that post rollover, and until just a couple days ago, Verisign was receiving a dramatically increasing number of root DNSKEY queries, to the tune of 75 times higher than previously observed, and accounting for ~7 percent of all transactions at the root servers we operate.

In July 2017, a new root zone Domain Name System Security Extensions (DNSSEC) KSK was first published in the DNS. That is the point at which validating resolvers all over the internet could begin the process of automatically updating their DNSSEC trust anchor, and many did. According to the best available data at the time, however, a small percentage of validators were not automatically updated. This, among other factors, led Internet Corporation for Assigned Names and Numbers (ICANN) to postpone the rollover and study the situation.

The rollover resumed in 2018, and in October, the root zone’s DNSSEC keys were first signed with the new KSK. Those of us closely observing the rollover heard of only a very small number of problem reports affecting end users, albeit after a significant amount of study and end-user outreach by Verisign and others, and everyone agrees the successful rollover was a significant milestone. There was, however, observable change in traffic to the root name servers. As the graph below shows, the rate of queries for the root’s DNSKEY data increased by a factor of five. Just prior to the rollover, Verisign’s root servers received about 15 million such queries per day. After the rollover, it increased to about 75 million.

In January of 2019, the root zone was published with the old KSK marked as “revoked.” Marking the old KSK as “revoked” tells validating resolvers that it should be removed as a trust anchor, since it will not ever be used again for generating DNSSEC signatures. Everyone involved expected this to be a non-event. However, we instead saw an even bigger increase in DNSKEY queries coming from a population of root server clients. As of March 21, 2019, Verisign’s root name servers receive about 1.15 billion DNSKEY queries per day, which is 75 times higher than pre-rollover levels and nearly 7 percent of our total steady-state query traffic.

Unlike the first increase, which remained level throughout the months of October, November and December, the second increase exhibits a disturbing trend which continued to grow and grow. With the assistance of some root server operator colleagues, we are just beginning to understand the cause of these increases.

On March 22, 2019, the old key (marked as revoked) was removed from the root zone. This took place according to the revised plan and schedule, in order to accommodate the root zone Zone Signing Key (ZSK) rollover taking place on April 1, 2019.

We are pleased to report that, with the removal of the revoked key, DNSKEY query rates are now returning to pre-revocation levels. Thus, it would seem that the presence of the revoked key in the zone triggered some unexpected behavior in a population of validating resolvers.

We hope to have a good understanding and explanation of this behavior in the coming weeks. To that end, we plan to continue collaboration with other operators, implementors and DNS researchers. Once the root cause is fully understood, the insights and experiences gained from this, the internet’s first KSK rollover coming to completion, will be especially useful as we begin to think about the best way to carry out future rollovers for the root zone.