|
AREAS AFFECTED BY THE OUTAGE / 21 OCT 2016 – Source: Level3 Outage Map
Major internet sites were disrupted for several hours this morning as internet infrastructure provider Dyn reported it was under a cyberattack, mainly affecting traffic on the U.S. East Coast. Twitter, Spotify, Airbnb, Reddit, Visa and various media sites were among organizations whose services were reported to be down on Friday morning. Amazon also disclosed an outage that lasted several hours on Friday morning.
— Doug Madory, director of internet analysis at Dyn, in an email said: Dyn received a global DDoS attack on its Managed DNS infrastructure in the east coast of the United States. DNS traffic resolved from east coast name server locations experienced a service interruption during the attack. Updates will be posted as information becomes available. Services were restored to normal as of 13:20 UTC.
— Update: As of around 12 PM ET, Dyn reported that it is investigating another DDoS attack, and is continuing to attempt to “mitigate” the attack. Box, Twitter and other sites appear to be down again. The White House press secretary has also said that the Department of Homeland Security is investigating the attacks.
— Update from Dyn: “Our engineers continue to investigate and mitigate several attacks aimed against the Dyn Managed DNS infrastructure.”
— Gillian Christensen of the U.S. Department of Homeland Security says the agency is “investigating all potential causes.”
— “The attack on DYN comes just hours after DYN researcher Doug Madory presented a talk on DDoS attacks in Dallas, Texas at a meeting of the North American Network Operators Group (NANOG),” says Brian Krebs whose own site recently underwent historic DDoS attack. “Madory’s talk ... delved deeper into research that he and I teamed up on to produce the data behind the story DDoS Mitigation Firm Has History of Hijacks. ... I have no data to indicate that the attack on Dyn is related to extortion, to Mirai or to any of the companies or individuals Madory referenced in his talk this week in Dallas. But Dyn is known for publishing detailed writeups on outages at other major Internet service providers. Here’s hoping the company does not deviate from that practice and soon publishes a postmortem on its own attack.”
— Update, 3:50 p.m. ET / Brian Krebs reports: “Security firm Flashpoint is now reporting that they have seen indications that a Mirai-based botnet [see earlier report on Mirai] is indeed involved in the attack on Dyn today. Separately, I have heard from a trusted source who’s been tracking this activity and saw chatter in the cybercrime underground yesterday discussing a plan to attack Dyn.”
— “This was not your everyday DDoS attack,” Kyle York, Dyn’s chief strategist. Nicole Perlroth reporting in the New York Times: “Dave Allen, the general counsel at Dyn, said tens of millions of internet addresses, or so-called I.P. addresses, were being used to send a fire hose of internet traffic at the company’s servers. He confirmed that a large portion of that traffic was coming from internet-connected devices that had been co-opted by type of malware, called Mirai.” ... Dale Drew, chief security officer at Level 3: “Roughly 10 percent of all devices co-opted by Mirai were being used to attack Dyn’s servers.”
— Update, 7:53 p.m. ET / Dyn issues Preliminary Findings Report with additional detail: “On Friday October 21, 2016 at approximately 11:10 UTC, Dyn came under attack by a large Distributed Denial of Service (DDoS) attack against our Managed DNS infrastructure in the US-East region. Customers affected may have seen regional resolution failures in US-East and intermittent spikes in latency globally. Dyn’s engineers were able to successfully mitigate the attack at approximately 13:20 UTC, and shortly after, the attack subsided. At roughly 15:50 UTC a second DDoS attack began against the Managed DNS platform. This attack was distributed in a more global fashion. Affected customers may have seen intermittent resolution issues as well as increased global latency. At approximately 17:00 UTC, our engineers were again able to mitigate the attack and service was restored.” ... “A more in-depth analysis will be distributed in the form of a Root Cause Analysis report at a later date.”
— “Stop Taking Down the US Internet” / WikiLeaks posts on Twitter: “Mr. Assange is still alive and WikiLeaks is still publishing. We ask supporters to stop taking down the US internet. You proved your point. ... The Obama administration should not have attempted to misuse its instruments of state to stop criticism of its ruling party candidate.”
— Update, Oct 24 / China’s Xiongmai to recall up to 10,000 webcams after hack,” Sijia Jiang reporting in Reuters from Hong Kong: “Up to 10,000 webcams will be recalled in the aftermath of a cyber attack that blocked access last week to some of the world’s biggest websites… the company would recall the first few batches of surveillance cameras made in 2014 that monitor rooms or shops for personal, rather than industrial, use.”
— Update, Oct 26 / Dyn has released an analysis summary of the attack: “Early observations of the TCP attack volume from a few of our datacenters indicate packet flow bursts 40 to 50 times higher than normal. This magnitude does not take into account a significant portion of traffic that never reached Dyn due to our own mitigation efforts as well as the mitigation of upstream providers. There have been some reports of a magnitude in the 1.2 Tbps range; at this time we are unable to verify that claim.”
Sponsored byIPv4.Global
Sponsored byCSC
Sponsored byWhoisXML API
Sponsored byVerisign
Sponsored byRadix
Sponsored byDNIB.com
Sponsored byVerisign
Has Dyn considered dropping all incoming requests which would result in NXDOMAIN while the attack is in progress? Presumably the attack traffic pattern, which uses randomised domain names, means that the overwhelming majority of requests are resulting in NXDOMAIN responses. Intermediates will attempt to cache these, and this crowds out all the actually-useful responses which could otherwise be cached. If responses were limited to only those where a record was found, not only would outgoing traffic fall dramatically, but the intermediate caches would have a chance to fill up with useful information, hopefully mitigating the effect of the attack.
I say this as someone with PhD-level study of this kind of problem, but not as someone with extensive relevant operational experience. I’d be interested to hear the thoughts of someone who works closer to the coal-face.