Home / Blogs

Complexity and Crashes

It’s a familiar story by now: on the 8th of August, 2016, Delta lost power to its Atlanta data center, causing the entire data center to fail. Thousands of flights were cancelled, many more delayed, and tens of thousands of travellers stranded. What’s so unusual about this event is in the larger scheme of network engineering, it’s not that unusual. If I think back to my time on the Escalation Team at a large vendor, I can think of hundreds of situations like this. And among all those events, there is one point in common: it takes longer to boot the system than it does to fix the initial problem.

There was the massive Frame Relay hub and spoke network, built up over years, servicing every site for a particular retailer. A single interface flap caused hours of downtime. Then there was the theme park, and the routing protocol convergence failure that costs eight hours of down time. There was the bank with the misconfigured application that took several days to recover from. And now there is Delta. Perhaps it took an hour to put the fire in the generator room out, but it took another six and a half to bring the systems back on line once the fire was out. I still think about the network administrator who’s backup plan was to shift a cable from one 7500 to another in the case of failure. Time to swap the cable? A few minutes. Time to reconverge the network and get all the applications running again? Several hours.

The closest analogy I can find is one of those plate spinning acts. Once all the plates are spinning, it’s impressive. But watch carefully how long it takes to get all the plates spinning, and how long it takes to bring them all down. Spinning each plate takes a minute or two. Bringing them all down at once only takes a minute or two, as well. It doesn’t matter if the plate spinner has another set of plates to replace the ones broken in a crash, by the time he’s done respinning the plates, the audience is gone.

The problem, you see, isn’t disaster recovery. It’s not even having a backup, or a plan to switch to another data center.

The problem is complexity.

We’re just too quick to add “another system, another protocol, another ...” to an existing system in order to support some requirement handed to us by someone who believes networks and data systems should be able to “do anything.” This is not only an application problem, it’s a networking problem, too. It’s just so much cheaper to build another layer 2 overlay on our networks than it is to insist applications be written to support layer 3 networking directly. It’s so much easier to buy that new application for all the bells and whistles, and then say “make it run on our network” to the network engineers, than it is to perhaps scale back expectations a little in order to build a supportable network.

Maybe it’s time for all of us to take a lesson from Delta, and the many network failures before Delta over the last twenty years. That next overlay you deploy to “solve” and application level problem isn’t a solution. It’s just another hidden bit of technical debt waiting to blow the network, and your business, up in a few months.

We need to start calculating the ROI across not just disaster recovery, but complexity, as well.

By Russ White, Infrastructure Architect at Juniper Networks

Filed Under


Comment Title:

  Notify me of follow-up comments

We encourage you to post comments and engage in discussions that advance this post through relevant opinion, anecdotes, links and data. If you see a comment that you believe is irrelevant or inappropriate, you can report it using the link at the end of each comment. Views expressed in the comments do not represent those of CircleID. For more information on our comment policy, see Codes of Conduct.

CircleID Newsletter The Weekly Wrap

More and more professionals are choosing to publish critical posts on CircleID from all corners of the Internet industry. If you find it hard to keep up daily, consider subscribing to our weekly digest. We will provide you a convenient summary report once a week sent directly to your inbox. It's a quick and easy read.

I make a point of reading CircleID. There is no getting around the utility of knowing what thoughtful people are thinking and saying about our industry.

Co-designer of the TCP/IP Protocols & the Architecture of the Internet




Sponsored byVerisign

Threat Intelligence

Sponsored byWhoisXML API


Sponsored byDNIB.com

Domain Names

Sponsored byVerisign

IPv4 Markets

Sponsored byIPv4.Global

New TLDs

Sponsored byRadix

Brand Protection

Sponsored byCSC