Not matter how much robustness and redundancy you build around your multi-tiered infrastructure you are bound to suffer outage(s). I’m not implying the failure of a single server, but a complex outage that’s usually external to the operation of the infrastructure. What matters is how you communicate outage notification when things do go awry. I think the words that I’m searching for are transparency and openness.

We’ve seen over and over again the lack of notification and continual updates that either gets overlooked or ignored during an outage. I do understand the first and foremost goal for any organization during an outage is to stabilize their network infrastructure but what’s even more frustrating is lack of communication channel (status blog, public health dashboard, etc) where customers have no clue let alone getting through the NOC to get a straight answer.

What matters to most, including myself is how an outage analysis is communicated during downtime events in a timely manner and the outcome of such event that leads to lessons learned (post-mortem) as it serves as a great example of how not to do things.

The intention here is not to criticize companies, but quite the opposite; those who have chosen to publicize the causes of outages should be applauded by customers for being open. They have shared their efforts to learn and improve the availability, robustness, scalability, and performance of network services. After all, every ISP encounters the same challenges.

Fred Brooks nicely articulated:

You can learn more from failure than success. In failure you’re forced to find out what part did
not work. But in success you can believe everything you did was great, when in fact some parts may not have
worked at all. Failure forces you to face reality.

However, the “real” costs include some losses that are harder to quantify but may be far greater. For example,

• Lost revenue from dissatisfied customers moving to competitors or taking new business to competitors
• The cost of a tarnished image; the lessened ability to credibly market future “premium” differentiated services and position them against competitors.

There seems there can be three simple rules of outages:

1. Things break, design for that inevitability
2. Everyone recognizes rule #1,
3. Keep your customers and community informed which matters a lot to all and by being open and honest the stress of taking a new approach to communications is easily outweighed.

Keep in the mind a good deal of what I’ve outlined here will seem a lot like common sense which is a good thing. Quite often the simplest approaches to problem solving are the best ones, and openness and transparency is no exception.

There, got that off my chest.

Full Disclosure: Moderator for wiki.outages.org and I’m doing this to provide transparency to what I do and why I do it.