After the botched burglary at the Watergate Apartments, every scam and scandal that hit the headlines became a ‘gate’—Irangate, Contragate, you name it.

The Heartbleed bug is possibly the closest thing to Watergate that this generation of computer security had seen till the past few days—an exploit in a component that is “just there”—something you utterly rely on to be there and perform its duties, and give very little thought to how secure (or rather, insecure) it might be. So, fittingly, every such catastrophic bug in an ubiquitous component is now a ‘bleed’.

The latest bug to hit the headlines is, therefore, ‘Bashbleed’, a remotely exploitable vuln in the venerable old Bourne Again Shell (Bash), which has been around since 1989, gradually gaining in popularity over the years and deployed on various Unix/Linux distributions as well as on Apple Macs.

Finding a remotely exploitable vulnerability after years in code that has been around for over two decades is of course a major shock. Various security practitioners with a sense of black humor are calling it ‘shellshock’ because that is the exact state of mental trauma in which an admin ends up in when faced with a bug that has multiple attack vectors, and requires urgent patching on hundreds or even thousands of systems (s)he might operate, whether or not they let people ssh into their servers at all.

There are even servers that might not be all that easy to patch. At the low cost and ultra high volume end, embedded systems whose typical upgrade path is “throw old box into the trash, drive over to the store and buy a new one”, all the way to ICS and SCADA systems.

It is a major relief to every ISP security practitioner that Bashbleed doesn’t affect most consumer grade broadband routers and wifi access points, which run busybox, and therefore the ash shell, which is not vulnerable to bashbleed. Other embedded, ICS and SCADA systems, however, continue to be vulnerable. These systems power everything from auto parts factories to dams and nuclear power plants, and yet many just can’t be upgraded—something that people like Joe St.Sauver of the University of Oregon have been saying for over a decade now.

So, what makes this bug so nasty? The sheer number of attack vectors—mailservers, webservers (including the administration web interfaces of some seriously expensive gear)—and the fact that once this vector is exploited, ANY system command that can be entered on a bash shell is available for you to execute.

So you could send an unpatched qmail smtp server something like —

MAIL FROM:<() { :; }; nc -e /bin/bash localhost 7777>

and have it spawn a bash shell for you.

This bash shell could then be used to, say, download a kernel exploit, or perhaps a spambot, from a malicious webserver. There are already bots that are doing something of the sort.

Now for a few best practices straight out of security 101:

Lesson 1 – Always Sanitize Input Data in applications that you develop, or run.

A mailserver that validated SMTP commands you fed it according to the SMTP standard RFC 5322 would absolutely refuse to accept and pass on to the underlying system anything that wasn’t a syntactically valid email address—which that shell command is most definitely not.

Nor would a webserver that sanitized HTTP header data.

Or a web application that sanitized data input. A few examples of how to write web applications that sanitize input data are available in this older but still excellent article by Philip Tellis). More information is available in this SANS security checklist for web application development.

Lesson 2 – Stay Informed

Run regular scans of your network using something like Nessus or IBM Appscan. Of course, don’t ever try to scan others’ systems or networks, unless you have an explicit agreement with that system or network to pentest them for vulnerabilities.

Needless to say, unless you’re a research organization that does global internet scanning and measurement of vulnerabilities, steer clear of trying to scan the entire Internet for vulns unless and until you have a clear procedure and protocol in place, and are transparent about your efforts.

In both cases, you’re more than likely to get taken for a malicious actor if you do that—and almost certainly are going to be in breach of your Internet provider’s terms of use policy.

Look out for unsanitized input data that leads to code injection, as part of a comprehensive vulnerability assessment.

Pay attention to your logs—preferably with tools that let you have a 360 degree view of your logs. You can use something like Logentries and New Relic, or Splunk to aggregate logs from across your devices and track them for anomalous events.

Use network analysis tools such as Netflow and Jflow, as well as security systems that leverage them, to mine network traffic for attack information.

Lesson 3 – Make fixes (relatively) easy and pain free

Treat every update equally—whether it is a server update or a code update. Deploy it as a package and leverage your existing distribution’s package management network to perform updates. Even customized updates that you roll with patches that you need. This fits perfectly within the laser focus on automating and tracking deployment of builds that characterizes a DevOps environment—which you should strongly consider adopting, for a variety of reasons.

Use something like Spacewalk or Redhat’s Satellite, which is derived from Spacewalk, to update Redhat based linux distributions such as RHEL or Centos, SuSE manager for SuSE linux, or other equivalent systems available for your Linux distribution.

For those who run rather more diverse environments where you end up having to patch everything from SSL load balancers to other Unices, you will need custom Chef and Puppet recipes, or possibly a commercial product like IBM’s SmartCloud Orchestrator

Lesson 4 – Keep track of the security scene

Read various security blogs, follow known security researchers on facebook and twitter, and do work with your colleagues to track the mailing lists for software that you deploy and care about.

There’s a new bug almost every other day in one product or the other, some minor and some highly serious. Being aware of what is going on that affects your environment is essential.

In conclusion—I’ll point out that Bruce Schneier, channeling ‘This is Spinal Tap’, said about Heartbleed—“‘Catastrophic’ is the right word. On the scale of 1 to 10, this is an 11.”

I am not sure he’s rated Bashbleed on that same scale, but I personally would rate this just as serious, if not more. Go out there and look for whatever needs to be secured on your systems. Now, if you haven’t ever done so before. Do it now too, even if you do this religiously on a regular basis.