As you’ve undoubtedly heard, the Equifax credit reporting agency was hit by a major attack, exposing the personal data of 143 million Americans and many more people in other countries. There’s been a lot of discussion of liability; as of a few days ago, at least 25 lawsuits had been filed, with the state of Massachusetts preparing its own suit. It’s certainly too soon to draw any firm conclusions about who, if anyone, is at fault—we need more information, which may not be available until discovery during a lawsuit—but there are a number of interesting things we can glean from Equifax’s latest statement.

First and foremost, the attackers exploited a known bug in the open source Apache Struts package. A patch was available on March 6. Equifax says that their “Security organization was aware of this vulnerability at that time, and took efforts to identify and to patch any vulnerable systems in the company’s IT infrastructure.” The obvious question is why this particular system was not patched.

One possible answer is, of course, that patching is hard. Were they trying? What does “took efforts to identify and to patch” mean? Were the assorted development groups actively installing the patch and testing the resulting system? It turns out that this fix is difficult to install:

You then have to hope that nothing is broken. If you’re using Struts 2.3.5 then in theory Struts 2.3.32 won’t break anything. In theory it’s just bug fixes and security updates, because the major.minor version is unchanged. In theory.

In practice, I think any developer going from 2.3.5 to 2.3.32 without a QA cycle is very brave, or very foolhardy, or some combination of the two. Sure, you’ll have your unit tests (maybe), but you’ll probably need to deploy into your QA environment and do some kind of integration testing too. That’s assuming, of course, that you have a compatible QA environment within which you can deploy your old, possibly abandoned application.

Were they trying hard enough, i.e., devoting enough resources to the problem?

Ascertaining liability here—moral and/or legal—can’t be done without seeing the email traffic between the security organization and the relevant development groups; you’d also have to see the activity logs (code changes, test runs, etc.) of these groups. Furthermore, if problems were found during testing, it might take quite a while to correct the code, especially if there were many Struts apps that needed to be fixed.

As hard as patching and testing are, though, when there are active exploitations going on you have to take the risk and patch immediately. That was the case with this vulnerability. Did the Security group know about the active attacks or not? If they didn’t, they probably aren’t paying enough attention to important information sources. Again, this is information we’re only likely to learn through discovery. If they did know, why didn’t they order a flash-patch? Did they even know which systems were vulnerable? Put another way, did they have access to a comprehensive database of hardware and software systems in the company? They need one—there are all sorts of other things you can’t do easily without such a database. Companies that don’t invest up front in their IT infrastructure will hurt in many other ways, too. Equifax has a market capitalization of more than $17 billion; they don’t really have an excuse for not running a good IT shop.

It may be, of course, that Equifax knew all of that and still chose to leave the vulnerable servers up. Why? Apparently, the vulnerable machine was their “U.S. online dispute portal”. I’m pretty certain that they’re required by law to have a dispute mechanism, and while it probably doesn’t have to be a website (and some people suggest that complainants shouldn’t use it anyway), it’s almost certainly a much cheaper way to receive disputes than is paper mail. That opens the possibility that there was a conscious decision that taking the risk was worthwhile. Besides, if many applications needed patching and they had limited development resources, they’d have had to set priorities on whic web servers were more at risk. Again, we need more internal documents to know.

Some text in the announcement does suggest either ignorance or a conscious decision to delay patching—the timeline from Equifax implies that they were able to patch Struts very quickly after observing anomalous network traffic to that server. That is, once they knew that there was a specific problem, rather than a potential one, they were able to respond very quickly. Alternatively, this server was on the “must be patched” list, but was too low down on the priority list until the actual incident was discovered.

We thus have several possible scenarios: difficulty in patching a large number of Struts applications, ignorance of the true threat, inadequate IT infastructure, or a conscious decision to wait, possibly for priority reasons. The first and perhaps last would seem to be exculpatory; the others would seem to leave the company in a bad moral position. But without more data we can’t distinguish among these cases.

A more interesting question is why it took Equifax so long to detect the breach. They did notice anomalous network traffic, but not until July 29. Their statement says that data was exposed starting May 13. Did they have inadequate intrusion detection? That might be more serious from a liability standpoint—unlike patching, running an IDS doesn’t risk breaking things. You need to tune your IDS correctly to avoid too many false positives, and you need to pay attention to alerts, but beyond dispute an enterprise of Equifax’s scale should have such deployed. It is instructive to read what Judge Learned Hand wrote in 1932 in a liability case when some barges sank because the tugboat did not have a weather radio:

Indeed in most cases reasonable prudence is in fact common prudence; but strictly it is never its measure; a whole calling may have unduly lagged in the adoption of new and available devices. It may never set its own tests, however persuasive be its usages. Courts must in the end say what is required; there are precautions so imperative that even their universal disregard will not excuse their omission… But here there was no custom at all as to receiving sets; some had them, some did not; the most that can be urged is that they had not yet become general. Certainly in such a case we need not pause; when some have thought a device necessary, at least we may say that they were right, and the others too slack… We hold [against] the tugs therefore because [if] they had been properly equipped, they would have got the Arlington [weather] reports. The injury was a direct consequence of this unseaworthiness.

It strikes me as entirely possible that Equifax’s exposure is greater on this issue than on patching.

This is a big case, affecting a lot of people. The outcome is likely to change the norms of how corporations world-wide protect their infrastructure. I hope the change will be in the right direction.

* * *

Update – Monday, Sep 18:

A news report today claims that Equifax was hacked twice, once in March (which is very soon after the Struts vulnerability was disclosed) and once in mid-May. The news article does not say if the same vulnerability was exploited; it does, however, say that their sources claim that “the breaches involve the same intruders”.

If it was the same exploit, it suggests to me one of the possibilities I mentioned above: that the company lacked an comprehensive softare inventory. After all, if you know there’s a hole in some package and you know that you’re being targeted by attackers who know of it and have used it against you, you have very strong incentive to fix all instances immediately. That Equifax did not do so would seem to indicate that they were unaware that they were still vulnerable. In fact, the real question might be why it took the attackers so long to return. Maybe they couldn’t believe that that door would still be open…

On another note, several people have sent me notes pointing out that Susan Mauldin, the former CSO at Equifax, graduated with degrees in music, not computer science. I was aware of that and regard it as quite irrelevant. As I and others have pointed out, gender bias seems to be a more likely explanation for the complaints. And remember that being a CSO is a thankless job.

Update – Thursday, Sep 21:

In the Sep. 18 update above, I noted that Equifax had been breached in March, and quoted the article as saying that the attackers had been “the same intruders” as in the May breach. In a newer news report, Equifax has denied that:

“The March event reported by Bloomberg is not related to the criminal hacking that was discovered on 29 July,” Equifax’s statement continues. “Mandiant has investigated both events and found no evidence that these two separate events or the attackers were related. The criminal hacking that was discovered on 29 July did not affect the customer databases hosted by the Equifax business unit that was the subject of the March event.”

So: I’ll withdraw the speculation I posted about this incident confirming one of my hypotheses and wait for further, authoritative information. I repeat my call for public investigations of incidents of this scale.

Also worth noting: Brian Krebs was one of the very few to report the March incident.