A student at a well-known US university wrote me and asked whether, given the huge national interest in getting the industry to unite behind (at least) one format, did I think that the FTC should’ve played a stronger role in pushing the industry to adopt an authentication format?

I said:

Nope. Part of the reason it’s taking so long to agree on a standard is that the process is infested with academic theoreticians who are more interested in arguing about hypotheticals and pushing their pet spam solutions than in doing something useful, but the main reason is that it’s a hard problem. Making changes to the e-mail system is akin to open heart surgery on a beating heart, in that you can’t stop it while you’re working on it, and the consequences of an ill considered change are bad.

There is a perception in some circles that any authentication is better than no authentication and we should just pick something and get on with it, but that’s wrong. Bad authentication is worse than no authentication because a bad system will sometimes pass bad mail and fail good mail, meaning either that mail gets even less reliable than it is now, or more likely that mail systems only pretend to use it and outsiders end up scratching their heads wondering why it didn’t help. (Most of the people who claim to use SPF or Sender-ID are just pretending, the few foolish or desperate ones that really do reject lots of legit mail.)

Because of all of the patent license nonsense in the IETF MARID group, we never got to the point of addressing the main problems with SPF and Sender-ID which is that they don’t work. If you are a big company that sends all of its mail from one place, and none of your recipients ever forward their mail, then SPF and Sender-ID mostly work. If you’re anyone else, they’re hopelessly broken. For example, Cornell assigns every student an email address, and the address is theirs to keep forever. After students leaves, they can log into Cornell’s mail server and tell it to forward their mail to their current address, which is nice since it means that their mail addresses don’t have to change when they switch ISPs, or when the cable company decides to change its name from mediaone.com to comcast.net. As a result, there are thousands of people with valid cornell.edu addresses sending mail from whatever ISP they happen to use, something that SPF and Sender-ID are utterly unable to deal with. Even a normal ISP like Comcast has enough roaming users to make SPF unreliable.

DomainKeys better matches the way that mail is really sent although it also has some technical details that need to be worked out. At the moment it’s stuck in a swarm of the aforementioned infestation, which seems to be a chronic problem in the IETF whenever actual engineering is attempted. But I don’t think pressure from the FTC would help, partly because pressure to Do Something would as likely as not lead to a broken Something, partly because the people delaying DK or any other authentication scheme are unlikely to be impressed by the FTC.

He later wrote back and commented that he’d seen a lot of apparent momentum behind a hybrid of SPF/Sender-ID and DKIM, although he’s not clear what such a hybrid would look like.

I answered:

And neither is anyone else. This is a political proposal from people who don’t understand the technology, not a technical one. SPF/S-ID and DK/DKIM ask fundamentally different questions. SPF asks where the message came from, sort of like looking at the postmark on an envelope, while DK asks who sent it, sort of like writing mail on hard-to-forge letterhead. They don’t interfere, and you can use both on the same message, but combining them is sort of like the old Mad Magazine combined bicycle pump and orange juice squeezer, two unrelated parts duct taped together.

There are three somewhat separate factions in the authentication fight. One is the ESPs, companies that send bulk mail. They send vast amounts of mail from fixed sources, referred to slightly unfairly as spam cannons. They are unusual in that they care far more than their recipients do about getting their mail delivered, and they all work for third parties so they have always wanted to be able to claim that they’re just the postman and the responsibility for abuse rests on their clients. Their mail varies from squeaky clean to rather spammy depending on the ESP. SPF and Sender-ID works fine for them since they’re all fixed sources.

The second faction is ISPs. They’re the major mail recipients, and they send a combination of normal user mail and a lot of spam from zombies. SPF works fairly poorly since they have a lot of roaming users. Web mail systems like Yahoo and Hotmail and hosting companies also fall into this category although they tend to send no zombie mail but (particularly Hotmail) spam due to crooks mechanically signing up for lots of accounts and spamming through them.

The third faction is institutions, corporate networks and the like. They tend to send modest amounts of mail and no spam since they have corporate firewalls that keep the zombie-ware out. SPF works OK for them, except perhaps for salesmen on the road, but their aggregate volume is much less than the ESPs, and the mail all goes through a central gateway so dropping in a DKIM signer wouldn’t be a big problem.

This means that to a first approximation, if you reject all of the mail that passes SPF, you won’t lose much that you care about since it’ll mostly be ESPs with a sprinkling of spammers who set up SPF. Perverse but true. You’ll lose some transactional mail, but that’s a special category since transactional messages are high value to both senders and recipients and are likely to be special cased in any authentication and reputation systems.

A more important issue, one on which the silence is deafening, is that authentication systems are useless without some sort of reputation database. You get a message, it’s 100% authenticated that it came from flurble.net but you’ve never heard of flurble.net. Now what? The unstated assumptions seem to be that for now we all have our informal private lists of friendly domains that we will whitelist, and eventually there will be shared reputation systems to plug into. The faith in shared reputation systems is touching, particularly considering all of the moaning and groaning there is about DNSBLs, the reputation systems that exist now.

He referred me to a paper by John Palfrey at the Berkman Center on “A Comparative Analysis of Spam Laws: The Quest for Model Law”, and noted that most spam laws outside the US are opt-in, but CAN SPAM is opt-out. He couldn’t figure out, then, how an opt-out solution is workable… any ideas on what the argument here is?

The main proponent of opt-out is the Direct Marketing Association, which is well funded and politically well connected and cannot imagine how e-mail could be any different from postal mail. Some years ago Bob Wientzen, then the head of the DMA, told me how wonderful it would be to put a $1 coupon for Tide into ever US consumer’s e-mail mailbox. He could not or would not grasp that my mailbox would also contain a thousand other coupons for a thousand other products that I don’t use. As far as I can tell, the DMA has never asked their members what the position on opt-out should be, and the members would say no if asked since the number of DMA members that actually send opt-out ads can be counted on your fingers.

I looked at Palfrey’s paper and although it has some good ideas, a lot of it makes some naive assumptions and would lead to perverse results. A blanket ban on UCE is a terrible idea, since the problem with spam is that it is sent in bulk to people who don’t want it, not that it is commercial. Non-bulk commercial messages aren’t a problem, e.g., I don’t mind a legitimate message saying “I noticed your mailing list message asking for advice on building a pickle peeler and if you don’t want to do it yourself, I sell them for under $20” and neither would anyone else if we weren’t all hypersensitized by spam pretending to be personal. On the other hand, a million messages saying “Buddha Loves You” is just as disruptive as a million ads for Tide.

It’s a complicated problem.