|
A student at a well-known US university wrote me and asked whether, given the huge national interest in getting the industry to unite behind (at least) one format, did I think that the FTC should’ve played a stronger role in pushing the industry to adopt an authentication format?
I said:
Nope. Part of the reason it’s taking so long to agree on a standard is that the process is infested with academic theoreticians who are more interested in arguing about hypotheticals and pushing their pet spam solutions than in doing something useful, but the main reason is that it’s a hard problem. Making changes to the e-mail system is akin to open heart surgery on a beating heart, in that you can’t stop it while you’re working on it, and the consequences of an ill considered change are bad.
There is a perception in some circles that any authentication is better than no authentication and we should just pick something and get on with it, but that’s wrong. Bad authentication is worse than no authentication because a bad system will sometimes pass bad mail and fail good mail, meaning either that mail gets even less reliable than it is now, or more likely that mail systems only pretend to use it and outsiders end up scratching their heads wondering why it didn’t help. (Most of the people who claim to use SPF or Sender-ID are just pretending, the few foolish or desperate ones that really do reject lots of legit mail.)
Because of all of the patent license nonsense in the IETF MARID group, we never got to the point of addressing the main problems with SPF and Sender-ID which is that they don’t work. If you are a big company that sends all of its mail from one place, and none of your recipients ever forward their mail, then SPF and Sender-ID mostly work. If you’re anyone else, they’re hopelessly broken. For example, Cornell assigns every student an email address, and the address is theirs to keep forever. After students leaves, they can log into Cornell’s mail server and tell it to forward their mail to their current address, which is nice since it means that their mail addresses don’t have to change when they switch ISPs, or when the cable company decides to change its name from mediaone.com to comcast.net. As a result, there are thousands of people with valid cornell.edu addresses sending mail from whatever ISP they happen to use, something that SPF and Sender-ID are utterly unable to deal with. Even a normal ISP like Comcast has enough roaming users to make SPF unreliable.
DomainKeys better matches the way that mail is really sent although it also has some technical details that need to be worked out. At the moment it’s stuck in a swarm of the aforementioned infestation, which seems to be a chronic problem in the IETF whenever actual engineering is attempted. But I don’t think pressure from the FTC would help, partly because pressure to Do Something would as likely as not lead to a broken Something, partly because the people delaying DK or any other authentication scheme are unlikely to be impressed by the FTC.
He later wrote back and commented that he’d seen a lot of apparent momentum behind a hybrid of SPF/Sender-ID and DKIM, although he’s not clear what such a hybrid would look like.
I answered:
And neither is anyone else. This is a political proposal from people who don’t understand the technology, not a technical one. SPF/S-ID and DK/DKIM ask fundamentally different questions. SPF asks where the message came from, sort of like looking at the postmark on an envelope, while DK asks who sent it, sort of like writing mail on hard-to-forge letterhead. They don’t interfere, and you can use both on the same message, but combining them is sort of like the old Mad Magazine combined bicycle pump and orange juice squeezer, two unrelated parts duct taped together.
There are three somewhat separate factions in the authentication fight. One is the ESPs, companies that send bulk mail. They send vast amounts of mail from fixed sources, referred to slightly unfairly as spam cannons. They are unusual in that they care far more than their recipients do about getting their mail delivered, and they all work for third parties so they have always wanted to be able to claim that they’re just the postman and the responsibility for abuse rests on their clients. Their mail varies from squeaky clean to rather spammy depending on the ESP. SPF and Sender-ID works fine for them since they’re all fixed sources.
The second faction is ISPs. They’re the major mail recipients, and they send a combination of normal user mail and a lot of spam from zombies. SPF works fairly poorly since they have a lot of roaming users. Web mail systems like Yahoo and Hotmail and hosting companies also fall into this category although they tend to send no zombie mail but (particularly Hotmail) spam due to crooks mechanically signing up for lots of accounts and spamming through them.
The third faction is institutions, corporate networks and the like. They tend to send modest amounts of mail and no spam since they have corporate firewalls that keep the zombie-ware out. SPF works OK for them, except perhaps for salesmen on the road, but their aggregate volume is much less than the ESPs, and the mail all goes through a central gateway so dropping in a DKIM signer wouldn’t be a big problem.
This means that to a first approximation, if you reject all of the mail that passes SPF, you won’t lose much that you care about since it’ll mostly be ESPs with a sprinkling of spammers who set up SPF. Perverse but true. You’ll lose some transactional mail, but that’s a special category since transactional messages are high value to both senders and recipients and are likely to be special cased in any authentication and reputation systems.
A more important issue, one on which the silence is deafening, is that authentication systems are useless without some sort of reputation database. You get a message, it’s 100% authenticated that it came from flurble.net but you’ve never heard of flurble.net. Now what? The unstated assumptions seem to be that for now we all have our informal private lists of friendly domains that we will whitelist, and eventually there will be shared reputation systems to plug into. The faith in shared reputation systems is touching, particularly considering all of the moaning and groaning there is about DNSBLs, the reputation systems that exist now.
He referred me to a paper by John Palfrey at the Berkman Center on “A Comparative Analysis of Spam Laws: The Quest for Model Law”, and noted that most spam laws outside the US are opt-in, but CAN SPAM is opt-out. He couldn’t figure out, then, how an opt-out solution is workable… any ideas on what the argument here is?
The main proponent of opt-out is the Direct Marketing Association, which is well funded and politically well connected and cannot imagine how e-mail could be any different from postal mail. Some years ago Bob Wientzen, then the head of the DMA, told me how wonderful it would be to put a $1 coupon for Tide into ever US consumer’s e-mail mailbox. He could not or would not grasp that my mailbox would also contain a thousand other coupons for a thousand other products that I don’t use. As far as I can tell, the DMA has never asked their members what the position on opt-out should be, and the members would say no if asked since the number of DMA members that actually send opt-out ads can be counted on your fingers.
I looked at Palfrey’s paper and although it has some good ideas, a lot of it makes some naive assumptions and would lead to perverse results. A blanket ban on UCE is a terrible idea, since the problem with spam is that it is sent in bulk to people who don’t want it, not that it is commercial. Non-bulk commercial messages aren’t a problem, e.g., I don’t mind a legitimate message saying “I noticed your mailing list message asking for advice on building a pickle peeler and if you don’t want to do it yourself, I sell them for under $20” and neither would anyone else if we weren’t all hypersensitized by spam pretending to be personal. On the other hand, a million messages saying “Buddha Loves You” is just as disruptive as a million ads for Tide.
It’s a complicated problem.
Sponsored byRadix
Sponsored byVerisign
Sponsored byCSC
Sponsored byVerisign
Sponsored byDNIB.com
Sponsored byWhoisXML API
Sponsored byIPv4.Global
I also wonder if any of these authentication schemes would be able to deal properly with the .(JavaScript must be enabled to view this email address) addresses supported by the .name registry, where the user doesn’t actually own last.name by itself. In this case, the mail is always forwarded somewhere else.
DKIM provides multiple “selectors” per domain for multiple keys. If the .name registry wanted to give each user his own selector for mail signing, they could do that, allowing them to cancel the selectors for people who misused them. If they’re going to sell email addresses individually, that’s not my problem and they’d best have a plan to police their misbehaving users if they want their mail to be deliverable.
The whole point of authentication schemes is to assign responsibility for mail, in DKIM’s case at the domain level. There are a lot of people, notably ESPs who mail for other companies, who really wish they could disclaim responsibility for mail with their name on it. Tough noogies. If it’s got your name and your signature, it’s your problem.
I don’t understand the resistance to SPF/S-ID because it doesn’t have to be universally implemented to provide clear benefits to those who do implement it.
In your example with Cornell e-mail addresses, Cornell would be free to publish an SPF record that put no restriction on who could send mail from the cornell.edu domain. (Or, they could require that outgoing mail be routed through their mail server, but I’ll even accept for the sake of argument that this would present an unbearable burden). So SPF would provide no protection to Cornell against non-Cornell users spoofing mail from cornell.edu. But likewise it would do no harm to Cornell. At the same time, a company rightly concerned with protecting its image (like, for example, PayPal) would publish a very restrictive SPF record thus ensuring that any SPF-aware mail transport agent could immediately recognize a malicious email attempting to phish by spoofing a paypal.com return address.
While I don’t think that publishing SPF records is such an administrative burden, the whole thing would still work even if SPF adoption was low enough that MTAs needed to default to passing a message from a domain without any SPF information published. So the system can only do good—why the resistance?
There’s two somewhat separate reasons not to waste any effort on SPF.
One is that it’s considerably harder to get SPF right, even for simple configurations, than it looks. I was talking to someone at Godaddy a while ago that said he was fixing about two broken customer SPF records per day every day. And even if it’s right, senders rarely have any idea which of their recipient addresses are in fact forwarders. In the Cornell example, some addresses are really at Cornell, some are forwarded for alumni which will make the SPF on the next hop fail, and there is deliberately no way to tell the difference. This is what I meant when I said that people who say they use SPF are just pretending.
The other reason is keeping our powder dry. Getting people to do any sort of broad mail software upgrade is a huge amount of work, and I most definitely do not want to tell anyone to go to the effort of adding SPF support, knowing that it will never check more than a small fraction of their incoming mail and a fair amount of what it tells them will be wrong.
John: Great post. Yes, the reputation database piece is key. I’ve been thinking about some ways to do reputation right, and in way that will make it feasible for large and small legitimate mail sources to maintain a good reputation. I’ve been developing and hope to make and see announcements about ‘competitive’ services anon. FWIW, Meng is working on one, IIRC.
Daniel Tobias: CSV, like DKIM, is an authentication scheme (http://en.wikipedia.org/wiki/Certified_Server_Validation) that would work fine with .name email addresses.
Unlike SPF and DKIM, it is easy to set up. (The guy at Godaddy would have no records to fix, instead of 2 a day. All Godaddy customers would use Godaddy’s mail server, which would already have a correct CSV record and reputation. Most domains won’t need CSV records at all…) Also, it will cover all legit mail with a relatively small number of records. It’s also perhaps 1000% more efficient than DKIM or SPF.
But the adoption motivations are KEY. A solution to the spam problem would be a problem for much of the ISP and ESP factions. With the most egregious spammers going out of business, more focus will be on them, as sources of mixtures of wanted and unwanted mail. The ISPs, and to a lesser degreee, institutions, are having to put more resources into keeping their networks free of zombie spam. All ISPs benefit from there being less spam to store and ship around. ESPs and ISPs that keep their noses clean have the largest net gain from adopting email authentication, which is probably why they are early adopters. There’s a lot of folks who would like to maintain the status quo.
Clearly, important big businesses in the institional faction, despite the large costs instituions pay to receive spam, prefer the status quo. Since the U.S. Congress does their bidding, the sick joke that is CAN-SPAM makes clear what their goals are.