As the year comes to a close, it is important to reflect on what has been one of the major actions in the anti-spam arena this year: the quest for email authentication. With email often called the “killer app” of the Internet, it is important to reflect on any major changes proposed, or implemented that can affect that basic tool that many of us have become to rely on in our daily lives. And, while many of the debates involved myriads of specialized mailing lists, standards organizations, conferences and even some government agencies, it is important for the free and open source software (FOSS) community as well as the Internet community at large, to analyze and learn lessons from the events surrounding email authentication in 2004.

“THE GHOST OF CHRISTMAS PAST”

The quest for email authentication did not start from scratch. Authentication systems are a well known field in computer security, and have been researched for quite some time. Nevertheless, it is only during this past year that email authentication has gained a prominent push mainly due to the ever increasing spam problem. As well known, the original email architecture and protocols was not designed for an open network such as the Internet. Therefore, the original designers failed to predict the virtual tidal wave of junk email that took advantage of lack of authentication in the Internet email. As the result, a junk email filter is considered one of the essential tools any Internet citizen must have in his toolkit today.

The push towards email authentication started in earnest with the publication of a proposal called RMX by a German engineer called Hadmut Danisch in early 2003. While other previous proposals have been published, none have gained any kind of traction. Hadmut’s proposal on the other hand coincided with the opening of the Anti-Spam Research Group (ASRG) of the Internet Research Task Force (IRTF), which as an affiliate body of the IETF. The IETF created and currently maintains the Internet email standards, and an IETF affiliate was a logical body to work on addressing the spam problem on the Internet at large. Being that the ASRG brought together a sizable chunk of the anti-spam world, RMX gained more exposure that none of the previous work in the field ever had. What followed was a succession of proposals forked off the original RMX proposal until the spring of 2004 when most of them were basically confined to the dustbin of history together with RMX. In the end, only two proposals with any sizable following were left: Sender Policy Framework (SPF) and Microsoft’s Caller-ID.

The author of SPF, Meng Wong, managed to attract a large community to his proposal, giving it a much larger deployed base than any competitor. In many ways this effort can be compared to some of the open source projects, except this time this was an open standard rather than a piece of software. On the other side of the ring, so to speak, was Microsoft which surprised the email world with their own proposal called Caller-ID at the RSA conference in early 2004. Eventually, the IETF agreed to consider standardization of email authentication by opening a working group called MARID in March of 2004. With the merger of SPF and Microsoft’s new Sender-ID proposal, hopes were running high about the coming success of email authentication and the coming demise of spam. Yet, ironically this working group earned itself a record by being one of the shortest in the existence of the IETF - it has lasted a little over six months until being formally shutdown in September of 2004.

“ALL THAT IS GOLD DOES NOT GLITTER”

During the work of IETF’s MARID group the quest for the email authentication begun to permeate circles outside the usual cadre of anti-spam geeks. Technology publications, and even the mass media have begun to take note of the efforts occurring on an obscure mailing list tucked away among 200 other even more obscure groups, prodded in many cases by the public relations spokesmen of various companies in the anti-spam space, including Microsoft. Yet in many ways that was one of the fatal blows to the group and any hope of a common standard for email authentication.

Several major issues arose during the operation of the working group. The first major issue that has been bubbling beneath the surface was technical in nature. SPF has come from a group of proposals that worked with the parts of the email infrastructure that was unseen by most users. This included email servers that exchanged email among ISPs and were unseen. In the technical lingo this type of authentication was known as “path authentication”. It focused on authenticating the path the message took place between servers, and dealt with machines instead of end users. Sender-ID approached the problem from a different viewpoint. Prodded by financial companies and the fact that Microsoft itself makes more email client software than server software, Sender-ID dealt with the end user. It focused on “message authentication”, based on the path the message took. While the goals make have been admirable, many technical questions arose as to whether Sender-ID would work. Most of them were rooted in the basic differences between path authentication vs. message authentication, and remained unresolved.

The second major issue that arose was one of intellectual property rights. Microsoft filed for patents on parts of Sender-ID and was not forthcoming with information during the operation of the MARID WG. While the actual patent application were eventually published towards the end of life of the WG that came too late. The damage to the trust among the group members, and different parts of the community has already been done. The main point of contention was not necessarily the patents applications themselves - rather it was the mandatory patent license that Microsoft had drawn up. The language in the Sender-ID patent license was construed in a way that prevents use by any software licensed under the General Public License (GPL). Whether that was intentional or not we may never know, but the trust between Microsoft and the FOSS community which was strenuous at best was broken.

The third major issue which played itself outside the mailing lists and hallways of the anti-spam world was the media. Given that the spam problem was only increasing, the media pounced on what was seen as the golden grail for stopping spam. Unfortunately, as most reporters are not knowledgeable in either Internet architecture or email protocols, they frequently reported email authentication as the final cure for spam. These created great expectations for email authentication which were blown away once the hard truth settled in: email authentication did not stop spam. Unlike what many had believed, email authentication did not address the spam problem directly. Rather, it was only the first step towards a larger solution with reputation and accreditation systems planned for the future. However, as this truth sunk in, many of the companies and community members were not as positive towards email authentication as before.

The various disagreements, technical and non-technical, led some of the group participants to create their own alternatives proposals or look to crypto-solutions such as Yahoo’s DomainKeys. As a result, any useful work of the MARID group slowed to a crawl with the IETF eventually shutting down the group. A major factor in that decision was letters from two large members in the FOSS community against Sender-ID: the Apache Foundation and the Debian Project.

“LET’S VISIT UNCLE SAM”

With the shutdown of MARID WG in September of 2004, both Sender-ID and SPF were left to fend for their own. While some have assumed that Sender-ID was left of the dead after being rejected by the IETF shortly before the closure of MARID, Microsoft was quietly gathering support for Sender-ID among the industry. Microsoft’s goals become clear at the FTC’s Email Authentication Summit in November of 2004: Sender-ID was pushed as an accepted email authentication standard to be mandated by the FTC. Among the sizable PR gains that Microsoft gained was the endorsement of Sender-ID by AOL, and a letter signed by representatives of 25 major email companies and ISPs, a list which curiously included Meng Wong, the author of SPF. The PR advantage was so great, that SPF was not even listed on the FTC’s website for the conference. At the same time, other alternative proposals such as CSV and BATV have begun promulgating among the industry, all of which born during the death throes of MARID.

The SPF community being faced with the choice of joining or rejecting Sender-ID, was split. Majority of the community as judging by the mailing list traffic opposed Sender-ID/SPF combination. Nevertheless, some members including Meng Wong, the original author, endorsed Sender-ID. This has led to a lot of infighting with an election of an “SPF Council”. At this time, the SPF community is the midst of a political discussion about its future.

At the same time, a separate low-key effort in the IETF is taking place to address some of the cryptography solutions for Internet email. Proposals such as Yahoo’s DomainKeys, Cisco’s IdentifiedMail, etc. seek to achieve “message authentication” promised by Sender-ID but on a much more solid technical ground and with less IPR and PR issues. This effort is purposely left low key with even the mailing list itself hard to find, and certainly no media stories promising the end of spam. The IETF-MAILSIG effort as this is now called seeks to avoid the same problems that doomed MARID with hopes of developing useful technologies to reduce spam. Nevertheless, this effort was high-key enough for some of the companies involved to show case it at the FTC’s summit. Needless to say, the FTC is staying silent on its plans.

WHAT THE FUTURE HOLDS

While we still don’t have workable email authentication, the Sender-ID/SPF saga did accomplish a lot in many other ways. These events have shown to the technology community at large that the FOSS world plays an ever increasing role in the Internet as whole. The Apache Foundation and the Debian Project carried enough weight to the IETF to consider their opinion, marking probably the first time that FOSS opinions carried significant weight in the standards process.

This debacle has also lead to an increased awareness of the growing problems in the patent system with Sender-ID being cited as a prime example of a patent system gone wrong. While smaller sagas such as PanIP’s rampage on small e-businesses, Acacia’s assault of video streaming and other similar incidents have been happening for a while, the Sender-ID/IETF story has brought this issue to the forefront of the Internet community for at least a short time. What has followed has been positive developments with governments, corporations and individuals recognizing the increasing problems in today’s patent system and some beginning to seek reform.
As for spam, Microsoft, Cisco, the SPF community and many others are still working on it. Some of the positive developments coming out of the Sender-ID episode have been an increased awareness of how the email architecture actual works and the increased realization that better coordination among the Internet community is necessary.

As for email authentication - there is still 2005…