The following excerpt is from the Free Software Magazine, March 2005 Issue, written by Kirk Strauser. To read the entire article, you may download the magazine here [PDF]. Also thanks to Yakov Shafranovich for making us aware of this publication.

SMTP is an abbreviation for “Simple Mail Transfer Protocol”, and is the standard internet protocol for sending email from one system to another. Although the word “simple” belies the inherent complexity of the protocol, SMTP has proved to be a remarkably robust, useful, and successful standard. The design decisions that made it so useful, though, have given spammers and infectious code an easy way to spread their unwanted messages. Its recent evolution reflects the tug-of-war between those unsavory players and the administrators who want to protect their systems and their users.

Early history

When Jonathan Postel wrote the SMTP definition RFC 821 in 1982, the internet was minuscule in comparison with today’s pervasive mix of commercial, governmental, and private interests. At that time, it mostly comprised a small collection of military installations, universities, and corporate research laboratories. Connections were slow and unreliable, and the number of hosts was small enough that all of the participants could recognize each other. In this early setting, SMTP’s emphasis on reliability instead of security was reasonable and contributed to its wide adoption. Most users helped each other by configuring their mail servers as “open relays”. That meant that each cooperative host would accept mail meant for other systems and relay it toward its final destination. This way, email transfer on the fledgling internet stood a reasonable chance of eventual delivery. Most administrators were happy to help their peers ? and receive their help in return.

Spam has existed since at least 1978, when an eager DEC sales representative sent an announcement of a product demonstration to a couple hundred recipients. The resulting outcry was sufficient to dissuade most users from repeating the experiment. This changed in the late 1990s: millions of individuals discovered the internet and signed up for inexpensive personal accounts and advertisers found a large and willing audience in this new medium.

Spam becomes a problem

The helpful nature of open relays was among the first victims of the spam influx. In the young commercial internet, high-speed connections were prohibitively expensive for individuals and small businesses. Spammers quickly learned that it was easy to send a small number of messages ? with recipient lists thousands of entries long ? to helpful corporate servers, which would happily relay those messages to their targets. Administrators noticed sudden spikes in their metered service bills (and in the number of complaints) and realized that they could no longer help their peers without incurring significant monetary costs and bad will.

First steps to secure the internet

Although the nature of the problem was clear, the solutions were not. The SMTP standard, which was designed with reliability as a key feature, had to be re-implemented to purposefully discard certain, recognized messages. This was a foreign idea and no one was sure how to proceed. The first step was to close the open relays. Administrators argued loudly, and at great length, whether this was a necessary move, or even a good one at all. In the end though, it was universally agreed that the trusting nature of the old internet was dead, and in fact harmful in the current setting. Some users took this idea a step farther and decided that they would not only close their own systems, but would no longer accept messages from other open relays. They eventually began to share their lists of those relays with peers by adding specially formatted entries to their domain name servers and allowing their neighbors to query their servers for this data. This was the beginning of the first “DNS blackhole lists”, and they were highly controversial. For example, administrators debated whether it was acceptable to actively test remote servers to see if they were open relays, and discussed which procedures a system administrator should follow to remove his or her host from the list after correcting the problem.

The first victims of “collateral damage” were those whose mail servers were blocked through no fault of their own. This often happened when over-zealous blacklist operators added entire blocks of addresses to their lists, rather than just the offending addresses. As one group of operators argued that the lists should err on the side of caution to prevent these problems, others believed that this would put extra pressure on the open relay administrators. In one form or another, this debate continues.