While people may debate the death of email, there is no question that many email servers are already overloaded with spam. Current spam solutions are beginning to address the problem, but so far they all suffer from the arms race issue - as fast as we come up with new ways to fight spam, spammers are finding new ways to deliver it to us.

While the functionality of email will certainly continue, the current system must change. When the change comes, it will deliver the future of email to Microsoft.

Open standards established email, but enabled spam

One key enabler of spam is that current standards mandate an open system. Everybody can add themselves to the system with no cost or central authority. This has worked very well to encourage the rapid growth of email and its network effects, and is why nearly everybody on the planet uses email.

However, the free nature of open email neuters many blacklists - several anti-spam groups track servers that are run as “open relays” and are the source of significant spam. The spammers, however, simply hop to a new server as each server is blacklisted. It costs nothing but a little bit of time.

Closed email cannot reach critical mass

Closed systems are great for controlling access and ensuring identity. Some companies are moving to closed IM systems for business-critical communications. Recently, AT&T announced a plan to create a closed email system for its employees and business partners. They were going to create a “whitelist” of servers at their partner companies that were allowed to send them email rather than using the constantly changing “blacklist” of bad servers.

Of course AT&T cancelled the plan promptly. In additional to the logistical troubles of maintaining such a list, you can’t apply an 80/20 rule to email. 80% of useful emails may come from your major partners, but another 20% may be from outside. Any anti-spam company can tell you that you’ve got to target lower than 1 in 1000 false positives to be acceptable to customers. A new closed email that some have proposed will never have the critical mass to take on the current email system. It will always need a gateway to the outside world, and that gateway will always get hit with spam.

Halfway closed works

Fortunately, there is a way out.

After all, there’s an interesting characteristic of every spam fighting solution. None of them are as effective as they could be at detecting spam, because each is working very hard to avoid false positives.

It follows, then, that if you could remove a great majority of the “good” email from consideration, the remaining email could be subjected to much more stringent tests while still maintaining the same low false positive rate. The new version of Microsoft Outlook applies this logic - it has a spam filter built in, which includes the option “automatically deliver mail from anybody in my address book”. As soon as they figure it out, Microsoft will extend that to include more subtle gradations like “anybody I’ve ever sent mail to is very likely not to be a spammer”, among other variations.

To use a specific example, take 100 messages, 50 of which are good. Your spam filter takes out over 90% of the spam - leaving you with 4 ads for viagra. It lets through 48 of the good ones, leaving you with 2 missed emails.

If you were able to identify that 25 of those 50 messages were definitely good (your whitelist), then you’d only subject the remaining 25 to the spam filter. That leaves you with 1 missed email - or, if 2 was OK, you could double the effectiveness of the filter and only get 2 ads for Viagra.

The upshot is a direct benefit - even if your network is semi-closed, just subjecting outside emails to more stringent spam tests, it works. The more emails come from people in your closed network, the more stringent the spam tests can be and the less spam you will get from outside of it. In other words, the whitelist does not need to be perfect - it just improves the larger it gets.

Economics comes to the rescue?

So how does one create the server whitelist?

One solution is to charge people to participate in the system. Anybody who pays a fee is automatically on the whitelist. Several companies (such as the Bonded Sender program) are working to provide this solution to legitimate bulk emailers (e.g. travel specials from United Airlines, etc.). United Airlines pays a bond to the company, which they lose if they actually send any significant quantity unsolicited email. The company then provides this list to all of the anti-spam companies so they can properly distinguish bulk email from spam.

It is a brilliant economic solution, since it imposes an incremental cost only on spam. Of course you could register with the service and send spam, but then you’d forfeit the bond and have to get a new one. However, if you register with the service then send only legitimate emails, there is no incremental cost - only the fixed one-time cost of putting up the bond in the first place.

This could extend to companies in the Fortune 500 who wish to deal with each other as well (or AT&T and its suppliers), but so far nobody wants to shell out the extra cash to participate. Even the bulk emailers will soon have other free or less expensive options via Project Lumos, a similar effort put forth by a consortium of ISPs, or Cloudmark’s registered sender system.

?but economics also delivers the world to Microsoft

However, corporations are already shelling out big bucks for email - specifically for Microsoft Exchange or IBM/Lotus which between them have 75% of the corporate market.

Microsoft could just provide a stamp on each outgoing message (think public key cryptography) identifying that it came from a specific exchange server. This would be verified with Microsoft, which would provide a whitelist of valid exchange servers to every anti-spam company.

That’s all they would need to do. If somebody then used an MS Exchange server to send spam, it would get blacklisted. Unlike normal blacklisting of open relays, the spammer couldn’t just switch to another server. They’d have to buy another one from Microsoft for significant cash. This has the same effect as the bonded email program - incremental spam costs money but incremental legitimate emails do not.

40% of corporations use Microsoft servers for email. That would immediately remove 40% of all legitimate corporate email from consideration as spam, and basically double the effectiveness of anti-spam solutions overnight.

This is where the twist comes in. At the same time as the effectiveness of anti-spam solutions increases, anybody NOT using Microsoft Exchange to send emails suddenly has twice the number of their legitimate emails rejected by anti-spam solutions as their effectiveness is cranked up.

At that point, think of how valuable it is to a corporation to purchase Microsoft Exchange instead of a competing server. After all, if you purchase a competing server, you aren’t part of the “closed network” and your emails are subject to significant filtering. By purchasing MS Exchange, your outgoing emails are suddenly treated as good, and you have eliminated most of your incoming spam problem as well. For consumers, they could automatically provide a rate-limited equivalent with each version of Microsoft Outlook (e.g., only the first 500 emails each day are stamped as valid).

More importantly, Microsoft now benefits from a strong network effect. The more people join the network, the more valuable it is to be inside and the more painful it is to be outside the network, as people continue to tighten the spam engines while maintaining the same low false positives rates.

Microsoft is already thinking about solving the problem of identity by identifying servers rather than individuals. They are trying it out with a consortium of ISPs on the consumer side, where they don’t dominate.

Just don’t expect Microsoft to use the consortium approach on the corporate side, where they have the dominant share.

The coup de grace?free anti-spam solution

It’s not a perfect world for Microsoft yet, however. As it stands now, IBM/Lotus could also do the same thing. Since they have 35% market share, if they also published their server list, anti-spam companies would use it as well.

So Microsoft must do one thing first - something they are quite experienced at doing. They must incorporate the anti-spam solution free into the next version of MS Exchange.

This has several immediate implications: Microsoft will be in the market for one or more of the most effective server-side or outsourced anti-spam solutions (are you listening Cloudmark and Postini?). Anybody not acquired had better find another business, since MS Exchange customers will get the service for free. Finally, Microsoft’s share of the corporate email market will jump significantly - perhaps IBM’s as well, if they adopt a similar strategy fast enough.

So the world will once again be reasonably safe from spam overload, but the price is yet another area of Microsoft domination. Either alone or with IBM, within 5 years Microsoft will end up owning most of the corporate email traffic in the world.