|
Engineers in the Internet Engineering Task Force, in the Messaging Anti-Abuse Working Group, and elsewhere have been debating how to handle e-mail-server blocklists in an IPv6 network. Let’s take a look at the problem here.
We basically have three ways to address spam, in our goal of reducing the amount of spam in our inboxes:
1. Prevent its being sent in the first place.
2. Refuse to accept it when it’s presented for relay or delivery.
3. Discard it or put it into a “junk mail” folder at (or after) delivery.
The last is handled by what we usually think of as “spam filters”, which analyze the content and other aspects of the messages. Dealing with the first involves law enforcement, as well as adoption of best practices for legal email marketers. To implement the second, we try to do various analyses during the actual transmission of the email messages, in order to respond at the protocol level with some sort of refusal. It’s rather like standing between your postal carrier and the mailbox at your house, and telling the carrier that she may put this envelope into the box, but she should take those two catalogues and the credit-card offer right back to the post office with her.
And one can actually imagine doing that, by looking at the envelopes and applying rules such as, “If it’s pre-sorted, it’s probably junk,” and, “The more urgent it claims to be, the more likely it is to be junk.” But a better way, still, would be if we could get this to happen as soon as the junk mail entered the postal system, by having a way to say, “See that guy who’s dropping that pile of mail at the post office? He only sends junk, and when you see him coming just make him go away. Don’t even let him bring his pile in the door.”
We have that in our email systems, in what we call IP blocklists (or blacklists). These are lists of the numeric Internet addresses of email servers that we think send so much spam that we won’t even let them come to the door. When one of these servers makes an Internet connection to one of our mail servers, we don’t even start an email protocol exchange with them—we just refuse the connection. We make them go away.
Estimates vary as to what portion of attempted spam this blocks, but at least some estimates are on the order of 90%. Despite the problems with this mechanism (legitimate mail servers do find themselves on blocklists, for various reasons, and sometimes have a hard time getting the list-managers to remove them), it’s a critical one in the fight against spam, saving a great deal of time and computing resources by cutting the spam messages off much earlier in the process.
But note that it deals with IP addresses. Today, of course, that means IPv4 addresses, those things that look like 192.168.0.1, and that there are around 4 billion of. 4 billion is a large number, but, as we’ve seen, it’s notably finite and manageable. It’s reasonable to take every IP address we ever see trying to send mail, and keep it on a list, sorting the addresses into the “good” ones and the “bad” ones. It’s feasible to block Internet connections from the ones in our list that are marked “bad”.
Not so when we consider IPv6. Bumping the IP address from 32 bits to 128, bumping the 4 billion up to a billion billion billion or so—the number doesn’t matter, at that point—makes it infeasible to keep a list of bad addresses. There are enough addresses there to allow the bad guys to use a new one every time, so we’d never see repeats. There are, of course, ways we can group addresses into large blocks, and know that any address we see in one of those blocks will be bad, but even that isn’t enough to make it work.
We could switch to a “pass list”, a whitelist of known good addresses—that would still be small enough to be manageable—and refuse anything else. But that makes it very hard for an organization to deploy a new server, or for a new organization to join in.
John Levine has one approach: leave the email system on IPv4 for the foreseeable future. Even, John points out, when many other services, customer endpoints, mobile and household devices, and the like have been—have to have been—switched to IPv6, we can still run the Internet email infrastructure on IPv4 for a long time, leaving the IP blocklists with v4 addresses, and a system that we’re already managing fine with.
Of course, some day, we’ll want to completely get rid of IPv4 on the Internet, and by then we’ll need to have figured out a replacement for the IP blocklist mechanism. But John’s right that that won’t be happening for many years yet, and he makes a good case for saying that we don’t have to worry about it.
At least not until he and I have long been retired.
Sponsored byRadix
Sponsored byVerisign
Sponsored byCSC
Sponsored byDNIB.com
Sponsored byVerisign
Sponsored byIPv4.Global
Sponsored byWhoisXML API
First you block per /128
If there are more than N blocks per /64, you block the /64 (LAN)
If there are more than N blocks per /56, you block the /56 (Residential Site)
If there are more than N blocks per /48, you block the /48 (Generic Site)
If there are more than N blocks per /40, you block the /40 (Mostly a PoP)
If there are more than N blocks per /32, you block the /32 (ISP)
The big question is getting N right on each level.
BTW: Step 2 (“Refuse to accept it ...”) can also be triggered by Spam filters, which is what one is doing when doing it right: no bounces caused by the receiving server, as the origin server handles it and the if it is a real person they will nicely get a proper bounce with reasoning, otherwise it just disappears in the spam/discard folder never to be seen by the recipient, which can just save the love story of your life.
There is an open development environment that uses an event processing network platform, which with a simple app could accomplish this using any email apps API.
Driving the Live Web Kynetx changes everything.
Even if you plan to do whitelists, there’s still a huge problem with DNS caches due to the vast size of the v6 address space. I’ve been looking at different ways to publish DNSxLs in the DNS that don’t require a different query per IP. For a rough draft, see:
http://tools.ietf.org/html/draft-levine-iprangepub-01
I think IP blacklists are now a part of the problem, not the solution. It can hardly be denied that their efficiency is a thing of the past. They don’t scale well, introduce obvious SPOFs, are hard and expensive to maintain, cause you to lose control of your email blocking. Whitelists are even worse: we don’t need to steer to a world where we would have to use (or worse, pay) AOL, Hotmail or Google to be able to get our email out.
Last but not least, I’m surprised you don’t mention greylisting, which is a well established technique for spam blocking. Way more efficient than blacklists or whitelists, with a big bonus: easy to maintain, no SPOF. They may become a stale technology someday but they’re still clearly useful.
Having written what I think is the only peer-reviewed paper on Greylisting (in the CEAS conference a few years ago), I agree that it still works surprisingly well, but it’s no substitute for a well run BL.
Spamhaus users report that their BLs block 80% to 90% of incoming spam, and I see no reason to disbelieve them.
My experience differs. The main problem with spamhaus is that they can stop replying overnight to your SMTP server requests, just based on request rate, without any advance warning. That makes them a very, very, very annoying and dangerous SPOF. However I agree that Spamhaus is well run. In fact, to date it is probably the only remaining well-run DNS BL. Their 80-90% estimated block rate is probably based on a pure-spamhaus solution. Most of these blocks can be obtained more easily by greylisting or other methods, not having the SPOF problem. Their being the only well-run BL is telling about the difficulty of running such a list.
If you depend on Spamhaus (or any other external service), you should buy a susbscription. They only cut off people who don’t pay and exceed rather generous query limits.
Gandi has occasionally rate limited my WHOIS queries (I do them to check updates to abuse.net) but I don’t call them an SPOF just because they won’t provide unlimited service for free.
Until email authentication and reputation systems will work reliably, public MXes should stick to IPv4. This will prevent spammers from even asking for large IPv6 blocks, effectively segregating them into IPv4.
Thinking about it… a typical setup for IPv6 dynamic-addresses is not to have reverse-DNS for hosts at all. If they are there, they are database/automatically generated due to the impossible size of a zone-file.
In the IPv4 world, I know that many sites will not accept email without Reverse DNS pointer. Some insist on correct Forward-Confirmed RDNS (e.g. as required for SPF checks, not that SPF solves real problems with forgery, just backscatter).
I’m thinking, would it now be sensible for IPv6 MX’es to just *require* a working FcRDNS pointer (excluding typical dynamic hosts entirely), and use blacklists based upon the DNS name of the host, instead? This changes the blocking and it is easy to escalate the blocking to a higher level in the forward-DNS pointers… This changes the game to one where the spam-sender needs to keep getting new domain names pointed to their server, instead of IPs… I guess both could be part of the listing.
Combine this with some standard ‘do not accept pointers back to “ip6.arpa.”’, and
disallowing dynamic-generic reverse-DNS-pointers (maybe there should be a standard isps will can use…?)...
Just thoughts… Interesting puzzle to consider!
If spammers hop from IP to IP, as seems likely in an IPv6 world, just the queries to find out that the IPs don’t have rDNS will blow out DNS caches. Managing IPv6 mail is a difficult problem, and few people outside the large networks who’ve experimented with it appreciate just how much harder it is than IPv4.