|
A message on Dave Farber’s Interesting People list complained that Comcast was blocking mail forwarded by DynDNS, a popular provider of DNS and related services for small-scale users.
... Wholesale blocking of all mail intended for customers from a particular intermediate distributor, merely because they route it through an external service that adds value.
In response, I opined:
Actually, they’re blocking it because a lot of it is spam. This is a problem that every mail forwarder and every mail system encounters; the only unusual thing here is that DynDNS is whining about it. It’s yet another way that spammers have broken the mail for the rest of us.
Traditionally, like 10 or 20 years ago, systems that forwarded mail, passed along everything that showed up for the forwarded address. But today, when upwards of 90% of all mail is spam, if you do that, most of what you’re forwarding is spam. I can tell you from experience, since I run systems on both ends of forwards (I’m [email protected] among other places) that no matter how clearly you explain to people that the forwarded spam is stuff they’ve asked for, they will still complain about it, and the automated systems that large ISPs have to use to maintain their spam filters will correctly mark the forwarding IP as a spam source.
The usual next suggestion is that the forwarder should put some sort of flag into the mail to mark it as forwarded so don’t blame us. That doesn’t work either since any mark your forwarder can make, spammers can also make. Manually maintained lists of known honest forwarders simply aren’t practical at the scale of an ISP like Comcast. (For that matter, it’s not very practical for my tiny 1000 user mail system, either.) I’ve adjusted my system so that if a user requests mail be forwarded elsewhere, it runs the mail through SpamAssassin and only forwards stuff that isn’t marked as spam. Yeah, you might lose some real mail that way. It’s another reason that spam stinks.
The real way out of this is to realize that forwarding is a lot less useful than it used to be. These days, every popular Mail User Agent (MUA), including the big web mail systems, can easily be set up to collect mail from multiple inboxes so rather than doing forwards with SMTP, you collect mail with POP or IMAP, and as a free bonus, it’s pre-sorted by inbox.
Sponsored byRadix
Sponsored byVerisign
Sponsored byIPv4.Global
Sponsored byCSC
Sponsored byVerisign
Sponsored byDNIB.com
Sponsored byWhoisXML API
I think the problem here is that Comcast is unresponsive to the needs of forwarding services. By contrast, AOL has a policy in place for forwarders, documented on their website. It specifies a set of headers to put in the forwarded mail, which as you mention, could be forged by spammers, but AOL also requires that you notify them, so they know whether your IP addresses should be trusted to send these headers or not. If Comcast had a similar policy, or any policy, regarding forwarders, services like DynDNS wouldn’t have to resort to asking their customers help to get a response from Comcast.
AOL doesn’t do anything special with forwarded mail, and if you forward a lot of spam to AOL, you have the same problems as if you forward it to other large ISPs. (Don’t argue, I know the postmaster.)
Really, the solutions are to have good enough spam filters that you don’t have to worry about losing stuff when you don’t forward it, and to encourage people to pick up mail directly so forwarding isn’t needed.
We agree that spammers have significantly affected the way mail functions, as you say “broken the mail for the rest of us.” Our goal with our MailHop system is to offer our customers the ability to manage their own email settings. As you do, we run all of our mail through SpamAssassin. This recent email to customers was designed to communicate a change in policy we have made to ensure continued and reliable delivery to Comcast users.
We believe the power to manage any and all settings or preferences should be that of our customers, and that offering is a high priority of ours. However, with delivery adversely affected by other provider’s policies, we took corrective action to ensure deliverability, which is our first priority.
To the best of my knowledge, AOL does in fact have special whitelists, monitoring, and other tools designed to work with forwarding services, such as MailHop. If not, I’d like to know who we’ve been talking to at AOL for the past 2 years to accomplish the level of deliverability we have with them.
Back to the original point - forwarding is something people understand and can use to their own benefit. Its an analogue to the snail mail system; when you move, you forward your mail. People get that. People generally don’t have multiple mailboxes that they have to go check, so POP’ing or IMAP’ing 3 or 4 boxes isn’t something people get, which is why we don’t offer the service. We try to make things easy for our customers - one of those things is e-mail forwarding.
We just went through this with comcast ourselves after they were deferring mail from our mail forwarders to comcast users. We finally got in contact with some postmasters and their abuse department and we went ahead and set up a feedback loop with them.
After personally inspecting upwards of 200+ abuse messages reported back “as spam” through this feedback loop by Comcast users I have found that MOST OF THEM are NOT spam, they are:
- newsletters that the end user themselves have opted into
- mailings from businesses that they have an obvious pre-existing business relationship with (Amazon, overstock, etc)
- Comcast’s own customer newsletter
We all know spam when we see it: forged headers, deceptive subjects, promoting some nebulous garbage product and I was hard pressed to find much of this via the messages reported back via the feedback loop. Why? Maybe because we (the forwarder) are actually serving some (useful) function after all by standing in front of the destination mailbox and applying our own spam filters, greylisting and optional SPF validation.
I spoke of this instance with my systems group and this seemed old news to them “yeah, we find that with every feedback loop we have set up, it’s all real mail getting reported back to us, not actual spam”.
There’s no real answer for it other than maybe affording some extra latitude for known whitehat mail forwarders in scoring databases and third party reputation systems or pushing back functionality to the end users to override or whitelist their personal mail forwarding services.
Picking up mail directly is a good option, but it is not always the best solution.
Backup MXes are an instance of forwarding. We currently have reliable networks, but we can never be sure they will always work that way. Hence, we’re better be prepared to have functioning backup MXes, including the ability to easily set them up on a network we don’t control directly (e.g. third party.)
As a second example, one may just want to forward to a site because of the picking up functionality they offer, e.g. using mobile phones or whatever.
IMHO, the difficulties we experiment originate from the lack of control that users have on the mechanism. The mailbox is the mailbox, not the forwarding mechanism. So, why should forwarded spam be reported there? Possibly, because there is no other place to report it. There is no previous mailbox button. No tool builds a tree of the addresses that eventually drop mail at a given mailbox. No smooth way to manage forwarding.
Systems like MailHop are nice and appealing. However, the mechanisms they use are not standardized. Therefore it is not possible to surf smoothly along a forwarding chain. Most of times, only the forwarder knows the next hop of a forwarded address, so that forwarding chains look much more like singly linked lists than doubly linked ones. (Reading unreliable Received headers just supports debugging.) If we want to use forwarding, we need more data for managing it.
This comment is not the place to give a full description of an example of how forwarding could be standardized in such a way that enough data is collected at each node. However, I guess readers can easily imagine that it is feasible. In addition, consider newsletters and mailing lists that are further instances of forwarding. Collecting the required data implies, for example, that there are
two
systems maintaining a record that says “John Doe subscribed to XYZ newsletter on <date>”. Isn’t that required by current opt-{in/out} legislation? How come we do without?