Learn about Linux
Download Linux
Get Linux helpSpecial Offers
Feature
Email Sender ID: The hype and the reality
Share
Print
Comments
In 1998, Jim Miller sent an email to Paul Vixie which outlined a basic plan for rejecting forged email. In 2002, Vixie published his Repudiating Mail-From" protocol based on Miller's work. Mail-From in turn inspired two other techniques, "Reverse MX" by Hadmut Danisch and "Designated Mailer Protocol" by Gordon Fecyk.
In 2003, Meng Weng Wong picked what he felt were the best features of Reverse MX and DMP and came up with SPF. To date, SPF has become far and away the most popular of the DNS-based anti-spam efforts. But the morphing has continued. Microsoft has wrapped its own Caller ID proposal around SPF and now calls it Sender ID.
How it works
SPF/Sender-ID requires changes to DNS and MTAs in order to work. The changes to DNS involve the addition of new records which identify machines authorized to send mail for a specific domain. MTAs (like Sendmail or Postfix) would add new functionality to verify that the email has not been forged; that the address of the person who sent the mail is in fact authorized to send mail for that domain.
Of course, it's not nearly so simple as I describe. And some issues are not yet well settled, including which bit of information should be used to identify the sender. The TCP/IP protocol upon which the Internet is based contains the IP address of the machine attempting the mail transfer. That's called the envelope sender address. The email itself also includes a header record indicating who the email is from. It's called the header sender address. Your email client shows you the header sender address when it displays the "From" line of the email.
Spammers typically forge the header sender address. That's why the note purportedly from your Aunt Nadine turns out to be about enlarging your manhood instead of the family reunion. There has been much debate along the way over which of the two -- the envelope sender address or the header sender address -- to use for verification. Prior to Microsoft's sandwiching SPF into Sender ID, the choice was to use the envelope header. Now it's the email header.
There has also been some quibbling over Microsoft's choice of XML for the new DNS records. Some feel it is akin to HTML email, an aberration that has done much to spread viral infections to the unwary. A darker sin may be that XML is driving the "unspecified patent issue" that is also of much concern.
Technically, many of the competing proposals show real promise in putting a dent in the volume of spam. But economics, IP concerns, and not-quite-open standards all threaten to prevent them from ever fulfilling that promise.
The economics of spam
Hadmut Danisch makes a pertinent observation in his paper on Reverse MX:
As has been recently illustrated in the initial session of the IRTF's Anti Spam Research Group (ASRG) on the 56th IETF meeting, sending spam is a business with significant revenues.But a much bigger business is selling anti-spam software. This is a billion dollar market, and it is rapidly growing. Any simple and effective solution against spam would defeat revenues and drive several companies into bankrupt, would make consultants jobless.
Therefore, spam is essential for the anti-spam business. If there is no spam, then no Anti-Spam software can be sold, similar to the anti-virus business. There are extremely strong efforts to keep this market growing. Viruses, Worms, and now spam are just perfect to keep this market alive: It is not sufficient to just buy a software. Databases need to be updated continuously, thus making the cash flow continuously. Have a single, simple, and permanent solution to the problem and - boom - this billion dollar market is dead. That's one of the reasons why people are expected to live with spam. They have to live with it to make them buy anti-spam software. Content filters are perfect products to keep this market alive.
In this day and age -- in the United States, at least -- commercial interests have great influence with both the legislative and the executive branches of government. To think that the anti-spam industry will sit quietly by and allow their cash cow to be slaughtered without a fight is self-deluding. Neither are the courts necessarily a hostile place for spam artists. A major spammer -- reportedly the second largest in existence -- recently settled a case brought by New York state attorney general Eliot Spitzer for what amounted to a pittance -- $50,000.00 -- compared to the $20,000,000.00 fine that had been threatened.
As reported in USA Today:
Spitzer spokesman Brad Maione said neither Scott Richter nor OptInRealBig.com admitted any wrongdoing in the settlement.When announcing his suit, Spitzer said special Hotmail e-mail accounts set up by his investigators found thousands of e-mails in May and June 2003 that carried bogus "from" and "subject" lines, often indicating that the messages were part of ongoing conversations instead of being unsolicited commercial come-ons.
The spammer's lawyer was quoted in the same article as saying, "The fact (that) the attorney general settled for $50,000 while initially talking about $20 million in damages 'speaks for itself...'"
Intellectual Property issues
IP figures in this story in two ways.
First, Microsoft's licensing terms exclude free software from being able to use the proposed standard now called Sender ID, as Richard Stallman's recent post to an IETF mailing list made clear. Stallman wrote:
Microsoft's Sender-ID license is directly incompatible with free software regardless of which free software license is used. Free software means users are free to run it, study and modify the source, and to redistribute it with or without changes. Free to do so means there is no requirement to ask or tell anyone that you are doing so.The Microsoft license for Sender-ID directly forbids release of software with all these freedoms, so it is impossible for any program to be free software under Microsoft's regime.
Stallman wasn't the only one unhappy with Microsoft's involvement, as Larry Seltzer recently noted at eWeek:
Nothing got members of the working group more upset and caused the chair to threaten banishment more than complaints about Microsoft's involvement in the drafting of the specification. Midway through the process, Microsoft and Meng Wong negotiated a convergence of SPF with Microsoft's Caller ID for E-mail specification.While many participants objected to working with Microsoft simply on principle, the biggest objection had to do with intellectual property rights. Microsoft's contributions to Sender ID are based on their Caller ID for E-mail specification, and they make unspecified patent claims on this technology.
The second major IP problem with Microsoft's involvement comes in the form of a claim of infringement by Failsafe Designs on both the name SenderID and the methods used to verify the sender in Microsoft's "Caller-ID." As reported August 11, 2004, by InternetNews, "F. Scott Deaver, owner of Failsafe Designs, says Microsoft is guilty of the 'outright theft' of his product name and intellectual property (IP), and will seek legal and financial redress from the Redmond, Wash., software giant and anyone else that uses his technology that verifies e-mail is coming from the domain it claims." I've also heard -- off-the-record -- that the SenderID project has been told to expect legal trouble because of claims of infringement.
Although I have not been able to verify this with Failsafe Designs, I've been told by Meng Wong that if SenderID stalls or dies because of these IP issues, there is no reason why SPF (or "Classic SPF" as he now calls it) can not go forward on its own. As a matter of fact, AOL and a number of other firms have already implemented SPF. The claims against Microsoft are yet another reminder that IP law -- especially in regard to software patents -- is a spectre that hangs over all software development, not just free software.
Conclusions
Technical methods of verifying sender identification are going to go forward. Exactly which ones make it and which don't are the only real questions. But the non-technical barriers make it seem unlikely to this writer at least that they will have much of a lasting impact, given today's commercial environment.
Paul Vixie, who wrote the "Repudiating Mail-From" paper in 2002, noted in a recent online post attached to a CircleID.com interview with Meng Wong about SPF that he has low expectations for success. He said:
Finally, an observation. After co-founding MAPS and turning that crank for a number of years, I know that spam is "like a drug" in that people will go to almost any lengths, no matter how absurd, to send more of it. No "designated sender scheme" will ever be able to cut down the amount of spam that's sent, or received. All it can do is help domain holders avoid the brand dilution of having their domain name forged by spammers. This is a valuable contribution, but we must make it clear that none of these schemes will stop or even slow spam, and that their benefits accrue to domain holders, not to spam recipients.<p
Comments
on Email Sender ID: The hype and the realityNote: Comments are owned by the poster. We are not responsible for their content.
Re:Notrhing like getting directly to the point.
Posted by: Anonymous Coward on August 27, 2004 04:24 AMRe:Notrhing like getting directly to the point.
Posted by: Anonymous Coward on September 08, 2004 04:36 PMWhat I find funny is that when millions of people download mp3s they (the industry) can track down the individuals and cease and desist them to a pulp.. when 50 or so jack-asses spam the universe and your mom, nobody seems to be able to stop them. Furthermore, people have been screaming blue murder about the lack of security features in Windows and little happens, then, when worms are rampant, spam is no longer limited to 50 individuals but lives in a worm vector all of it's own, the industry wants to shove some expensive, safe, ridiculous solution down your throat for a problem THEY created through either their own nonchalance or ineptitude.
WHAT EVER HAPPENED TO PRODUCT LIABILITY?
Yeah, closed source companies always moan about how they can offer guarantees that GPL software cannot... GUARANTEE THIS lamos!!
Yeah, you get my drift... the whole worm issue, spam and all included are in my opinion the most coercive form of marketing ever used. bill has a license from the devil and is going to shove trusted crap and dicko-encrypto mail right down YOUR throat and charge you for it up the gazoo...
Well mr Bill "hell's" Gates, read my lips:
All you bases are now ours!
Thats right, server's are predominatly running OSS or *nix flavours, routers, switches, pabx', embedded devices, portables, and laptops have OSS as fastest growing development and the desktop is under fierce attack with Linux taking the point deep into enem(a)y territory! Why heck even cell phones and carrier grade systems are being ported to some form or other of OSS.
If anything, this whole sender-id initiative will bolster the will of OSS org's and devvers and drive a mass consolidation attempt the likes of which will rival the kernel development team's rally around Linus. That's right, you hear me loud and clear, the solution to mail will come from OSS and it's devver will hit the hall of fame in the same manners as that penguin lover whose code comments make us piss our pants laughing....
Were it described we would see that the main reason for sender-ID is that spammers abuse other systems to send their spam and use false sender information when they do. The reason they abuse those other systems is to hide their identity. So there's two parts to the problem, just from this simple analysis: the false names and the abuse of the other systems. When do you ever see any analysis on what can be done if the problem is approached from the abuse standpoint rather than from the false sender standpoint? Never - and yet combating the spam as abuse is far simpler than sender-ID, blocklists, or filters. The abuse is widespread, meaning that the abuse can be fought at many locations. The abuse is still mostly very simple, meaning that simple counter-methods can be effective. Such an approach already works, and is devastating to the spammers. That's why they have available a tool called "Honeypot Hunter" - to detect and avoid fakes. If they look for honeypots how they look can be found and then the honeypots can be made to again deceive the lookers.
In 2000 I stopped spam to millions with a simple tool on an even-then obsolescent computer. I did it on a mail server, meaning I had to distinguish spam form valid email. That, of course, is decidedly easy: simply treat as spam all the email you'd never accept in the first place if you rejected illicit attempts to relay through the server. Accept them but don't deliver them. That's simple. If you set up a non-server for the same technique then there's no thought at all needed: everything you receive is illicit, none need be delivered. There is one exception: the spammers precede the spam with test messages to determine if the system will relay. Those you deliver. It's amazingly simple.
Nowadays open relay abuse is finally fading, although it still happens from China, Korea and Taiwan (with the addresses also in those countries: they relay through open vulnerable systems in the US and probably elsewhere.) Open proxy abuse is still fairly common, and that can also be easily combated, by fake open proxy systems. Here there's never any discrimination needed: if it's email coming through your (fake) open proxy it's spam. Again, too, there's a need to allow the spammers to have success with their tests.
For zombie abuse there have been fake zombies, but those could get harder. If the ISPs are finally going to develop and follow some best practices I suggest they start with a dual-goal first step: detect abuse in their own space and report it back to the ISP that is in charge of the IP address of the apparent source. These ISPs, also following the best practices, should detect the source of the incoming traffic (if any) to the IP address identified as the source of abuse and report that to the ISP for that IP address. If the abuse originates at the local IP Address then they know the source and should take appropriate action. The two goals are (1) detect abuse that can be seen in their own space and (2) become responsive and proficient in tracing abuse to its apparent source.
There have been successful attacks made on the spammers by these methods. Michael Tokarev in Moscow caused spammer Alan Ralsky great inconvenience using a simple open relay honeypot. Ron Guilmette, in California, had great success using a small network of open proxy honeypots. He got over 100 spammer accounts closed in under 3 months, acting as an individual. The method has great power - but if groups such as the IETF continue to insist on complex long-term solutions the easier solutions already at hand will be ignored. that's a tragic mistake.
Pay more attention to Hadmut Danisch. Not only is spam a cash cow for the anti-spam software companies there's also the basic fact of human nature that seems to be intensified in computer people: they want to devise complex solutions. Let one think of an approach to spam and his attention shifts from dealing with the spam problem to establishing his method as the one used. I stopped relay spam with nothing more than command files on an ancient Vaxstation. That's partly because stopping spam wasn't my original goal. What I started out to do was to secure the mail server so it wouldn't relay. the traditional approach of rejecting illicit messages when they were coming in wasn't easily available so I chose an equivalent approach: stop them on the way out. From that I learned the obvious, that spammers test to find open relays, and then expanded what I was doing to make it intentionally deceive spammers as to the nature of the system so that it would trap spam.
There are objections you can raise to that but they aren't definitive objections. What I did worked and that is the point. While I read of others struggling to keep their filters good enough to keep up with the spammer creativity in avoiding filters I could simply look at the spam I was trapping and see the new spammer anti-filter efforts. Against me they were worthless, because I wasn't trying to filter spam at the destination. By acting mid-path my task was made extremely simple: if it came, it was spam. (Of course I had to first properly handle the valid email - but none of that was relay mail from unauthorized sources so dealing with it was easy. If I had created a system with no other purpose then there'd have been no valid email, and that's what Michael Tokarev (and others) did.
Sender-ID isn't needed to end spam. What is needed is an effort that is designed to end spam now, using what can be done now. That is simplest at intermediate steps in the spam delivery path. Look at it this way: is it really necessary for the spammers to always be correct when they conclude a system is abusable? That's what makes their lives easy. If the internet were lightly salted (under 1% of systems) with fake abusable systems the spammers would face a far more difficult task. That approach can be started today, and each system added has a very high probability of having some useful effect in ending spam.
"Minas Beede," posting "anonymously."
It's been widely reported with about 80% of spam uses forged headers. Several people and groups have published their own analysis, all with similar results. The majority of spam uses faked sender and header information.
When do you ever see any analysis on what can be done if the problem is approached from the abuse standpoint rather than from the false sender standpoint? Never [...]
Spamhaus has, for quite some time, done analysis of how much spam originates from a small group of spammers abusing the system.
Recently, AOL, Earthlink, Microsoft and other have filed many lawsuits against these abusive individuals. The FCC (with the DMA's money funding much of the work) is in the process of bringing criminal charges against spammers. The law is a slow process, but analysis has been done and work is in process to stop the abuse at the source.
If the ISPs are finally going to develop and follow some best practices I suggest they start with a dual-goal first step: detect abuse in their own space and report it back to the ISP that is in charge of the IP address of the apparent source.
Again, things are beginning to change here too. Until recently, rivals AOL, Microsoft, Earthlink and others did not cooperate. Only months ago, they finally published a concensus of best practices. Recently, even Comcast has started contacting customes whose machines have been turned into zombies.
A lot of work is being done on a lot of fronts to combat the spam problem. Maybe it will eventually help, maybe not. But just because spam has only increased (perhaps as fast as it would have in the absence of any efforts, perhaps not)... there is no justification to rant that nothing is being done other than sender authentication. Many people, organizations and companies are working on many approaches.
bear out that something like 95 percent of everyone does use their software, and that's what almost all of the spam comes from.
Big spam shops are big business, they all use MS OSs and tools. MS itself is big on email marketing. It has spammed it's customers more than once, and a few months ago said it was turning all the msn and hotmail email addresses over to
"reponsible" marketing "partners". Meaning spammers.
MS is fully spam friendly. Some of us have had a hunch that MS in some form or another is actually teamed with some mass mailing cartels in order to drive the need for a MS patented technology to replace everything that isn't Exchange.
Paranoid? The question is, are you paranoid enough?
Remember, all this friendly talk about MS not charging for a license to use software that
contains patents that they are not disclosing,
and promising to play nice, is the same MS we keep catching time and time again with their little
fingers in the cookie jar, (as it were).
MS respects no law unless forced.
Then they wiggle around it.
They always have, they always will.
Their past speaks for itself.
Didn't MS go after slashdot for some reader quotes taken from<nobr> <wbr></nobr>/PUBLIC/ MS documents, over copyright violation?
Think carefully before you sign.
They are *NOT* your friends, they eat their
friends.
"that the person who sent the mail is in fact authorized to send mail for that domain."
what he should have said is:
"that the *MTA* which sent the email is in fact authorized to send mail for that domain."
Thanks. Before I correct that, though, let me ask to be sure. Does it matter whether the envelope or the header from address is being used?
Joe Barr
Anyways, what about Thunderbird's free junk mail filter? I get about 20 junk mails a day, and don't have to sort through them to get to the good stuff. It just works!!!
SenderID, or whatever else it is called, like all dns based "anti-spam systems" is more a threat to the individual user than to the big spammer.
Then set up Your SPF record to point at your IP and your ISP's SPF record.
Also set up your SMTP server to require you to log into it first. This closes the open host.
What's so hard about all of that? (Other than the learning curve, But you're trying to set up a mail server!)
Lucky you. I get 2000 a day. Mozilla does a fantastic job of knocking out about 98% of them but that still leaves 50 or so to do manually. Each day! I too would like to be able to rum my own SMPT server so a method that allows for this would be great. Perhaps rather than blocking mail that isn't trusted it could merely mark it as not trusted. This would allow the recipient client to junk the untrusted e-mails unless they happen to be sent from a mailserver they personally trust (ie yours).
You should also ensure that you're properly masquerading the envelope and using your providers MX for your relay if you're on a dynamic IP. Many (if not all) of the ISP's will not accept direct mail from hosts with dynamic ip addresses (I don't and haven't for a long, long time).
email is dead as a serious communication medium
Posted by: Anonymous Coward on August 27, 2004 03:41 PMBig companies use a variety of communication protocols internally and with major suppliers. Some may look like email but are not.
Spam killed off corporate email some time ago.
Re:email is dead as a serious communication medium
Posted by: Anonymous Coward on August 27, 2004 08:44 PMNot sure what you mean about looking like email but not being email - if it arrives in my mailbox, it sure is email.
Re:email is dead as a serious communication medium
Posted by: Anonymous Coward on August 31, 2004 01:45 AMSo, no, corporate email is alive and well.
Re:email is dead as a serious communication medium
Posted by: Anonymous Coward on September 08, 2004 11:15 AM1) trademark their e-mail sending domain (say postman.isp.com)
2) Grant a license to their customers to use that trademark for the limitted purpose of sending e-mail
3) Sue anyone who forges the e-mail address for attempting dilution
4) Profit?
I am not suggesting the above would work, but does anyone know of a case in which it was trierd as a claim?
An insightful, concise, well-written article
Posted by: Anonymous Coward on August 27, 2004 11:41 PMDNS could be used to hold MTA ID signatures/keys, though private signature key-lists could be maintained by any cooperating group of MTAs. Heck, we used to do that much with UUCP maps, in some sense.
This is off-the-shelf technology. No patent-mongers need apply.
Thanks
Frank
Notrhing like getting directly to the point.
Posted by: ccchips on August 27, 2004 02:31 AMThe funny thing is that my gechnical-support colleagues bob their heads up and down when I say it, but nobody seems to have any sense of how to address it.
Maybe that's because everyone in the US has "the gets" these days, and are waiting for when they "get theirs."
When I saw my corporate bosses going in for tools that required subscriptions and periodic updates to control spam, I knew what was going on right away.
If there's a Devil, and he buys souls, he's got to be forking out an awful lot of money these days....
#