That's Not SPAM that's HAM!!!

I learned a new word the other day... We all know that e-mail that is not relevant is call SPAM, but what do you call valid authentic e-mail... that's right HAM.  Unfortunately, I learned this new little tidbit while extensively researching why our e-mails from CCV were not getting into the e-mail boxes at the valley's largest ISP Cox Communications.  During this long journey I found out the truth that we in IT spend dis-apportioned amount of time worrying about SPAM defense (getting SPAM out of our inboxes) and little to no time of SPAM offense (keeping our e-mails from being classified as SPAM).

Until a few weeks ago I knew very little of the process and procedures we should be taking to make sure our e-mails make it to their intended inboxes.  Below I try to boil down the key points of what I learned in hopes it can help you jump start your SPAM offense.  I'll try to keep it brief and concise.

• To determine if an e-mail is SPAM most ISPs use a SpamAssassin type point system.  Hundreds of characteristics of each e-mail are reviewed and points are assigned to characteristics that would be similar typical of SPAM.  Get 5 points and your sitting in the SPAM folder.  Actually, it's a little more complex than that.  If you're interested read more.
  
• Make sure you are not on a black-list.  You can do a free check at MX Toolbox, or pay a small amount for a service like BlackListedIP.  Turns out we were not on any.
  
• Make sure your reverse DNS (rDNS) for the IP address of your e-mail server returns the same domain as the mail that it is sending.  For example the rDNS entry for our server 67.132.248.66 better return with a domain of ccvonline.com.  This can easily be checked at the following website (type in your mail server's IP in the Reverse DNS box).  Don't assume that it's right, check it now... I'll wait.  If you assumed it was correct you might be surprised... Ours returned quest.net as Qwest is the provider of out T1.  Few calls to Qwest (joy...) and we had that fixed.
  
• Make sure you have an Sender Policy Framework (SPF) record.  I was under the impression that most people weren't using SPF in SPAM classification... I was wrong... An SPF record is a coded message you put in your DNS server that tells email recipients what IP addresses are allowed to send mail from your domain.  Creating the SPF syntax is a breeze. 
  
  Step 1: Use wizard here http://old.openspf.org/wizard.html to create the correct syntax.  Ours for example is:
  
  v=spf1 ip4:67.132.248.66 ~all
  
  Which basically says that all mail from the ccvonline.com domain should come from the IP 67.132.248.66
  
  Step 2: Insert your SPF record into the TXT field of your DNS server.  This might be tricky depending on how your hosting your DNS.  Many ISPs don't support user defined TXT fields.  If this is the case simply move your DNS hosting to DynDNS.  It's simple to add it at DynDNS plus you'll get lots of other DNS tricks with their service.
  
• Get on a whitelist.  Most whitelists aren't worth much as they are not well used by ISPs.  There is one though that seems well supported.  The whitelist is provided by Habeas.  While getting on this is is very expensive for commerical use, they do offer a non-profit price of $200 if your send less than 50K e-mails a month.  Being on this list supposedly gives you four bonus SpamAssassin points (basically you start with -4 points).  Habeas also offers other services like a full audit of our e-mail infrastructure and an e-mail monitoring package.  The former feature allows you to add 400 of their e-mail addresses to your e-mails to check delivery at every major ISP.  Because of the severity of our problem I signed up for the works for one year.  I hope to be able to share more about what I learn in the future.  Sharing makes me feel a little better about spending the money.
  
  There are other services like Habeas (listed below).  Each are similar in features and price.  Habeas in my research seems to be the most respected and highest quality (even one of the competitors admitted the quality was better...)   Also, in researching I found that Habeas' white-list is used by more ISPs.
  
  ReturnPath
  Pivotalveracity
  
• If you're sending out a large e-mail check the SPAM score of the content.  Depending on the words or format your e-mail might be more likely to be considered SPAM.  You can do this one of two ways.
  
  Option 1: Use an on-line service like http://spamcheck.sitesell.com/. 
  
  Option 2: Download SpamAssassin for Windows and run the SPAM check from the command-line.  You can download the WIndows binary here.
  
• Throttle back your SMTP server.  Out of the box Exchange allows up to 100 concurrent connections to a single domain.  Many ISPs start dropping e-mails if they get too many from a single sender at once.  We throttled back ours to 10 concurrent connections.

 

After applying several of these tips (and getting on Habeas' whitelist) our e-mail woes are gone.  We still have to go through the Habeas audit so I'm sure I'll be learning and posting more.  If you want some more information right away check-out the MAAWG (Messaging Anti-Abuse Working Group) Sender BCP document.

Comments

[Austin Spooner]

Jon- thanks for sharing all the info you learned during your research

[Matthias Leisi]

Disclosure: I'm involved with the project mentioned below.

There is a non-profit whitelist at http://www.dnswl.org/ -- it is included in SpamAssassin since June 2007 and being used by a good number of different users (eg some ISPs, universities, but also some anti-spam systems for "calibration" of their data).

-- Matthias