I have been having no end of problems with my outbound email messages sent by my Drupal sites since I switched from shared hosting to VPS hosting. A typical shared host offers a well configured SMTP server with documentation about how to configure the MX and SPF records. On the other hand, a VPS requires the administrator to install and configure the mail transfer agent, which can be surprisingly complex. Even more difficult is the fact that the VPS provider can’t really provide a one-size-fits-all set of instructions for configuring the MX and SPF records, since there are many more variables controlled by the VPS administrator. So, suffice to say that email configuration is probably the most dreaded task of most VPS administrators. In fact, many prefer to avoid the pain and suffering altogether and use a third party email provider. However, in my case, since one of my sites sends a lot of forum activity notification emails, I prefer to save on the cost of a third party email provider and do it myself. This has turned out to be a ridiculously complex trial and error process, but I think I finally found the missing key.

I won’t even begin to get into the mail transfer agent (MTA) configuration, which is Postfix in my case. In reality, sending out an email isn’t the most difficult part of the process. I was able to configure Postfix and send out emails by following any number of guides online. However, getting the popular email service providers to actually accept the message is much more complex. In particular, I was having trouble getting Gmail to receive my messages and not classify them as spam.

Debugging spam classification in Gmail – What NOT to do

The first tip when debugging problems with email delivery to Gmail is to avoid the temptation to mark messages as Not Spam. My first reaction when testing my setup and finding emails from my site classified as spam in Gmail was to simply mark the message as Not Spam, which apparently solves the problem as subsequent messages will often pass the spam filter. However, this will not make Gmail accept messages from your server sent to other users’ Gmail accounts. It is highly unlikely that most of your users, or far worse, potential users, will be motivated enough to look for your messages in the Spam folder. New signups that are expecting their account activation email to arrive in their inbox will probably check only the inbox and will most likely wait a maximum of 30 seconds for it to arrive. Even more difficult, if you mark a message from your server as Not Spam on Gmail, you will not be able to accurately evaluate the effect of any future configuration changes on Gmail spam classification. So if your first attempts at getting messages to pass the Gmail spam filter fail, just leave them in the Spam folder and keep trying. When you finally get the configuration right, your messages will arrive in the inbox irrespective of your prior attempts that ended up as spam.

Drupal 6 and the Return-Path header problem

The second tip is specific to Drupal 6. For some reason, Drupal 6 has an irritating and unresolved bug that causes it to send out messages with a Return-Path address in the header is not the same as the From address. Apparently, the PHP mail() function sets the Return-Path based on the Linux user account name of the web server. For example, in my case if messages were sent with a From address of info@example.com, the Return-Path in the header was nginx@members.linode.com. This is a sure recipe for classification as spam on Gmail and many other email providers as well. Fortunately, there is a dead-simple Drupal 6 module that fixes this dreadful bug. Many thanks to the Return-Path module, which sets the Return-Path to the same address as the Drupal site configured default address.

MX and SPF records

The most difficult part of the configuration has to do with setting the MX and SPF records. In my case, I send email from several domains that are hosted on the same VPS, and I don’t necessarily want curious users to look through the email headers and see that emails from info@lose-weight.com are sent from the hamburgers.com server. (Please not that these examples are purely fictitious. I hate weight loss products and I also hate hamburgers.) However, in order to not be classified as spam, it is important that there be an MX record that specifies the server name that sends the mail for that domain. So one option would be to buy a toplevel domain with a neutral name, such as acme-email-server.com, and then set the VPS hostname and the MX records to that same domain name. That way, messages from lose-weight.com and hamburgers.com will both have acme-email-server.com as the originating mail server in the headers, which shouldn’t offend anybody. Another cheaper option is to use the standard Linode host name. In my case, the default Linode Reverse DNS (rDNS) hostname under the “Remote Access” tab of the Linode settings is something like li222-111.members.linode.com. So under the “Mail server” field of my Linode MX record settings, I put li222-111.members.linode.com. Likewise, I made sure the hostname of my CentOS VPS was also set to li222-111. This is a nice generic name that I don’t mind users seeing in the message headers.

Finally, I tried to set the SPF record according to the tutorials and documentation. I found the SPF Wizard to be helpful in walking me through the different options available. I finally decided on the following: v=spf1 mx a ip4:44.33.22.111 a:li222-111.members.linode.com ~all Basically, this record states that: mx = servers for which an MX record exists on this domain are allowed to send email for this domain. a = the current IP address of this domain is allowed to send email for this domain. ip4:44.33.22.111 = this IP (version 4) address is allowed to send email for this domain. a:li222-111.members.linode.com = this server is allowed to send email for this domain. ~all = if receiving mail servers receive a message from this domain that doesn’t match this record, they should accept it but mark it as suspicious.

Some of these clauses are probably redundant, but I wanted to make sure to cover all the options. Strangely, however, even with this SPF record in place, messages continued be flagged as spam by Gmail. Finally I carefully examined the headers of the messages after they arrived in my Gmail spam folder, and discovered something interesting: Received-SPF: softfail (google.com: domain of transitioning info@example.com does not designate 124d:3dd2::ff32:9eff:fce2:9cd6 as permitted sender) client-ip=124d:3dd2::ff32:9eff:fce2:9cd6; And there was the problem. Unlike my previous VPS provider, my Linode is communicating with Gmail over IPv6. So, I also needed to add the IPv6 address of my server to the SPF record as a designated IP address that can send email for this domain: v=spf1 mx a ip4:44.33.22.111 ip6:124d:3dd2::ff32:9eff:fce2:9cd6 a:li222-111.members.linode.com ~all As soon as this change was propagated out the Linode’s DNS servers, messages magically started to arrive in my Gmail inbox and the headers now said: Received-SPF: pass (google.com: domain of info@example.com designates 124d:3dd2::ff32:9eff:fce2:9cd6 as permitted sender) client-ip=124d:3dd2::ff32:9eff:fce2:9cd6;

“Pass”. The most beautiful 4-letter word in existence. Messages are now flowing smoothly, users are happy again, and so am I.