Dec 9: Email authentication

This blog post is part of the FastMail 2014 Advent Calendar.

The previous post on 8th December was about our rich text email editor. The following post on 10th December is the second security post, on availability.

Technical level: medium.

What is it?

Email authentication is a collection of techniques to verify that an email is legitimately from the sender it claims to be from.

History

Back when email was first designed the world was a much more trusting place. The early internet consisted mainly of government agencies and educational institutions, and the notion that anyone would forge sender details in an email wasn’t considered.
Although standards have moved on since then, compatibility is always a concern when implementing new standards, as a result, email standards do not have sender authentication at their core.

In today’s internet, spam and phishing emails are or course a big problem, email authentication has become something which needs to be addressed, and a number of techniques have been developed to help achieve this. Here is a brief summary of some of the popular ones, and how FastMail uses each one.

Real-time Blackhole List (RBL)

RBL, also known as a DNS-Based Blackhole list check is a method where inbound IP addresses are checked against a list of known bad servers. This gives an early indication of which mail is more likely to be spam.
In addition to checking incoming mail against these lists we also monitor our own IP addresses on the lists as an early warning system. When one of our addresses is listed we take steps to find and remove the problem account while sending mail via one of our other outgoing servers to minimise disruption to our legitimate users.

Sender Policy Framework (SPF)

SPF is a domain based authentication system. It allows a recipient to verify that the sending server is authorised to send email for a particular domain.

Domain owners can publish a list of valid addresses in their DNS records, and can suggest how to deal with email which does not come from an address on that list.

Unfortunately, the sender address verified by SPF is not necessarily the sender address that you see in your email client. The address checked by SPF is the return path sent in the SMTP transaction, which may differ from the address in the email’s From: header.

Microsoft attempted to fix this by introducing SenderID, which is similar to SPF but can verify the From address header. However, there were numerous problems with this as a standard and it isn’t widely implemented.

At FastMail we use SPF as one of the many factors to spam filter incoming mail. For outgoing mail we specify our servers explicitly so that they get a positive score for successful SPF, but also say “?all” to allow for other systems to send from addresses on our domains. If you have your own domain with us, then of course then you can setup your own SPF records to be as strict or as liberal as you need.

DomainKeys Identified Mail (DKIM)

DKIM is a merging of two older standards; DomainKeys, and Identified Internet Mail. The intention of DKIM is to verify that an email associated with a particular domain was sent by an authorised agent of that domain and has not been modified since being sent. Based on Public Key Cryptography, a domain owner creates a public and private key pair, publishes the public part, and then uses the private part to sign the body and selected headers of an email. The receiver of an email is then able to check that signature against the public part of the key to verify that the sender of the email had access to the private part of the key, and therefore is authorised to send email on behalf of that domain.

Problems: Again, there is no stipulation that the domain in the From: header signs the email. It is possible for an email to be signed by any domain and pass basic DKIM checks.

At FastMail we sign all outgoing emails with a key for our messagingengine.com domain, and also with a key for the domain of your email address (e.g. fastmail.com).

If you use your own domain with us, we will automatically sign emails with a DKIM key if you host your DNS with us. We also make it super easy to setup both SPF and DKIM.

For incoming mail, again, we use DKIM as a tool in spam filtering. DKIM is also used to validate official emails from FastMail, We use the DKIM signature combined with the headers to validate that the email was sent by an official FastMail staff account and add a green tick next to legitimate emails.

Author Domain Signing Practices (ADSP)

ADSP is an extension to DKIM whereby a domain owner can publish a policy stating how email from their domain should be signed. A domain owner can state one of the following.

1. Legitimate email from this domain may or may not be signed by the
   domain.
2. Legitimate email from this domain will be signed by the domain.
3. Legitimate email from this domain will be signed by the domain, and
   non signed email should be discarded.

The domain used for ADSP is the domain in the From header of the email, and is the one most likely seen by the recipient.

Domain-based Message Authentication, Reporting & Conformance (DMARC)

DMARC brings together the SPF and DKIM checks and ties them in with the sender shown in the From address of the email.

In order to be considered ‘DMARC aligned’ an email must pass SPF and DKIM checks, the SPF domain must match the domain of the From address, and at least one DKIM signature must also match that domain. This provides a good level of certainty that the email is not forged.

Domain owners who choose to publish DMARC records can suggest what should be done with messages which do not pass DMARC tests, reject outright, or quarantine (treat as spam).

The reporting part of DMARC is a tool for domain owners rather than end users. Email receivers who fully implement DMARC build reports on email received, and send reports to domain owners who request them (via the published DMARC record). This report shows some basic aggregate information on number of emails received, the servers they were received from, and their SPF and DKIM status. This allows domain owners to discover how their domain is being used, which can then inform decisions on the best SPF/DKIM/DMARC policies to publish.

We are investigating how we can use DMARC to benefit FastMail users. Given the wide range of ways in which our users are using our services it would be bad to publish reject or quarantine policies. The number of legitimate emails which would be blocked by this would far outweigh any possible benefit. Implementing over restrictive policies in an environment where email is used in a diverse way would result in pain for users.

Challenges

The biggest issue by far is Mailing Lists. An email sent to a mailing list will typically be re-sent from the mail server of the list, breaking SPF, and usually has some alterations made to it (such as subject changes or unsubscribe links added), which break DKIM signatures.

It is also fairly common that a third party would send email on your behalf, for example, a company might contract out their support and ticketing system to a third party, and emails from the ticketing system would be sent from the companies domain. Care needs to be taken to ensure that these emails are also considered in SPF policies, and that the third party is able to properly DKIM sign these messages.

Third party senders has been one of the challenges we faced while implementing our green tick and phishing warnings system. This blog is hosted by WordPress, and sends email on behalf of FastMail (the blog notifications). We needed to make sure that these emails from WordPress could be identified and validated against the WordPress DKIM signature, check that they were sent from our WordPress blog, and make sure those emails were not marked with the phishing warning box. This needs to be done on a case by case basis as what identifies emails on WordPress isn’t likely to be the same as what identifies emails on other services such as Twitter.

Another common source of SPF failures is forwarding. If, for example, a user has migrated to FastMail from another provider, and has set up their old provider to forward email to their FastMail address, then we would see the IP address of the forwarding server, not the originating server, and this could result in an SPF failure for an otherwise legitimate email. There are some standards which attempt to address this such as Sender Rewriting Scheme (SRS), which involves rewriting the envelope sender to one at the forwarding domain. This fixes the SPF problem, but if the originating domain uses DMARC, then the email would no longer be aligned as the from addresses will no longer match.

The trouble of course is that phishing emails can be sent from entirely unrelated domains and are still successful. So sender authentication doesn’t always help.

Another problem with email authentication is a misunderstanding of what is being verified. We can take technical steps to verify that an email did come from a legitimate email account, but we can make no claim over how trustworthy the author of that email is. Anybody can purchase a domain, set it up properly with SPF, DKIM, and DMARC, and then use it to send spam or phishing emails. Also any service, including FastMail, faces the problem that accounts can be compromised and used to send bad content. Detecting and dealing with this is a whole other blog post.