Myths and Legends of SPF

SPF is an abbreviation for Sender Policy Framework (SPF) for Authorizing Use of Domains in Email. Email domains use this protocol to specify which Internet hosts are authorized to use this domain in the SMTP HELO and MAIL FROM commands. You do not have to use any additional software to publish the SPF policy and therefore this procedure is extremely simple: Simply add a TXT record containing the policy to the DNS zone. An example of this type of entry is given at the end of this article. There are numerous manuals and even online constructors for working with SPF.

The first ever version of the SPF standard was approved more than 10 years ago. During this time, numerous implementations and application practices have been developed. In addition, a new version of the standard has been released. But the most surprising is that for some reason SPF, more than any other standard, has grown over 10 years with an incredible amount of myths and misconceptions that wander from article to article and with an enviable regularity pop up in discussions and answers to questions on the forums. At the same time, the protocol itself seems very simple: implementation takes only a couple minutes. Let’s try to recall and analyze the most common misconceptions.

TL;DR — please see the recommendations at the end.

1. Misconception: SPF will protect my domain from spoofing

Fact: SPF does not protect the sender’s address that is visible to the user.

Explanation: SPF does not work with the contents of the message that the user sees, in particular, the sender’s address. SPF authorizes and verifies addresses at the mail transport level (SMTP) between two MTAs (envelope-from, RFC5321.MailFrom aka Return-Path). These addresses are not visible to the user, and they can differ from those in the From header that the user sees (RFC5322.From). Thus, nothing prevents a message with a fake sender in the ‘From’ header from being authorized with SPF.

Use DMARC to protect visible domain name from spoofing.

2. Misconception: After implementation, SPF will improve security and combat spam

Fact: most likely, you will not see any significant changes in terms of security and spam.

Explanation: SPF is originally an altruistic protocol, so it does not provide any advantages to anyone who publishes the SPF policy. Theoretically, if you implemented SPF, this protocol could protect someone else from receiving fake emails from your domain. But in fact, even this assumption is not true, because the results of applying SPF are rarely used directly (we’ll discuss all of this later). Moreover, even if all domains published SPF, and all recipients forbade receiving messages without SPF authorization, such an approach would hardly reduce the amount of spam.

SPF does not protect against spoofing or spam directly, nevertheless, this protocol is actively and successfully used to deploy spam filtering systems, as well as to protect against counterfeit emails, since it allows you to check each message against a specific domain and its reputation.

3. Misconception: SPF negatively (positively) influences email deliverability

Fact: It all depends on the type of message, the way it is delivered and your reputation

Explanation: SPF is not meant to affect email deliverability within a standard flow, and adversely impacts the improper implementation or indirect flows of messages, when users receive such messages from a server that differs from that from which the message was sent, for example, this applies to redirected emails. But spam filtering systems and reputation-based classifiers take into account the availability of SPF and reputation of authorizing domain, and this generally gives a positive result with respect to the standard message flow. Unless, of course, you yourself are a spammer.

4. Misconception: SPF provides authorization of the email sender

Fact: SPF provides authorization of the email server that sends a message on behalf of a domain

Explanation: Firstly, SPF works only at the domain level, and not at the level of individual email addresses. Secondly, even if you are a legitimate email user of a specific domain, SPF does not allow you to send messages from anywhere that you wish. In order for your message to successfully pass SPF validation, you must send it only from an authorized server. Thirdly, if you authorized a server using SPF (for example, you could allow sending emails from your domain via any ESP or hosting provider), and this server does not impose any additional restrictions, then all users of this server are authorized to send messages on behalf of your domain. Please keep this in mind when implementing SPF and providing authentication of email messages in general.

5. Misconception: Email messages not authorized by SPF will be rejected

Fact: In general, SPF authorization or lack thereof does not have a significant impact on the delivery of email messages.

Explanation: SPF is only an authorization standard, and it explicitly indicates that actions to be applied to email messages that were not authorized are outside the scope of the standard and are governed by the recipient’s local policy. If there is a ban on receiving such messages, this leads to problems with messages going through indirect delivery routes, for example, when using redirection or mailing lists, and you should consider this fact in the local policy. In practice, it is not recommended to use a strict ban in case of an SPF authorization failure. Standard allows (but does not require) strict ban only when the domain publishes the -all (hardfail) policy in the absence of other filters. In most cases, SPF authorization is used as one of the factors in the weighted systems. At the same time, this factor will have an insignificant weight, because violation of SPF authorization is usually not a reliable indicator of spam: many spam messages successfully pass SPF authorization, and legal ones often can not do this, and it is unlikely that we will ever witness cardinal changes in this field. If we look at it this way, there is no difference between -all and ~all.

SPF authorization is not so important in terms of message delivery or spam filtering, but it allows for confirmation of the sender’s address and the relationship with the domain, as well as the use of the reputation of the domain instead of the IP reputation for this message.

DMARC policy has a much more significant influence on the decision-making on further actions in relation to handling a message that has not passed authorization. DMARC allows you to reject (or quarantine) all or part of messages that have not been authorized.

6. Misconception: SPF recommends using -all (hardfail), since it is safer than ?all or ~all

Fact: In fact, -all does not affect security in any way, but it negatively affects the delivery of messages.

Explanation: -all results in the blocking of messages that were sent through indirect routes by those few recipients who use SPF directly and block messages. At the same time, this policy will not have a significant impact on most spam and fake messages. At the moment, ~all (softfail) is considered the most appropriate policy and it is used by almost all large domains, even those that impose very strict security requirements (such as paypal.com). -all can be used for domains that are not used for sending legitimate emails. DMARC considers -, ~ and ? as equivalents.

7. Misconception: It is sufficient to configure SPF only for domains that are used to send mail

Fact: It is also necessary to configure SPF for domains that are used in HELO on mail servers. In addition, it is recommended to apply a blocking policy for MX, A records and the wildcard that are not used to send emails.

Explanation: In some cases, in particular, when delivering NDR (a non-delivery report), DSN (delivery status notification) and some auto-responses, the address of the sender in the SMTP envelope (envelope-from) will be empty. In this case, SPF checks the host name from the HELO/EHLO command. You need to check the name from this command (for example, by opening the server configuration or by sending an email to a public server and checking the headers) and enable SPF for this name.

Spammers can use not only the same domains that you use to send messages, they can send spam on behalf of any host that has an A- or MX record. Therefore, if you publish SPF from altruistic considerations, then you need to add SPF for all such records, and it is also desirable to add a wildcard (*) for nonexistent records.

8. Misconception: It’s better to add a special SPF type record to DNS (instead of TXT)

Fact: It must be a TXT record.

Explanation: According to the current version of the SPF standard (RFC 7208), SPF type DNS records are deprecated and should no longer be used.

9. Misconception: it is recommended that you include as many of the available elements in SPF as possible (a, mx, ptr, include), because this can reduce the likelihood of an error

Fact: it is necessary to minimize the SPF record and it is recommended to specify only addresses of the networks via ip4/ip6.

Explanation: There is a limit of 10 DNS queries for resolving the SPF policy. Exceeding this limit will result in a permanent policy error (permerror). Moreover, DNS is an unreliable service, so there is a probability of a failure (temperror) for each request, which increases with the number of requests. Each additional a or include record requires an additional DNS request; as for include, it is also necessary to request all elements specified in the include record. mx requires to request MX records and an additional A record request for each MX server. ptr requires an additional request, moreover, it is inherently unsafe. Only the addresses of the networks listed through ip4/ip6 do not require additional DNS requests.

10. Misconception: TTL for the SPF record should be smaller (larger)

Fact: As for most DNS records, it is better to choose TTL from the range of 1 hour to 1 day and reduce it in advance during the deployment or implementation of planned changes, or increase, when the policies are stable.

Explanation: A higher TTL reduces the likelihood of DNS errors and, as a consequence, SPF temperrors, but it increases the response time when it is necessary to make changes to the SPF record.

11. Misconception: If I don’t know which IP addresses can be used to send my messages, then it is better to publish the policy with +all

Fact: A policy with an explicit +all or an implicit rule that enables mailing on behalf of the domain name from any IP address will negatively affect the delivery of emails.

Explanation: Such a policy does not make sense, and it is often used by spammers to ensure SPF authentication of spam messages that are sent through botnets. Therefore, a domain that publishes such a policy risks being blocked.

12. Misconception: It does not make sense to use SPF

Fact: It is necessary to use SPF.

Explanation: SPF is one of the mechanisms for authorizing the sender in email and the way to identify the domain in reputation-based systems. Currently, large email service providers are gradually beginning to require the authorization of messages, and messages that do not have authorization can be subject to “penalties” in terms of delivery or display to the user. In addition, there may be no auto-responses and notifications on delivery or non-delivery for messages that have not passed SPF authorization. The reason is that such responses are usually sent exactly to the SMTP envelope address SPF authorizes and require that it is authorized. Therefore, SPF is required even if all messages are authorized by DKIM. Also, SPF is a must for IPv6 networks and cloud services: In such networks, it is almost impossible to use the reputation of IP addresses, and messages from addresses without SPF authorization will, as a rule, not be accepted. In accordance with the standard, one of the primary tasks of SPF is to use the reputation of a domain name instead of the IP reputation.

13. Misconception: SPF is self-sufficient

Fact: DKIM and DMARC are also necessary.

Explanation: DKIM is required to successfully forward email messages. DMARC is required to protect the sender’s address from spoofing. In addition, DMARC allows you to receive reports on violations of the SPF policy.

14. Misconception: Two SPF records are better than one

Fact: The record must be exactly one.

Explanation: This requirement is described in the standard. If there is more than one record, this will result in a permanent error (permerror). If it is necessary to merge several SPF records, simply publish a record with several include operators.

15. Misconception: spf1 is good, but spf2.0 is better

Fact: You should use v=spf1.

Explanation: spf2.0 does not exist. By publishing the spf2.0 record, you can expose yourself to the risk of unpredictable results. spf2.0 has never existed and it is not a standard, but it was mentioned in the experimental standard RFC 4406 (Sender ID), which was based on the assumption that such a standard would be adopted, since the corresponding discussions did take place. Sender ID, which was supposed to solve the problem of address spoofing, did not become a generally accepted standard, and you should use DMARC instead. Even if you decide to use Sender ID and publish the spf2.0 entry, it will not be a replacement for the spf1 entry.

I had almost finished writing this article, when I was suddenly intercepted by our Customer Support staff who strongly (and categorically) recommended that I recall the following nuances of SPF that they often have to deal with when solving various issues:

1. SPF policy should end with the all or redirect directive. There should be absolutely nothing after these directives.
2. all or redirect directives can be used in this policy exactly once, and they replace each other (that is, one policy can not include all and redirect simultaneously).
3. The include directive does not replace all or redirect. include can be used several times, but your policy should still be terminated with the all or redirect directive. include or redirect should be used to include a valid policy terminated with all or redirect. At the same time, include does not pay attention to the rule (-all, ~all, ?all) that is used for all in the included policy, however, for redirect it makes a difference.
4. include is used with colons (include:example.com), and redirect requires the sign of equality (redirect=example.com).
5. SPF does not cover subdomains. SPF DOES NOT СOVER SUBDOMAINS . SPF. DOES. NOT. COVER. SUBDOMAINS. (And DMARC, by default, covers them). You should publish SPF for each A or MX record in DNS, if those records are used or can be used to deliver emails.

Summary

• You should be sure that you create a policy for all MX and, preferably, all A records.
• Add SPF policies for the names used in the HELO/EHLO of the mail server.
• Publish the SPF policy as a TXT record with v=spf1 in DNS.
• Try to use the addresses as ip4/ip6 networks in your policy. Specify them at the beginning of the policy to avoid unnecessary DNS requests. Minimize the use of include, try to do without a, use mx only in case of extreme and irresistible necessity, and never use ptr.
• Specify ~all for domains that are really used to send emails, -all for unused domains and records.
• Use small TTL during the implementation and testing period, and then increase TTL to the appropriate values. Before making any changes, reduce the TTL beforehand.
• Be sure to validate your SPF policy, for example, here.
• Do not limit yourself to the deployment of SPF, try to implement DKIM and always implement DMARC. DMARC protects your messages from spoofing and allows you to receive information on violations of the SPF. You will be able to detect forgery, indirect message flows and configuration errors.
• If you read Russian (or just for fun) after implementing SPF, DKIM and/or DMARC, check them using https://postmaster.mail.ru/security/. SPF and DMARC are validated according to the current state; DKIM is checked using the statistics for the previous day only if there is correspondence with the boxes on Mail.Ru on the previous day or earlier.
• There are good SPF BCPs from M3AAWG

Sample SPF policy: @ IN TXT “v=spf1 ip4:1.2.3.0/24 include:_spf.myesp.example.com ~all”