The A-Z Of Email Development And Sending Emails

The first electronic mail was sent in 1971 via the . Since that time, progress has leaped forward and emails have become an inseparable part of daily life. By the end of 2019, the number of email users worldwide is expected to reach almost 3 billion people! Developers, in turn, set up things to let users send and receive emails. It's not rocket science, but you must know your oats with how email works. That's why the Mailtrap team decided to open the hood and guide you through the email basics that every developer must know. ARPANET Disclaimer: it would have been a very long text (much longer than it is) if I’d gone into detail for each subtopic. I decided to make it as brief as possible. Email basics Email is a way to send some digital data from a sender to a recipient via the Internet. Both the sender and the recipient have a unique email address that corresponds to a standard format: . For example, @ sender@gmail.com An email provider is a service where you can open an email account and get an email address to send and receive electronic mail. Gmail is truly the most famous web-based email provider. This is a very basic introduction to what email is. Now, let’s explore the more sophisticated things such as types of emails, what they consist of, and more. Types of emails: marketing vs. transactional If you ask Google how many types of emails there are, you’ll find versatile selections like five, seven, fifteen, and so on. From the developer’s perspective, marketing and transactional emails are in focus. are not exactly what engineers regularly deal with. However, they form bulk mail, when the same content is delivered to a large number of recipients at the same time. Marketing emails include messages aimed at selling or promoting something. They include newsletters, or feature updates, and so on. Marketing emails product launch are usually what developers have direct control over. They are generated and sent when a ‘transaction’ happens. Transactions include form submission, making a purchase, requesting a password reset, and more. The main value of transactional emails is that users want to receive them. Transactional emails Email headers, Cc, and Bcc explained An email header is a section of code that contains specific data about the email’s origin. The recipient’s email client usually displays the or : partial standard headers displays the sender’s email address. From: displays the subject line of the email. Subject: displays when (date and time) the email was created. Date: displays the recipient. In some cases, the recipient’s email address may be hidden. To: (or Return-Path:) displays the email address for return mail. Reply To: stands for carbon copy. It is a standard header aimed at notifying the recipients about the content of the message. CC: stands for blind carbon copy. It specifies an additional copy to the recipient who will get the email and see other direct or copied recipients. The BCC: recipient, however, won’t be visible to other recipients. BCC: The contain the about the email. Here are the most common ones: full headers tech information displays the mailbox of a recipient the email that was delivered to. Envelope-To: displays when (date and time) the email was received by the email client. Delivery Date: displays a list of all servers the email was routed through. As a rule, the Received: header is represented by several lines, of which the last one is where the email originated and the first one is the email destination. Received: displays details on SPF authentication. The result can be or pass for a received email, and for a rejected one. Received-SPF: neutral fail specifies information about the sender, the email, and the public key location required for authentication. DKIM-Signature: displays details on SMTP authentication. We’ll touch on this topic, as well as SPF and DKIM in the Email authentication explained section. Authentication-Results: displays a unique string assigned to the email when created. Message-ID: MIME-Version: indicates that the rules of Multipurpose Internet Mail Extensions are followed. MIME is an Internet standard that extends the format of email. The value of this header is constantly 1.0. displays the email format like HTML, plaintext or multipart. Content-Type: displays the actual content of the email. Message Body: displays information about any attachments. Content-Disposition: displays the encoding scheme for the content. Content-Transfer-Encoding: -[Level/Score/Report]: displays information on how spammy the email is according to SpamAssassin. X-Spam Sample Email Headers Return-Path: Received: from mail-sor-f41.google.com (mail-sor-f41.google.com. [209.85.220.41]) by mx.google.com with SMTPS id y1sor11976446oti.117.2019.10.23.23.32.35 for (Google Transport Security); Wed, 23 Oct 2019 23:32:36 -0700 (PDT) Received-SPF: pass (google.com: domain of sender@gmail.com designates 209.85.220.41 as permitted sender) client-ip=209.85.220.41; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b="UnP/Lig1"; spf=pass (google.com: domain of sender@gmail.com designates 209.85.220.41 as permitted sender) smtp.mailfrom=sender@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to:cc; bh=hrJ9yUbW0UidsTSyBmObH/qjSPBcHeZvop1/UXKVlUM=; b=UnP/Lig15hfjuV+5WGuJ1vfGscNuidgOpz+wP61KBaK3TK1bXbh4A1mX9LR028tD4S Pc5eP23vXRiwUnoXZ5qJI6ZVXzNejZbIprnZ42j6XvxqIo5nRd4OFqyaYt0unrFDEuvM MIME-Version: 1.0 From: The Sender Date: Thu, 24 Oct 2019 09:32:54 +0300 Message-ID: Subject: Test email To: The Recipient Cc: The RecipientCC Content-Type: text/html; charset="US-ASCII"; name="Test email Headers.html" Content-Disposition: attachment; filename="Test email Headers.html" Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: f_k24bxtnk0 Delivered-To: recipient@gmail.com Received : by 2002 :a02:c727: 0 : 0 : 0 : 0 : 0 with SMTP id h7csp1944549jao; Wed, 23 Oct 2019 23 : 32 : 36 - 0700 (PDT) Email content: plain text, HTML, images, encoding The header tells the email client what kind of data the email contains. The simplest option is the . But the MIME protocol supports other : Content-Type text/plain types of content according to IANA application (for example, ) application/json audio (for example, ) audio/mpeg font (for example, ) font/ttf example (any subtype following the media type syntax may be used in examples) image (for example, ) image/png message (for example, ) message/global-delivery-status model (for example, ) model/gltf-binary multipart (for example, ) multipart/form-data text (for example, ) text/css (for example, ) video video/mp4 HTML email The header will indicate if you use HTML in your email. and are different things. When you need to create HTML content for your email, you should bear in mind the list of specific rules: Content-Type text/html HTML web development HTML email building No rendering standards are applied since recipients may use different email clients. An HTML email will be rendered only once it ends up in the inbox. So, it must come through spam filters. An HTML email should be responsive – it should look on any device and on any screen size. as designed An HTML email should be previewed before sending. For more on coding HTML emails, read . Mailtrap’s Guide on How to Build HTML Email Multipart Email Multipart emails contain different types of content in one go. So, they have several headers. Check out the following example: Content-Type Message-ID: Subject: Test email To: The Recipient Content-Type: multipart/mixed; boundary="000000000000f372640595a23214" --000000000000f372640595a23214 Content-Type: multipart/alternative; boundary="000000000000f372620595a23212" --000000000000f372620595a23212 Content-Type: text/plain; charset="UTF-8" Hello --000000000000f372620595a23212 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello! --000000000000f372620595a23212-- --000000000000f372640595a23214 Content-Type: image/png; name="Capture001.png" Content-Disposition: attachment; filename="Capture001.png" Content-Transfer-Encoding: base64 Content-ID: MIME-Version: 1.0 From : The Sender Date : Thu, 24 Oct 2019 09 : 32 : 54 + 0300 Here you can see five different headers: Content-Type denotes that there is a mix of content. In our case, this is the plain text and HTML. multipart/mixed denotes that the email contains an attachment as well. multipart/alternative specifies the plain text part of the content. text/plain specifies the HTML part of the content. text/html specifies the attachment part of the content. image/png Boundaries among the email content parts are defined with the parameter, which is specified after boundary=”” Content-Type multipart/. Attached or embedded images in emails Enclosing and embedding are two basic approaches for attaching images to emails. When you enclose an image, you add it as a separate file. The HTML structure is not affected, and the attached image remains unchanged. The drawback is that your email becomes heavier. When you embed an image, it becomes an integral part of the HTML body. In this case, you do not have to worry about the email size, but your concerns shift to the attachment and how it will be encoded. hosting Content encoding Now, a couple of words about . This header may have one of the following values: Content-Transfer-Encoding - this is a default value, which implies no encoding. It ranges from 0000000 to 1111111, or 0 to 127 in base 10. That means ASCII 7-bit encoding has room for 128 different characters that can be represented as numbers. If the Content-Transfer-Encoding header is not present, then the 7bit encoding type is assumed. 7bit - a 65-character subset of US-ASCII is used for encoding, enabling 6 bits to be represented per printable character. The base64 mechanism encodes data of any kind, and the output is represented in lines of no more than 76 characters each and is not human-readable at all. base64 - the mechanism encodes the data in such a way that the resulting octets are unlikely to be modified by mail transport. The encoded form of the data remains largely recognizable by humans. quoted-printable In , there are also defined and values. They, however, have never been officially implemented. . Besides, these mechanisms imply NO encoding. RFC 1341 8bit binary “There are no circumstances in which the 8bit or binary Content-Transfer-Encoding is actually legal on the Internet” Email template explained An email template is a reusable HTML file for email campaigns. It allows you to tune up your email by adding styling elements such as images, cascading style sheets (CSS), and others. The main advantage of using email templates is time efficiency. With them, you produce your email campaigns faster, since most of the code is already there. Maximum email size When building an email, be it a transactional or a marketing one, you consider maximum email size limits. The first limit is for sending - it is set by your email service provider. The second limit is for retrieving - it is set by the email provider of your recipient. Limits are established mostly based on performance and security reasons. For example, with Gmail, you can send and retrieve emails up to 25MB. That’s why you may be willing to learn more about the . ways to optimize email size limits Basics of sending and receiving email In brief, each email comes through three checkpoints: the sender => the mail server => the recipient. If we expand this flow, we get the following: A mail user agent (MUA) hands over the email to the mail submission agent (MSA). From the MSA, the email gets to the mail transfer agent (MTA). Both the MTA and MSA are hosted on the same SMTP server. The MTA may transfer the email to another MTA or right to the mail delivery agent (MDA). The MDA delivers the email when the recipient’s MUA requests it. All these transferring parties connect to each other via different email protocols. Email protocols explained There are three email protocols: . Of them, one is for sending, and two are for receiving. Simple Mail Transfer Protocol (SMTP) is meant only to send emails. With SMTP, an email gets from the sender to the email server and between servers. Once the email is already at the mail server, Internet Message Access Protocol (IMAP) or (Post Office Protocol version 3) POP3 comes into play to retrieve it from there. SMTP, IMAP, and POP3 SMTP basics Underneath each email delivery lies an SMTP conversation that consists of three stages: SMTP handshake - the SMTP client establishes a TCP connection with the SMTP server. Email transfer. Termination – the client and the server say goodbye to each other. The SMTP conversation is carried out using the commands from the client and responses from the server. SMTP commands and responses The client talks using commands that consist of alphabetical characters - for example, , and others. with numerical codes such as , and so on. The numerical code is followed by a text meant for a human user to get the point. For more on SMTP commands and replies, refer to . HELO, MAIL FROM The server responds 220, 250, 354 RFC 5321 SMTP ports Ports are communication endpoints commonly used to identify the exact location of an Internet address. Each email is sent to a dedicated server and a predefined SMTP port. There are four standard : 25, 465, 587, and 2525. SMTP ports SMTP server An SMTP server is a computer program or app that sends or relays outgoing emails using SMTP. There are cloud-based and self-hosted servers. Cloud-based is the most common option for numerous businesses and startups. Gmail, Mailgun, SendGrid, and many more are SMTP server providers. For a self-hosted option, you need to build the whole of the SMTP server yourself. Usually, this is what large corporations opt for. At the same time, you can set up your own local SMTP server and use the email relay services for sending emails. email infrastructure SMTP relay From a technical standpoint, SMTP relay is the process of transferring emails from one mail transfer agent to another. Here is a visual explanation of when SMTP relay comes into play. In the context of email marketing, however, SMTP relay services usually refer to the actual SMTP servers. SMTP queues An email queue is a system within the SMTP server that creates an array of emails waiting to be processed for delivery. Emails are put in a queue that provides temporary storage before processing. SMTP email queues decouple the sender from the recipient - they communicate without being connected. SMTP relay vs. HTTP API SMTP relay is the most common email transfer method for sending transactional and bulk emails. It is based on the Simple Mail Transfer Protocol and is a good solution for casual email senders who need simplicity for basic tasks. Web API or HTTP API is a type of interface where the communication takes place using the Hypertext Transfer Protocol (HTTP). It makes email sending quicker because there is less back and forth required compared to the SMTP. Email authentication explained SMTP lacks a feature that would identify the origin of an email message and validate its domain. Hence, it can’t protect your app from frauds such as email spoofing, , and spam. That’s why you need email authentication. SPF, DKIM, and DMARC are three widely adopted standards to authenticate emails. phishing SPF (Sender Policy Framework) SPF is a protocol according to which the mail servers decide whether to receive or reject an incoming email. The information about authorized IP addresses within a particular domain is stored in the TXT record known as the SPF record. Here is a simple example of it: v=spf1 +a +mx redirect=example.com -all DKIM (DomainKeys Identified Mail) that’s added to every email sent from a given email address. It’s a seemingly random set of characters that are hidden in the source code of an email. Here is how it may look: DKIM is a digital signature bh=MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=; b=dzdVyOfAKCdLXdJOc9G2q8LoXSlEniSbav+yuU4zGeeruD00lszZ VoG4ZHRNiYzR DKIM-Signature: v= 1 ; a=rsa-sha256; d=example.net; s=newyork; c=relaxed/simple; q=dns/txt; t= 1117574938 ; x= 1118006938 ; h= from :to:subject:date:keywords:keywords; DMARC (Domain-Based Message Authentication Reporting and Conformance) is not a self-consistent authentication method. It leverages DKIM and SPF to perform a more advanced check on each email received. Domain owners can specify their own authentication procedure known as a DMARC policy and instruct an incoming server on what to do if an email fails to pass the DMARC test. Here is an example of a DMARC record: DMARC v=DMARC1; p=reject; rua=mailto:dmarc-rua@square.com, mailto :dmarc_agg@vali.email, mailto :postmasters@squareup.com; ruf=mailto:dmarc-ruf@squareup.com SMTP authentication or SMTP AUTH SMTP AUTH is the service extension of the ESMTP. It requires that an email sender (client) must have permission to use the email server. Authentication is carried out according to the SASL mechanism. The most common ones are PLAIN, LOGIN, and CRAM-MD5, but there are also many others. Email security explained SMTP comes with no security features. This entails vulnerability to different forms of threats and attacks. For example, malevolent people can access your emails or use your SMTP server for sending spam. Email providers are interested in protecting their customers and use security layers such as SSL or TLS. Another approach to securing email communication is to use an end-to-end encryption method such as S/MIME, PGP, and others. Email encryption methods Secure Socket Layer or SSL is a cryptographic protocol used in email transmission. It turns emails into unreadable strings of characters, so no one could use them if intercepted. Transport Layer Security or TLS is a successor of SSL. It offers a more reliable security profile. Unlike those above, STARTTLS is not an encryption protocol but an SMTP command that tells an SMTP server that an email client wants to use a secure connection using SSL or TLS. Read about to learn more about encryption in email transmissions. SSL, TLS, and STARTTLS Email testing checklist If you’re planning to send your email to thousands of recipients, testing is the first thing that needs to be done. For this, you can follow a simple , consisting of: email testing checklist Testing the email sending capability SMTP server testing Test emails sending Spam testing Email content testing Testing the email sending capability In this part, you need to test two things: whether your app is capable of sending emails, and whether the mail server functions well. Sending test emails A proper SMTP server for testing is a fake one. It can make your . But the main advantage is that your test emails won’t go outside your staging environment. There are two ways to set up a fake SMTP server for testing: email testing safe and easy The first one is to adopt a ready-to-use cloud-based solution or SaaS. You may consider a professional service such as Mailtrap.io. It has a dummy SMTP server underneath and provides a set of other useful features like spam analysis, HTML check, and many more. The second way is to install a local SMTP server for testing. The popular solutions include MailCatcher and MailHog. They’re flexible in use, but you’ll have to put in some time and effort setting them up. As an alternative, you can use a real SMTP server combined with temporary email accounts known as dummy emails. Services such as Mailinator or Maildrop can help you create those. Whereas this approach may be good for testing a very simple web app, you shouldn’t go with it for more complex projects. Why? There are many reasons such as a necessity to create hundreds of test email accounts, a short lifetime of those, plain text support only, and others. SMTP server testing Let’s say your app has successfully passed the test for email sending. Now, you need to check whether the SMTP server responds to the client commands and can send emails. This can be done manually using Telnet or you can opt for a dedicated online service such as MXToolbox or Wormly. Also, do not neglect to test your SMTP server for relaying. This includes checking the SMTP connection and the authentication mechanism. For this, you can also choose either an installable or web-based tool. Deliverability or spam testing Delivery denotes whether an email was accepted by a recipient server. If your email , it was not necessarily your fault. In some cases, the failed delivery can be out of the sender’s control. bounced soft or hard Deliverability denotes whether an email ended up in the inbox instead of a spam folder. So, you need to test how spammy your email is. Tools such as Spam Check or SendForensics allow you to check this and point out some weak points. At the same time, you may need to do some work and implement a few of the . best practices to improve email deliverability Test email content The final stage of email testing focuses on checking text and formatting. What you can do here is: Preview you email to learn how it will look on different devices, browsers, or email clients. Validate the markup (HTML, XHTML, and CSS) of your email. Brush up the body copy and (yes, spam filters check the grammar, as well). translation Test . the subject line Split test your email campaign. Something else you may be interested in is Bcc testing. The emails meant for Bcc’ed recipients may get to the spam folder, and there was no functionality to test and debug this. The Mailtrap team, however, marked this issue and delivered a feature to test Bcc in emails. To wrap up This email basics tutorial is designed for developers. However, it will be of use to anyone looking for layman’s terms of many email-related things in one place. Good luck with your email sending stuff! P.S. I will highly appreciate your feedback so don't hesitate to share it with me on :) Linkedin