Receiving and Parsing Email in Rails 5

With Griddler and Mailgun

Another aspect of email within a SaaS application is receiving mail. While this is far less normal or used in comparison to sending, it can be a great way to make end user’s responses to email or action items quicker.

At a high level, there are a few different layers to this. The topmost layer is the email service, and this book’s case, Mailgun. This service handles sending outbound emails, as well as routing incoming emails to an address/domain name specified in their interface. Once the email is routed, it will be redirected to a route and processor file in your application. This file will be responsible for parsing the incoming email address and using logic to decide what to do with it.

Note: This article is an excerpt derived from a chapter in my upcoming book Build A SaaS App in Rails 6. The book guides you from humble beginnings through deploying an app to production. The book is now in pre sale, and you can grab a free chapter right now!

Also, the beta for my new project Pull Manager is has been released. If you’re losing track of pull requests, have old ones lingering around or would just love a dashboard that aggregates these requests over multiple services (Github, Gitlab, and Bitbucket), check it out.

In the case of the Standup App we are building, we can have directions to respond to an Email Reminder to create a new standup right from their email response! Some of the tools that we will be using are Mailgun’s email routing service and ngrok, an HTTP tunneling service. ngrok is very useful for a few solution throughout the remainder of this book. It allows you to have a web accessible URL, which tunnels(connects) to ports within your local machine. Meaning, that you will have a http://somesubdomain.ngrok.com that will forward to your computer, and a specific port specified when you start ngrok. This allows you to test with external services such as Mailgun, Stripe(later), Github(later) and more!

Let’s get started with some setup:

• Download ngrok
• Unzip and move the executable file to where you would like.
• In *nix OS’s, open ngrok with: path/to/ngrok HTTP start 3000. This will be dependent of the port you are using for your Rails server. ngrok will now fire up a tunnel service with a randomly generated URL.
• Optionally, if you upgrade to a paid version of ngrok, you can set a subdomain, so you do not have to change your settings elsewhere every time you restart ngrok.
• In Mailgun go to the Routes tab to create the email route(modeled after MVC routes like Rails) that will send the email. Enter the following settings:
• Chose Match Recipient
• Enter development.standup.*@app.yourdomain.com for the recipient field. Mailgun routes allow wildcard character matching, which will allow you input extra characters in the email's reply-to. Meaning, adding something like a user's hash_id to have identifiable information from the incoming email.
• Check forward and enter http://yoursubdomain.ngrok.io/email_processor as the destination of that forward.
• You can leave the priority alone and give the route a name, before pressing the submit button.

Now with that done, we can begin to modify the application. The application changes will consist of three main parts. First, a new gem(and a companion adapter gem) will be added. griddler is the main library to handle incoming email easily; griddler-mailgun is the Mailgun specific adapter that allows the griddler functionality to work with Mailgun as an incoming mail router. Next, Griddler will need to add a few parts of configuration application-wide to make sure some basic configuration is met. Lastly, an email processor file will be added to handle recieving and parsing the email.

The great part about adding Griddler to your current application is that if you are using the Gemfile from Chapter 3, you already have it installed. If not, just add gem 'griddler' and gem 'griddler-mailgun' to your Gemfile and run a bundle install.

Next to configure and setup Griddler in the application you will need to add a new file in the initializer folder setting a few configuration values. Then, add a quick line to add the default Griddler routing into the routes.rb file.

First the Griddler configuration:

Griddler.configure do |config|config.reply_delimiter = '-- REPLY ABOVE THIS LINE --'config.email_service = :mailgunend

This will set the text the griddler library will look for in the email and tell it that it will use the installed Mailgun adapter.

Next, add a line to mount the library based routes into your application. Adding the line right abobve the root to: is fine:

Rails.application.routes.draw do...

mount using default path: /email_processor

mount_griddler

root to: 'activity#mine'end

By mounting Griddler with that syntax, it will automatically add a route to your application that will route from a specified endpoint to a Griddler based controller:

email_processor POST /email_processor(.:format)

griddler/emails#create

There are settings in Griddler’s GitHub documentation to change the defaults, but unless you want to get creative with route paths or email processor class names, it can be unnecessary.

The default settings expect a class EmailProcessor to exist and to handle parsing the incoming email with a method process. Griddler, however, does not care where the actual file is placed, but that the class exists and is loaded. Personally, I find that email processing fits most into the services definition and can be placed there.

To allow Griddler and its files to capture text from incoming replies, there are a few changes that will be needed.

First we will update the EmailReminderMailer to create a unique reply-to address and include that email address as part of the outgoing email:

class EmailReminderMailer < ApplicationMailerdef reminder_email(user, team)@user = user@team = teamreply_to = "'Standup App' <#{'development.' if Rails.env.development?}standup.#{@app.yourdomain.com">user.hash_id}@app.yourdomain.com>"mail(to: @user.email,subject: "#{team.name} Standup Reminder!",reply_to: reply_to)endend

Here we are using building the reply_to string by adding development if the current Rails environment is your local Rails application. This way, the separate Mailgun route made for development can route different from a production route you will add before deploying your application.

Next we will updated the mailer template to have ##- Please type your reply above this line -## and some text letting the email recipient know they can add a standup by replying:

## app/views/email_reminder_mailer/reminder_email.html.slimdoctype htmlhtml xmlns="http://www.w3.org/1999/xhtml"headmeta content="width=device-width" name="viewport" /meta content=("text/html; charset=UTF-8") http-equiv="Content-Type" /title= "#{@team.name} Reminder!"css:| *{margin:0;padding:0;font-family:"Open Sans",Helvetica,Helvetica,Arial,sans-serif;box-sizing:border-box;font-size:14px}img{max-width:100%}body{-webkit-font-smoothing:antialiased;-webkit-text-size-adjust:none;width:100%!important;height:100%;line-height:1.6}table td{vertical-align:top}body{background-color:#f6f6f6}.body-wrap{background-color:#f6f6f6;width:100%}.container{display:block!important;max-width:800px!important;margin:0auto!important;clear:both!important}.content{max-width:800px;margin:0auto;display:block;padding:20px}.main{background:#fff;border:1px solid #e9e9e9;border-radius:3px}.content-wrap{padding:20px}.content-block{padding:0 0 20px}.header{width:100%;margin-bottom:20px}.footer{width:100%;clear:both;color:#999;padding:20px}.footer a{color:#999}.footer a,.footerp,.footer td,.footer unsubscribe{font-size:12px}h1,h2,h3,a,th,td{font-family:"Open Sans",Helvetica,Arial,"Lucida Grande",sans-serif;color:#000;margin:40px 0 0;line-height:1.2;font-weight:400}h1{font-size:32px;font-weight:500}h2{font-size:24px}h3{font-size:18px}h4{font-size:14px;font-weight:600}ol,p,ul{margin-bottom:10px;font-weight:400}ol li,p li,ulli{margin-left:5px;list-style-position:inside}a{color:##3c8dbc;text-decoration:underline}.btn-primary{text-decoration:none;color:#FFF;background-color:##3c8dbc;border:solid ##3c8dbc;border-width:5px 10px;line-height:2;font-weight:700;text-align:center;cursor:pointer;display:inline-block;border-radius:5px;text-transform:capitalize}.last{margin-bottom:0}.first{margin-top:0}.aligncenter{text-align:center}.alignright{text-align:right}.alignleft{text-align:left}.clear{clear:both}.alert{font-size:16px;color:#fff;font-weight:500;padding:20px;text-align:center;border-radius:3px 3px0 0}.alert a{color:#fff;text-decoration:none;font-weight:500;font-size:16px}.alert.alert-warning{background:#f8ac59}.alert.alert-bad{background:#ed5565}.alert.alert-good{background:##3c8dbc}.invoice{margin:40pxauto;text-align:left;width:80%}.invoice td{padding:5px 0}.invoice .invoice-items{width:100%}.invoice .invoice-items td{border-top:#eee 1pxsolid}.invoice .invoice-items .total td{border-top:2px solid #333;border-bottom:2px solid #333;font-weight:700}@media only screen and (max-width:640px){h1,h2,h3,h4{font-weight:600!important;margin:20px 0 5px!important}h1{font-size:22px!important}h2{font-size:18px!important}h3{font-size:16px!important}.container{width:100% !important}.content,.content-wrap{padding:10px !important}.invoice{width:100% !important}bodydiv style="color: #b5b5b5;text-align:center;"| ##- Please type your reply above this line -##table.body-wrap style="width:100%"trtdtd.container width="800".contenttable.main cellpadding="0" cellspacing="0" width="100%"trtd.content-wraptable cellpadding="0" cellspacing="0" style="width:100%"trtd.aligncenter| Standup Apptrtd.content-blockh3= "#{@team.name} Reminder!"trtd.content-block= "Just wanted to remind you to add your standup for \the team: #{@team.name}"trtd.content-block.aligncenter= link_to "Add Your Standup", new_standup_url(), \{class:"btn-primary", style: "width:95%"}trtd.content-block= "You can quickly submit your standup by replying to \this email in the format:"prepre= "[d] This is a done item\n[t] This is a todo item\n\[b] This is a blocker".footertable width="100%"trtd.aligncenter.content-blocktd

Lastly, we will need to add an extra column to the Standups table to track the Message-ID coming from the Mailgun routed emails. As you can not count on an email service to provide "just once" delivery, we will need to track these unique IDs ourselves on the Standup table.

rails g migration AddMessageIdToStandups message_id

Next, before the end of the newly created migrations change method, you will want to add add_index :standups, :message_id. This index will allow quick lookups as the Standups table grows. Finally, migrate the actual change:

bin/rails db:migrate

With those changes out of the way we can now add the new EmailProcessor class that will parse the incoming email:

class EmailProcessor

attr_reader :email

def initialize(email)@email = emailend

TASK_TYPE_HASH = {'[d]' => 'Did','[t]' => 'Todo','[b]' => 'Blocker'}

def processif Rails.env.development?Rails.logger.info '-----------EMAIL-------------'Rails.logger.info email.to.first[:token]Rails.logger.info email.bodyRails.logger.info email.headers["Message-ID"]Rails.logger.info '-----------EMAIL-------------'end

# Get a user hash_id from reploy-to or bailreply_user = email.to.first[:token]&.split('<')&.last&.split('@')&.first&.split('.')&.lastreturn if reply_user.blank?

# Find a user by the hash_id or bailuser = User.find_by(hash_id: reply_user)return if user.nil?

# Bail if standup with incoming message-id existsreturn if Standup.exists?(message_id: email.headers["Message-ID"])

# Bail if a standup for today existstoday = Date.today.iso8601return if Standup.exists?(standup_date: today)

# Get content or bailtasks_from_body = email.body.scan(/(\[[dtb]{1}\].*)$/)return if tasks_from_body.blank? || tasks_from_body.empty?

build_and_create_standup(user: user,tasks: tasks_from_body,date: today,message_id: email.headers["Message-ID"])end

private

def build_and_create_standup(user:, tasks:, date:, message_id:)standup = Standup.new(user_id: user.id,standup_date: date,message_id: message_id)

tasks.each do |task|task_type, task_body = task.first.scan(/(\[[dtb]\])(.*)$/).flattenstandup.tasks << Task.new(type: TASK_TYPE_HASH[task_type], title: task_body)end

standup.saveendend

The class is relatively simple, but let’s go over it section by section:

class EmailProcessor

attr_reader :email

def initialize(email)@email = emailend

TASK_TYPE_HASH = {'[d]' => 'Did','[t]' => 'Todo','[b]' => 'Blocker'}

...

Here we are initializing the object when the class is called by Griddler, setting the email to a local email variable. Additionally, we are creating a hash to later use in the text content to Task type conversion.

...

def processif Rails.env.development?Rails.logger.info '-----------EMAIL-------------'Rails.logger.info email.to.first[:token]Rails.logger.info email.bodyRails.logger.info email.headers["Message-ID"]Rails.logger.info '-----------EMAIL-------------'end

...

This just adds some additional logging if the mail is processed in the local development environment.

...

# Get a user hash_id from reply-to or bailreply_user = email.to.first[:token]&.split('<')&.last&.split('@')&.first&.split('.')&.lastreturn if reply_user.blank?

# Find a user by the hash_id or bailuser = User.find_by(hash_id: reply_user)return if user.nil?

# Bail if standup with incoming message-id existsreturn if Standup.exists?(message_id: email.headers["Message-ID"])

# Bail if a standup for today existstoday = Date.today.iso8601return if Standup.exists?(user_id: user.id, standup_date: today)

# Get content or bailsafe_body = Rails::Html::WhiteListSanitizer.new.sanitize(email.body)tasks_from_body = safe_body.scan(/(\[[dtb]{1}\].*)$/)return if tasks_from_body.blank? || tasks_from_body.empty?

...

Here we are grabbing some information used in the parsing, as well as giving the process method chances to exit early if the incoming email is not sufficient for processing and Standup creation. In the first section, the incoming email address is parsed to find the user's hash_id. That string is then used to find a User. If there is no user, the method returns without adding a Standup.

The next line will exit the method early if there is already a standup with the current Message-ID. Again, this is making sure to guard against email providers not guaranteeing “just once” delivery. That is followed up by generating a variable for the current date and making sure there is no standup with the current user and current time.

Finally, the actual email content is parsed with a regular expression. Regular Expression is a programming language that allows you to pattern match on a string and even capture parts of the pattern matching. This particular pattern(which you can get a more thorough syntax explanation here) searches for lines that begin with [d], [t] or [r]. If those are present, it captures the content to the end of the line. The .scan method on the content's body, allows it to catch all occurrences of the above pattern. If the scan's output is empty, the process method exits.

build_and_create_standup(user: user,tasks: tasks_from_body,date: today,message_id: email.headers["Message-ID"])end

private

def build_and_create_standup(user:, tasks:, date:, message_id:)standup = Standup.new(user_id: user.id,standup_date: date,message_id: message_id)

tasks.each do |task|task_type, task_body = task.first.scan(/(\[[dtb]\])(.*)$/).flattenstandup.tasks << Task.new(type: TASK_TYPE_HASH[task_type], title: task_body)end

standup.saveendend

The last section here is a culmination of all of the stored information so far to be saved into a new Standup. The user, tasks_from_body, today and message_id are all passed into a method that will hand the actual save. The build_and_create_standup method creates a new Standup, with the user's ID, date and message_id. Once the object is created, the tasks strings are iterated over to build the Task with a type and assigned as child objects with the << syntax. Finally the new Standup object with children Tasks will be saved with standup.save

Lastly, you can test this all works if you reply back to an email(that was sent through Mailgun SMTP and not letter_opener) with the following text:

[d] Did a thing[d] And Another[t] Something to do[b] Something in the way. Some really long line about something or another

Testing this will require a new spec file with quite a few it blocks to test all the branches the EmailProcessor may encounter.

First, it would be best if we create a factory to be able to quickly generate an email to be used within the EmailProcessor's spec. This way the email can have defaults and then we can use the FactoryGirl .build commands to create a new email object with any different attributes when needed to test the processor.

The factory itself is pretty simple:

factory :email, class: OpenStruct do# Assumes Griddler.configure.to is :hash (default)to [{full: '[email protected]',email: '[email protected]',token: 'to_user',host: 'email.com',name: nil}]from(token: 'from_user',host: 'email.com',email: '[email protected]',full: 'From User <[email protected]>',name: 'From User')subject 'email subject'body '[d] Did a thing\n[t] Doing a thing\n[b] Blocked by a thing'headers {'Message-ID [email protected]'}endend

Now, with a factory available, the email_processor_spec, will be able to easily spin up new email objects as needed with specific changes to test all of the processors' conditional branches.

require 'rails_helper'

describe EmailProcessor dosubject(:email_processor) { EmailProcessor }let(:user) { FactoryGirl.create(:user) }let(:email) doFactoryGirl.build(:email,to: [{email: "@app.buildasaasappinrails.com">standup.#{user.hash_id}@app.buildasaasappinrails.com",token: "@app.buildasaasappinrails.com">standup.#{user.hash_id}@app.buildasaasappinrails.com"}])end

describe 'processes incoming email' do

it 'works as intended' doexpect { email_processor.new(email).process }.to change(Standup, :count).by(1)end

it 'fails on bad to' dobad_to = FactoryGirl.build(:email,to: [{ token: nil, email: '[email protected]' }])expect { email_processor.new(bad_to).process }.to change(Standup, :count).by(0)end

it 'fails on no user' dobad_to = FactoryGirl.build(:email,to: [{token: '[email protected]',email: '[email protected]'}])expect { email_processor.new(bad_to).process }.to change(Standup, :count).by(0)end

it 'only saves one per message-id' doexpect doemail_processor.new(email).processemail_processor.new(email).processend.to change(Standup, :count).by(1)end

it 'only saves one per date' doemail2 = FactoryGirl.build(:email, headers: { 'message-id': '123' })expect doemail_processor.new(email).processemail_processor.new(email2).processend.to change(Standup, :count).by(1)end

it 'fails on empty or bad body' doemail = FactoryGirl.build(:email, body: '90ioqwhdk.qhdu')email2 = FactoryGirl.build(:email, body: '')expect doemail_processor.new(email).processemail_processor.new(email2).processend.to change(Standup, :count).by(0)endendend

While long and containing six examples, this spec is actually pretty straightforward. It is first testing the happy path where everything is set up and working. Then tests each failing path that doesn’t create a standup, in order as those paths appear in the EmailProcessor's .process method.

A quick run of the whole rspec suite should show no failing tests and nearly perfect test/code coverage:

rspec spec ........................................................................................................................................................

Finished in 25.78 seconds (files took 10.16 seconds to load) 152 examples, 0 failures Coverage report generated for RSpec to standup_app/coverage. 428 / 431 LOC (99.3%) covered.