276 reads

How To Add Action Mailbox To A Rails 6 Application

by Rob RaceAugust 17th, 2020

Too Long; Didn't Read

The following is an excerpt from my book Build A SaaS App in Rails 6. The book is used in a new product I am building Mail Buffer. This article is based on an excerpt of this book's first chapter. This section covers sending and receiving mail from an application. We will use Sendgrid's email routing service and ngrok, an HTTP tunneling service. We need to create and store a password for the incoming mail and use a new feature called 'Rails Credentials'

Company Mentioned

featured image - How To Add Action Mailbox To A Rails 6 Application

Sending mail from a Rails application has been covered by hundreds or thousands of articles, however, there is not a ton of articles about receiving, parsing and using the new

ActionMailbox

The following is an excerpt from my book Build A SaaS App in Rails 6 and is used in a new product I am building Mail Buffer. Thus, I am standing by the tutorial here with real world usage!

Lastly, as this is an excerpt, this takes into account the chapter leading up to this section where the reader will have created an application, models for a

Standup

Todo

Did

Blockers

(which are

Tasks

) and a few mailers for outgoing mail.

Here goes nothing...

Another aspect of email within a SaaS application is receiving mail. While this is far less normal or used in comparison to sending, it can be a great way to make end user's responses to email or action items quicker.

At a high level, there are a few different layers to this. The topmost layer is the email service, and this book's case, Sendgrid. This service handles sending outbound emails, as well as routing incoming emails to an address/domain name specified in their interface. Once the email is routed, it will be redirected to a route and processor file in your application. This file will be responsible for parsing the incoming email address and using logic to decide what to do with it.

In the case of the Standup App we are building, we can have directions to respond to an Email Reminder to create a new standup right from their email response! Some of the tools that we will be using are Sendgrid's email routing service and ngrok, an HTTP tunneling service. `ngrok` is very useful for a few solutions throughout the remainder of this book.

It allows you to have a web-accessible URL, which tunnels(connects) to ports within your local machine. Meaning, that you will have a

http://somesubdomain.ngrok.com

that will forward to your computer and a specific port specified when you start

ngrok

This allows you to test with external services such as Sendgrid, Stripe(later), Github(later), and more!

Let's get started with some setup:

1. Download and use ngrok

Download ngrok
Unzip and move the executable file to where you would like.
In \*nix OS's, open ngrok with:
```
path/to/ngrok HTTP start 3000
```
. This will be dependent on the port you are using for your Rails server. ngrok will now fire up a tunnel service with a randomly generated URL.

Add

config.hosts << "yoursubdomain.ngrok.io"

config/environments/development.rb

Optionally, if you upgrade to a paid version of
```
ngrok
```
, you can set a subdomain, so you do not have to change your settings elsewhere every time you restart
```
ngrok
```
.

2. Follow your domain's DNS provider's instructions for adding an MX record for the subdomain or domain name you have chosen to use for inbound email. The MX record will be

mx.sendgrid.net

, with a priority of

Now, before we head to the next part, we need to create and store a password for the incoming mail. You can use any method you like to create a password, and if you just can't think of any way to do so, you can run

SecureRandom.alphanumeric

in Rails console.

Now, to store this password you just came up with, we will need to introduce a new small, but powerful, feature,

Rails Credentials

. Since Rails 5.2, Rails provides a built-in way for storing secure credentials, that are encrypted.

To start using Rails Credentials, you will want to fire off the edit command:

EDITOR="atom --wait" rails credentials:edit

The editor I am using in this example is Atom. However, you can use any editor you like, and it's command line caller. Also,

--wait

is used to make sure that the editor will wait for Rails to finish decrypting the file before opening it in the editor.

Now that we have the encrypted file open, add the following lines:

action_mailbox:
  ingress_password: *password you just made*

Save the file and close the tab/editor to make sure Rails saves the file and encrypts it. You will notice when Rails first opened the file, there was some output text in your terminal. Which includes the master key, DO NOT LOSE THIS KEY.

It will need to be used a the only text in a

config/master.key

to be able to encrypt and decrypt the credentials.

Now, with that out of the way, there is a little bit more setup we need to do for Action Mailbox.

First, two commands to get Rails and some tables set up for inbound email:

bin/rails action_mailbox:install
bin/rails db:migrate

Lastly, add the following line to the

config/application.rb

file to let Rails know we're using Sendgrid:

config.action_mailbox.ingress = :sendgrid

Ok, back to Sendgrid's setup:

In Sendgrid, go to the Inbound Parse link, under settings, to create the email route that will send the email. Enter the following settings:

Enter
```
app.yourdomain.com
```
for the domain field. You can optionally choose an additional subdomain to receive emails within the Inbound Parse.

Enter

http://actionmailbox:[email protected]/rails/action_mailbox/sendgrid/inbound_emails

as the destination URL.

Check
```
POST the raw, full MIME message
```
, before pressing the
```
submit
```
button.

Ok, we're getting nearly done with the setup here. Next, we will need to set up the applications inbound email route and run a command to generate that mailbox class.

First, in the

application_mailbox.rb

file that was created just a few commands ago, you will add

routing /standups\.[0-9a-fA-F]{8}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{12}@/i => :standups

This says when an incoming mail matches that email address syntax, it will be routed to the

standups

mailbox class for processing.

After that, to create the mailbox file, you will just need to run a Rails generator:

bin/rails generate mailbox forwards

A few changes will be needed to allow Action Mailbox and its files to capture text from incoming replies.

First, we will update the

EmailReminderMailer

to create a unique reply-to address and include that email address as part of the outgoing email:

class EmailReminderMailer < ApplicationMailer
  def reminder_email(user, team)
    @user = user
    @team = team
    reply_to = "'Standups App' <standups.#{@user.id}@app.yourdomain.com>"
    make_bootstrap_mail(
      to: @user.email,
      subject: "#{team.name} Standup Reminder!",
      reply_to: reply_to
    )
  end
end

Here we are adding the

reply_to

string to so when someone replies to a reminder email, the reply will be routed through Sendgrid, to Action Mailbox, and then finally by the Standup Mailbox class.

Next, we will update the mailer template to have

##- Please type your reply above this line -##

and some text letting the email recipient know they can add a standup by replying:

<div class="container">
  <span class="text-secondary text-center" style="font-size:8px">##- Please type your reply above this line -##</span>
  <h1 class="text-center mb-4">
    Standup App
  </h1>
  <div class="card mb-4">
    <div class="card-body">
      <h3 class="text-center mb-4"><%= @team.name%> Reminder! </h3>
      <p class="mb-4">
        Just wanted to remind you to add your standup for
        the team: <%= @team.name %>
      </p>
      <%= link_to "Add Your Standup", new_standup_url(), {class: "btn btn-primary btn-lg mx-auto mt-2", style: "width:95%"} %>
    </div>
  </div>
</div>

Lastly, we will need to add an extra column to the

Standups

table to track the

Message-ID

coming from the Mailgun routed emails. As you can not count on an email service to provide "just once" delivery, we will need to track these unique IDs ourselves on the Standup table.

rails g migration AddMessageIdToStandups message_id

Next, before the end of the newly created migrations

change

method, you will want to add

add_index :standups, :message_id

. This index will allow quick lookups as the

Standups

table grows.

Finally, migrate the actual change:

rails db:migrate

Lastly, to make sure sidekiq picks up the queues that Rails uses for Action Mailbox, we will need to adjust the

worker

in the

Procfile.dev

to:

worker: bundle exec sidekiq -q default -q mailers -q action_mailbox_routing -q active_storage_analysis

With those changes out of the way we can now add the new

StandupsMailbox

class that will parse the incoming email:

class StandupsMailbox < ApplicationMailbox
  TASK_TYPE_HASH = {
    '[d]' => 'Did',
    '[t]' => 'Todo',
    '[b]' => 'Blocker'
  }

  def process
    # Get a user id from reply-to or bail
    reply_user = mail.to.first&.split('<')&.last&.split('@')&.first&.split('.')&.last
    return if reply_user.blank?

    # Find a user by the id or bail
    user = User.find_by(id: reply_user)
    return if user.nil?

    # Bail if standup with incoming message-id exists
    return if Standup.exists?(message_id: inbound_email.message_id)

    # Bail if a standup for today exists
    today = Date.today.iso8601
    return if Standup.exists?(standup_date: today)

    # Get content or bail
    safe_body = Rails::Html::WhiteListSanitizer.new.sanitize(mail_body)
    tasks_from_body = safe_body.scan(/(\[[dtb]{1}\].*)$/)
    return if tasks_from_body.blank? || tasks_from_body.empty?

    build_and_create_standup(
      user: user,
      tasks: tasks_from_body,
      date: today,
      message_id: inbound_email.message_id
    )
  end

  private

  def build_and_create_standup(user:, tasks:, date:, message_id:)
    standup = Standup.new(
      user_id: user.id,
      standup_date: date,
      message_id: message_id
    )

    tasks.each do |task|
      task_type, task_body = task.first.scan(/(\[[dtb]\])(.*)$/).flatten
      standup.tasks << Task.new(type: TASK_TYPE_HASH[task_type], title: task_body)
    end

    standup.save
  end

  def mail_body
    @mail_body ||= begin
      if mail.multipart?
        mail.parts[0].body.decoded
      else
        mail.decoded
      end
    end
  end
end

The class is relatively simple, but let's go over it section by section:

ruby
class StandupsMailbox < ApplicationMailbox
  TASK_TYPE_HASH = {
    '[d]' => 'Did',
    '[t]' => 'Todo',
    '[b]' => 'Blocker'
  }

  ...

Here we are setting up the

StandupsMailbox

to inherit from the

ApplicationMailbox

class, which handles the routing. Additionally, we are creating a hash to later use in the text content to

Task

type conversion.

...

def process
  # Get a user id from reply-to or bail
  reply_user = mail.to.first&.split('<')&.last&.split('@')&.first&.split('.')&.last
  return if reply_user.blank?

  # Find a user by the id or bail
  user = User.find_by(id: reply_user)
  return if user.nil?

  # Bail if standup with incoming message-id exists
  return if Standup.exists?(message_id: inbound_email.message_id)

  # Bail if a standup for today exists
  today = Date.today.iso8601
  return if Standup.exists?(standup_date: today)

  # Get content or bail
  safe_body = Rails::Html::WhiteListSanitizer.new.sanitize(mail_body)
  tasks_from_body = safe_body.scan(/(\[[dtb]{1}\].*)$/)
  return if tasks_from_body.blank? || tasks_from_body.empty?

...

Here we are grabbing some information used in the parsing, as well as giving the

process

method chances to exit early if the incoming email is not sufficient for processing and

Standup

creation. In the first section, the incoming email address is parsed to find the user's

id

. That string is then used to find a User. If there is no user, the method returns without adding a Standup.

The next line will exit the method early if there is already a standup with the current Message-ID. Again, this is making sure to guard against email providers, not guaranteeing "just once" delivery. That is followed up by generating a variable for the current date and making sure there is no standup with the

current_user

and

current_date

To get the mail's content, we will want to do two things, make sure we get the correct body from the mail object as email clients can send both an HTML and plain text version.

Then we will want to make sure we only grab the content above the

##- Please type your reply above this line -##

string by using the

split

method (which splits a string into as many parts based on the separator specified in the argument):

def mail_body
    @mail_body ||= begin
      body = if mail.multipart?
        mail.parts[0].body.decoded
      else
        mail.decoded
      end
      body.split('##- Please type your reply above this line -##').first
    end
  end

Finally, the actual email content is parsed with a regular expression. Regular Expression is a programming language that allows you to pattern match on a string and even capture parts of the pattern matching. This particular pattern(which you can get a more thorough syntax explanation here searches for lines that begin with

[d]

[t]

[r]

. If those are present, it captures the content to the end of the line.

The

.scan

method on the content's body allows it to catch all occurrences of the above pattern. If the scan's output is empty, the

process

method exits.

 build_and_create_standup(
      user: user,
      tasks: tasks_from_body,
      date: today,
      message_id: email.headers["Message-ID"]
    )
  end

  private

  def build_and_create_standup(user:, tasks:, date:, message_id:)
    standup = Standup.new(
      user_id: user.id,
      standup_date: date,
      message_id: message_id
    )

    tasks.each do |task|
      task_type, task_body = task.first.scan(/(\[[dtb]\])(.*)$/).flatten
      standup.tasks << Task.new(type: TASK_TYPE_HASH[task_type], title: task_body)
    end

    standup.save
  end
end

The last section here is a culmination of all of the stored information so far to be saved into a new

Standup

. The

user

tasks_from_body

today

, and

message_id

are all passed into a method that will hand the actual save. The

build_and_create_standup

method creates a new

Standup

, with the user's ID, date, and

message_id

Once the object is created, the task's strings are iterated over to build the

Task

with a type and assigned as child objects with the

<<

syntax. Finally the new

Standup

object with children

Tasks

will be saved with

standup.save

Lastly, you can test this all works if you reply back to an email(that was sent through Sendgrid SMTP and not letter_opener) with the following text:

[d] Did a thing
[d] And Another
[t] Something to do
[b] Something in the way. Some really long line about something or another

Testing this will require a new spec file with a few

it

blocks to test all the branches the

StandupMailbox

may encounter.

First, we will need to add one small change to the

rails_helper.rb

file to make sure RSpec has access to the Rails Action Mailbox test helpers.

I'll add the new lines and a few lines above each, so you know where to put the new stuff:

...

require 'support/system_macros'
include Warden::Test::Helpers
require 'action_mailbox/test_helper' # <-- new line

...

RSpec.configure do |config|
  config.include Devise::Test::ControllerHelpers, type: :controller
  config.include Devise::Test::ControllerHelpers, type: :view
  config.include ActionMailbox::TestHelper, type: :mailbox # <-- new line

...

...and now the actual spec file itself:

require 'rails_helper'
include ActiveJob::TestHelper

RSpec.describe StandupsMailbox, type: :mailbox do
  let(:user) { FactoryBot.create(:user) }
  let(:to_email) { "standups.#{user.id}@app.buildasaasappinrails.com" }
  let(:body) do
    %Q(
      [d] Did a thing
      [d] And Another
      [t] Something to do
      [b] Something in the way. Some really long line about something or another

      ##- Please type your reply above this line -##
    )
  end

  subject do
    receive_inbound_email_from_mail(
      from: user.email,
      to: to_email,
      subject: 'Re: Reminder',
      body: body
    )
  end

  before do
    ActiveJob::Base.queue_adapter = :test
  end

  it 'saves standup record' do
    expect { subject }.to change(Standup, :count).by(1)
  end

  it 'saves task records' do
    expect { subject }.to change(Task, :count).by(4)
  end

  context 'fails on bad to email' do
    let(:to_email) {"[email protected]"}

    it 'does not save standup record' do
      expect { subject rescue nil }.to_not change(Standup, :count)
    end
  end

  context 'fails on no user' do
    let(:to_email) {"standups.9a859af8-30ca-4473-b073-47105352d936@app.buildasaasappinrails.com"}

    it 'does not save standup record' do
      expect { subject rescue nil }.to_not change(Standup, :count)
    end
  end

  context 'fails on useless body' do
    let(:body) { 'asdj;oisaduaskd' }

    it 'does not save standup record' do
      expect { subject rescue nil }.to_not change(Standup, :count)
    end
  end

  context 'fails on empty body' do
    let(:body) { '' }

    it 'does not save standup record' do
      expect { subject rescue nil }.to_not change(Standup, :count)
    end
  end
end

While long and containing six examples, this spec is actually pretty straightforward. It is first testing the happy path where everything is set up and working.

Then checks each failing path that doesn't create a standup, in order as those paths appear in the