With Griddler and Mailgun Another aspect of email within a SaaS application is receiving mail. While this is far less normal or used in comparison to sending, it can be a great way to make end user’s responses to email or action items quicker. At a high level, there are a few different layers to this. The topmost layer is the email service, and this book’s case, Mailgun. This service handles sending outbound emails, as well as routing incoming emails to an address/domain name specified in their interface. Once the email is routed, it will be redirected to a route and processor file in your application. This file will be responsible for parsing the incoming email address and using logic to decide what to do with it. Note: This article is an excerpt derived from a chapter in my upcoming book . The book guides you from humble beginnings through deploying an app to production. The book is now in pre sale, and you can grab a free chapter right now! Build A SaaS App in Rails 6 Also, the beta for my new project is has been released. If you’re losing track of pull requests, have old ones lingering around or would just love a dashboard that aggregates these requests over multiple services (Github, Gitlab, and Bitbucket), . Pull Manager check it out In the case of the Standup App we are building, we can have directions to respond to an Email Reminder to create a new standup right from their email response! Some of the tools that we will be using are Mailgun’s email routing service and ngrok, an HTTP tunneling service. is very useful for a few solution throughout the remainder of this book. It allows you to have a web accessible URL, which tunnels(connects) to ports within your local machine. Meaning, that you will have a that will forward to your computer, and a specific port specified when you start . This allows you to test with external services such as Mailgun, Stripe(later), Github(later) and more! ngrok http://somesubdomain.ngrok.com ngrok Let’s get started with some setup: Download ngrok Unzip and move the executable file to where you would like. In *nix OS’s, open ngrok with: . This will be dependent of the port you are using for your Rails server. ngrok will now fire up a tunnel service with a randomly generated URL. path/to/ngrok HTTP start 3000 Optionally, if you upgrade to a paid version of , you can set a subdomain, so you do not have to change your settings elsewhere every time you restart . ngrok ngrok In Mailgun go to the Routes tab to create the email route(modeled after MVC routes like Rails) that will send the email. Enter the following settings: Chose Match Recipient Enter for the recipient field. Mailgun routes allow wildcard character matching, which will allow you input extra characters in the email's . Meaning, adding something like a user's to have identifiable information from the incoming email. development.standup.*@app.yourdomain.com reply-to hash_id Check and enter as the destination of that forward. forward http://yoursubdomain.ngrok.io/email_processor You can leave the priority alone and give the route a name, before pressing the button. submit Now with that done, we can begin to modify the application. The application changes will consist of three main parts. First, a new gem(and a companion adapter gem) will be added. is the main library to handle incoming email easily; is the Mailgun specific adapter that allows the functionality to work with Mailgun as an incoming mail router. Next, Griddler will need to add a few parts of configuration application-wide to make sure some basic configuration is met. Lastly, an email processor file will be added to handle recieving and parsing the email. griddler griddler-mailgun griddler The great part about adding Griddler to your current application is that if you are using the Gemfile from Chapter 3, you already have it installed. If not, just add and to your Gemfile and run a . gem 'griddler' gem 'griddler-mailgun' bundle install Next to configure and setup Griddler in the application you will need to add a new file in the initializer folder setting a few configuration values. Then, add a quick line to add the default Griddler routing into the file. routes.rb First the Griddler configuration: Griddler.configure do |config|config.reply_delimiter = '-- REPLY ABOVE THIS LINE --'config.email_service = :mailgunend This will set the text the griddler library will look for in the email and tell it that it will use the installed Mailgun adapter. Next, add a line to mount the library based routes into your application. Adding the line right abobve the is fine: root to: Rails.application.routes.draw do... mount using default path: /email_processor mount_griddler root to: 'activity#mine'end By mounting Griddler with that syntax, it will automatically add a route to your application that will route from a specified endpoint to a Griddler based controller: email_processor POST /email_processor(.:format) griddler/emails#create There are settings in Griddler’s GitHub documentation to change the defaults, but unless you want to get creative with route paths or email processor class names, it can be unnecessary. The default settings expect a class to exist and to handle parsing the incoming email with a method . Griddler, however, does not care where the actual file is placed, but that the class exists and is loaded. Personally, I find that email processing fits most into the services definition and can be placed there. EmailProcessor process To allow Griddler and its files to capture text from incoming replies, there are a few changes that will be needed. First we will update the to create a unique reply-to address and include that email address as part of the outgoing email: EmailReminderMailer class EmailReminderMailer < ApplicationMailerdef reminder_email(user, team) = user = teamreply_to = "'Standup App' <#{'development.' if Rails.env.development?}standup.#{ .yourdomain.com">user.hash_id} .yourdomain.com>"mail(to: .email,subject: "#{team.name} Standup Reminder!",reply_to: reply_to)endend @user @team @app @app @user Here we are using building the string by adding if the current Rails environment is your local Rails application. This way, the separate Mailgun route made for development can route different from a production route you will add before deploying your application. reply_to development Next we will updated the mailer template to have and some text letting the email recipient know they can add a standup by replying: ##- Please type your reply above this line -## ## app/views/email_reminder_mailer/reminder_email.html.slimdoctype htmlhtml xmlns=" "headmeta content="width=device-width" name="viewport" /meta content=("text/html; charset=UTF-8") http-equiv="Content-Type" /title= "#{ .name} Reminder!"css:| *{margin:0;padding:0;font-family:"Open Sans",Helvetica,Helvetica,Arial,sans-serif;box-sizing:border-box;font-size:14px}img{max-width:100%}body{-webkit-font-smoothing:antialiased;-webkit-text-size-adjust:none;width:100%!important;height:100%;line-height:1.6}table td{vertical-align:top}body{background-color:#f6f6f6}.body-wrap{background-color:#f6f6f6;width:100%}.container{display:block!important;max-width:800px!important;margin:0auto!important;clear:both!important}.content{max-width:800px;margin:0auto;display:block;padding:20px}.main{background:#fff;border:1px solid #e9e9e9;border-radius:3px}.content-wrap{padding:20px}.content-block{padding:0 0 20px}.header{width:100%;margin-bottom:20px}.footer{width:100%;clear:both;color:#999;padding:20px}.footer a{color:#999}.footer a,.footerp,.footer td,.footer unsubscribe{font-size:12px}h1,h2,h3,a,th,td{font-family:"Open Sans",Helvetica,Arial,"Lucida Grande",sans-serif;color:#000;margin:40px 0 0;line-height:1.2;font-weight:400}h1{font-size:32px;font-weight:500}h2{font-size:24px}h3{font-size:18px}h4{font-size:14px;font-weight:600}ol,p,ul{margin-bottom:10px;font-weight:400}ol li,p li,ulli{margin-left:5px;list-style-position:inside}a{color:##3c8dbc;text-decoration:underline}.btn-primary{text-decoration:none;color:#FFF;background-color:##3c8dbc;border:solid ##3c8dbc;border-width:5px 10px;line-height:2;font-weight:700;text-align:center;cursor:pointer;display:inline-block;border-radius:5px;text-transform:capitalize}.last{margin-bottom:0}.first{margin-top:0}.aligncenter{text-align:center}.alignright{text-align:right}.alignleft{text-align:left}.clear{clear:both}.alert{font-size:16px;color:#fff;font-weight:500;padding:20px;text-align:center;border-radius:3px 3px0 0}.alert a{color:#fff;text-decoration:none;font-weight:500;font-size:16px}.alert.alert-warning{background:#f8ac59}.alert.alert-bad{background:#ed5565}.alert.alert-good{background:##3c8dbc}.invoice{margin:40pxauto;text-align:left;width:80%}.invoice td{padding:5px 0}.invoice .invoice-items{width:100%}.invoice .invoice-items td{border-top:#eee 1pxsolid}.invoice .invoice-items .total td{border-top:2px solid #333;border-bottom:2px solid #333;font-weight:700} only screen and (max-width:640px){h1,h2,h3,h4{font-weight:600!important;margin:20px 0 5px!important}h1{font-size:22px!important}h2{font-size:18px!important}h3{font-size:16px!important}.container{width:100% !important}.content,.content-wrap{padding:10px !important}.invoice{width:100% !important}bodydiv style="color: #b5b5b5;text-align:center;"| ##- Please type your reply above this line -##table.body-wrap style="width:100%"trtdtd.container width="800".contenttable.main cellpadding="0" cellspacing="0" width="100%"trtd.content-wraptable cellpadding="0" cellspacing="0" style="width:100%"trtd.aligncenter| Standup Apptrtd.content-blockh3= "#{ .name} Reminder!"trtd.content-block= "Just wanted to remind you to add your standup for \the team: #{ .name}"trtd.content-block.aligncenter= link_to "Add Your Standup", new_standup_url(), \{class:"btn-primary", style: "width:95%"}trtd.content-block= "You can quickly submit your standup by replying to \this email in the format:"prepre= "[d] This is a done item\n[t] This is a todo item\n\[b] This is a blocker".footertable width="100%"trtd.aligncenter.content-blocktd http://www.w3.org/1999/xhtml @team @media @team @team Lastly, we will need to add an extra column to the table to track the coming from the Mailgun routed emails. As you can not count on an email service to provide "just once" delivery, we will need to track these unique IDs ourselves on the Standup table. Standups Message-ID rails g migration AddMessageIdToStandups message_id Next, before the end of the newly created migrations method, you will want to add . This index will allow quick lookups as the table grows. Finally, migrate the actual change: change add_index :standups, :message_id Standups bin/rails db:migrate With those changes out of the way we can now add the new class that will parse the incoming email: EmailProcessor class EmailProcessor attr_reader :email def initialize(email) = emailend @email TASK_TYPE_HASH = {'[d]' => 'Did','[t]' => 'Todo','[b]' => 'Blocker'} def processif Rails.env.development?Rails.logger.info '-----------EMAIL-------------'Rails.logger.info email.to.first[:token]Rails.logger.info email.bodyRails.logger.info email.headers["Message-ID"]Rails.logger.info '-----------EMAIL-------------'end # Get a user hash_id from reploy-to or bailreply_user = email.to.first[:token]&.split('<')&.last&.split('@')&.first&.split('.')&.lastreturn if reply_user.blank? # Find a user by the hash_id or bailuser = User.find_by(hash_id: reply_user)return if user.nil? # Bail if standup with incoming message-id existsreturn if Standup.exists?(message_id: email.headers["Message-ID"]) # Bail if a standup for today existstoday = Date.today.iso8601return if Standup.exists?(standup_date: today) # Get content or bailtasks_from_body = email.body.scan(/(\[[dtb]{1}\].*)$/)return if tasks_from_body.blank? || tasks_from_body.empty? build_and_create_standup(user: user,tasks: tasks_from_body,date: today,message_id: email.headers["Message-ID"])end private def build_and_create_standup(user:, tasks:, date:, message_id:)standup = Standup.new(user_id: user.id,standup_date: date,message_id: message_id) tasks.each do |task|task_type, task_body = task.first.scan(/(\[[dtb]\])(.*)$/).flattenstandup.tasks << Task.new(type: TASK_TYPE_HASH[task_type], title: task_body)end standup.saveendend The class is relatively simple, but let’s go over it section by section: class EmailProcessor attr_reader :email def initialize(email) = emailend @email TASK_TYPE_HASH = {'[d]' => 'Did','[t]' => 'Todo','[b]' => 'Blocker'} ... Here we are initializing the object when the class is called by Griddler, setting the email to a local email variable. Additionally, we are creating a hash to later use in the text content to type conversion. Task ... def processif Rails.env.development?Rails.logger.info '-----------EMAIL-------------'Rails.logger.info email.to.first[:token]Rails.logger.info email.bodyRails.logger.info email.headers["Message-ID"]Rails.logger.info '-----------EMAIL-------------'end ... This just adds some additional logging if the mail is processed in the local development environment. ... # Get a user hash_id from reply-to or bailreply_user = email.to.first[:token]&.split('<')&.last&.split('@')&.first&.split('.')&.lastreturn if reply_user.blank? # Find a user by the hash_id or bailuser = User.find_by(hash_id: reply_user)return if user.nil? # Bail if standup with incoming message-id existsreturn if Standup.exists?(message_id: email.headers["Message-ID"]) # Bail if a standup for today existstoday = Date.today.iso8601return if Standup.exists?(user_id: user.id, standup_date: today) # Get content or bailsafe_body = Rails::Html::WhiteListSanitizer.new.sanitize(email.body)tasks_from_body = safe_body.scan(/(\[[dtb]{1}\].*)$/)return if tasks_from_body.blank? || tasks_from_body.empty? ... Here we are grabbing some information used in the parsing, as well as giving the method chances to exit early if the incoming email is not sufficient for processing and creation. In the first section, the incoming email address is parsed to find the user's . That string is then used to find a User. If there is no user, the method returns without adding a Standup. process Standup hash_id The next line will exit the method early if there is already a standup with the current Message-ID. Again, this is making sure to guard against email providers not guaranteeing “just once” delivery. That is followed up by generating a variable for the current date and making sure there is no standup with the current user and current time. Finally, the actual email content is parsed with a regular expression. Regular Expression is a programming language that allows you to pattern match on a string and even capture parts of the pattern matching. This particular pattern(which you can get a more thorough syntax explanation ) searches for lines that begin with , or . If those are present, it captures the content to the end of the line. The method on the content's body, allows it to catch all occurrences of the above pattern. If the scan's output is empty, the method exits. here [d] [t] [r] .scan process build_and_create_standup(user: user,tasks: tasks_from_body,date: today,message_id: email.headers["Message-ID"])end private def build_and_create_standup(user:, tasks:, date:, message_id:)standup = Standup.new(user_id: user.id,standup_date: date,message_id: message_id) tasks.each do |task|task_type, task_body = task.first.scan(/(\[[dtb]\])(.*)$/).flattenstandup.tasks << Task.new(type: TASK_TYPE_HASH[task_type], title: task_body)end standup.saveendend The last section here is a culmination of all of the stored information so far to be saved into a new . The , , and are all passed into a method that will hand the actual save. The method creates a new , with the user's ID, date and . Once the object is created, the tasks strings are iterated over to build the with a type and assigned as child objects with the syntax. Finally the new object with children will be saved with Standup user tasks_from_body today message_id build_and_create_standup Standup message_id Task << Standup Tasks standup.save Lastly, you can test this all works if you reply back to an email(that was sent through Mailgun SMTP and not letter_opener) with the following text: [d] Did a thing[d] And Another[t] Something to do[b] Something in the way. Some really long line about something or another Testing this will require a new spec file with quite a few blocks to test all the branches the may encounter. it EmailProcessor First, it would be best if we create a factory to be able to quickly generate an email to be used within the 's spec. This way the email can have defaults and then we can use the commands to create a new email object with any different attributes when needed to test the processor. EmailProcessor FactoryGirl .build The factory itself is pretty simple: factory :email, class: OpenStruct do# Assumes Griddler.configure.to is :hash (default)to [{full: ' ',email: ' ',token: 'to_user',host: 'email.com',name: nil}]from(token: 'from_user',host: 'email.com',email: ' ',full: 'From User < >',name: 'From User')subject 'email subject'body '[d] Did a thing\n[t] Doing a thing\n[b] Blocked by a thing'headers {'Message-ID '}endend to_user@email.com to_user@email.com from_email@email.com from_user@email.com 98984d@local.mail Now, with a factory available, the , will be able to easily spin up new email objects as needed with specific changes to test all of the processors' conditional branches. email_processor_spec require 'rails_helper' describe EmailProcessor dosubject(:email_processor) { EmailProcessor }let(:user) { FactoryGirl.create(:user) }let(:email) doFactoryGirl.build(:email,to: [{email: " .buildasaasappinrails.com">standup.#{user.hash_id} .buildasaasappinrails.com",token: " .buildasaasappinrails.com">standup.#{user.hash_id} .buildasaasappinrails.com"}])end @app @app @app @app describe 'processes incoming email' do it 'works as intended' doexpect { email_processor.new(email).process }.to change(Standup, :count).by(1)end it 'fails on bad to' dobad_to = FactoryGirl.build(:email,to: [{ token: nil, email: ' ' }])expect { email_processor.new(bad_to).process }.to change(Standup, :count).by(0)end standup@app.buildasaasappinrails.com it 'fails on no user' dobad_to = FactoryGirl.build(:email,to: [{token: ' ',email: ' '}])expect { email_processor.new(bad_to).process }.to change(Standup, :count).by(0)end standup.o8yhiukj@app.buildasaasappinrails.com standup.o8yhiukj@app.buildasaasappinrails.com it 'only saves one per message-id' doexpect doemail_processor.new(email).processemail_processor.new(email).processend.to change(Standup, :count).by(1)end it 'only saves one per date' doemail2 = FactoryGirl.build(:email, headers: { 'message-id': '123' })expect doemail_processor.new(email).processemail_processor.new(email2).processend.to change(Standup, :count).by(1)end it 'fails on empty or bad body' doemail = FactoryGirl.build(:email, body: '90ioqwhdk.qhdu')email2 = FactoryGirl.build(:email, body: '')expect doemail_processor.new(email).processemail_processor.new(email2).processend.to change(Standup, :count).by(0)endendend While long and containing six examples, this spec is actually pretty straightforward. It is first testing the happy path where everything is set up and working. Then tests each failing path that doesn’t create a standup, in order as those paths appear in the 's method. EmailProcessor .process A quick run of the whole rspec suite should show no failing tests and nearly perfect test/code coverage: rspec spec ........................................................................................................................................................ Finished in 25.78 seconds (files took 10.16 seconds to load) 152 examples, 0 failures Coverage report generated for RSpec to standup_app/coverage. 428 / 431 LOC (99.3%) covered.
