Creating a form forwarding service for AWS Lambda

Since originally publishing this blog post I’ve written an updated article. I’d recommend that you read that instead of continuing with this as some of the content below is now out of date.

Introducing Formplug v1, a form forwarding service for AWS Lambda_It’s estimated that approximately 269 billion emails are sent in a single day. Over 10 million were sent as you read…_hackernoon.com

In this post I’ll be introducing a service I’ve created that makes setting up email form forwarding easy. It’s built in Javascript using Node.js and the Serverless Framework. It deploys to AWS Lambda and uses Amazon Simple Email Service to send the emails. To check out the code see the Github repository below. Keep reading for an explanation of the problem and a walk-through of the technical solution.

danielireson/formplug-serverless_formplug-serverless - Form forwarding service for AWS Lambda_github.com

Problem

Recently I’ve been finding myself taking a preference towards static site hosting for my side projects. Github Pages works well because like most people I use Github to host my git repositories anyway. To deploy a site it’s as easy as a git push to the gh-pages branch and in a few short minutes the website will be live. In many cases static hosting is perfectly fine but on a couple of occasions now I’ve struggled to set up a simple email-based contact form — something which wouldn’t usually be considered too challenging of a task. There’s a couple of ways of handling this.

Option 1: no contact form

The obvious first option is to avoid contact forms altogether and instead just display an email address. This is the option that I’ve taken on many occasions because it’s the easiest but has a few downsides. It pushes extra work onto the users — they have to leave the site and open their email client. It’s also hard to specify what information is required when visitors first contact you, you might require qualifying information for a quote for example. You can have a message asking users for specific information but there’s no validation that can be done to confirm people are adhering to this request.

Option 2: mailto links

One solution for sending forms by email is to set the form’s action attribute to a mailto link set to an email address. When the user submits the form it opens their default email client with the form data as the body, as shown in the screenshot below. The result isn’t a great user experience. Not all users will have a default email client setup, and even when they do they’ll get presented with a draft email. It’s unclear whether the form has be submitted and what actions need to be taken by them.

Option 3: use server-side code

As a security feature you can’t send email client-side in the browser. If this were possible website visitors would be highly susceptible to malicious code sending email on their behalf. If you already host your apps on a server it’s painless to setup a contact form. Server-side scripting languages like PHP, Java or ASP have the capabilities that make it easy to send email. In PHP for example it can be achieved in two lines.

$message = $_POST['message'];mail('[email protected]', 'Form submission', $message);

If you don’t already have a server however, having to maintain one just for a contact form seems a little wasteful. There’s time involved in maintaining it as well as financial costs, and it introduces other considerations around scalability and security.

Option 4: use a third-party service

Managed form backend services exist that range in functionality and scope. FormKeep for example makes it easy to integrate forms into services such as Trello, Slack or Google Sheets, but can be expensive just for a simple contact form with plans start at $59/month. There’s also always a risk involved when utilising a third-party service; it can go offline at any moment’s notice and you have no control of it. If there service was to shutdown there would be no notification, contact forms would just suddenly stop working.

Solution

A self-hosted form forwarding service is the optimal solution in my opinion. You get the benefits of control as well as no end-user experience degradation. Choosing to develop the backend for AWS Lambda further reduces many of the downsides of self-hosting.

AWS Lambda lets you run code without provisioning or managing servers. You pay only for the compute time you consume — there is no charge when your code is not running. With Lambda, you can run code for virtually any type of application or backend service — all with zero administration. Just upload your code and Lambda takes care of everything required to run and scale your code with high availability.

There’s multiple approaches to building applications on AWS Lambda. For simple services you can use the AWS Management Console to create functions and set up triggers. For more complex services it’s better to develop locally and deploy to AWS afterwards, where you can enjoy the familiarities of your favourite code editor. Getting the project structure set up initially can be challenging as there’s lots of things to consider. Which AWS region is the service going to be deployed to? How are AWS services going to be provisioned? How are environment variables going to be managed? How is security going to be managed? Because of these considerations I like to use an opinionated framework called the Serverless Framework when working with AWS Lambda. It provides sensible defaults and allows us to to provision services at a higher and easier to understand abstraction. Using the CLI, deploying and updating a service is as simple as running serverless deploy.

If you’re looking to learn a bit more about the Serverless Framework I’d recommend you watch the following video below by its creator Austin Collins. The video is a couple of years old now but does a great job of introducing some of the core concepts. I’d also recommend going through the Serverless docs.

Architecture overview

The form forwarding service is composed of three lambda functions: encrypt, receive, send. The encrypt and receive functions are triggered through the Amazon API Gateway routes /encrypt and /to respectively. (An API Gateway route is essentially a public URL.) The send lambda function is only ever invoked by the receive lambda function. The receive lambda is responsible for handling the form submission and invoking the send lambda to send the email. The encrypt lambda can optionally be used to get an encrypted representation of an email address to use in the receive endpoint URL. This is explained in more detail later.

Receive lambda

The receive handler is triggered by the /to route which accepts one parameter, the email address to forward the form to. The endpoint is generated on the initial serverless deploy and returned in the terminal window. It takes the following structure.

https://apigatewayurl/to/[email protected]

HTML forms should POST to this endpoint. This will trigger the receive lambda (shown below). It’s responsible for validating the request, getting the form data and invoking the send lambda.

module.exports.handle = (event, context, callback) => {let data = receiveRequest.getParams(event)if (receiveRequest.validate(data, callback)) {eventInvoker.send(data).then(function () {httpRoute.render('receive-success', data, callback)}).catch(function (error) {httpRoute.render('receive-error', data, callback)})}}

Getting the form data

The form’s POST request will generate an API Gateway event which is automatically passed to the handler as the first parameter. An example of an event is shown below (with sensitive fields removed).

{resource:'/to/{_to}',path:'/to/[email protected]',httpMethod:'POST',headers: {Accept:'*/*','Accept-Encoding':'gzip, deflate','cache-control':'no-cache','CloudFront-Forwarded-Proto':'https','CloudFront-Is-Desktop-Viewer':'true','CloudFront-Is-Mobile-Viewer':'false','CloudFront-Is-SmartTV-Viewer':'false','CloudFront-Is-Tablet-Viewer':'false','CloudFront-Viewer-Country':'GB','content-type':'application/x-www-form-urlencoded',Host:'','User-Agent':'','X-Amz-Cf-Id':'','X-Amzn-Trace-Id':'','X-Forwarded-For':'','X-Forwarded-Port':'443','X-Forwarded-Proto':'https'},queryStringParameters:null,pathParameters: {_to:'[email protected]'},stageVariables:null,requestContext: {path:'/dev/to/[email protected]',accountId:'',resourceId:'',stage:'dev',requestId:'',identity:{cognitoIdentityPoolId:null,accountId:null,cognitoIdentityId:null,caller:null,apiKey:'',sourceIp:'',accessKey:null,cognitoAuthenticationType:null,cognitoAuthenticationProvider:null,userArn:null,userAgent:'',user:null},resourcePath:'/to/{_to}',httpMethod:'POST',apiId:''},body:'first=test&second=test&third=test',isBase64Encoded:false}

We’re only interested in some of the event data and so we extract this through the getParams function. Specifically we’re interested in getting the form data from event.body, the email address from event.pathParameters and the query string from event.queryStringParamters.

module.exports.getParams = function (event) {let data = Object.assign({}, querystring.parse(event.body), event.pathParameters, event.queryStringParameters)return data}

The node querystring module is used to parse the POST form data from the request body. This takes the string first=test&second=test&third=test and turns it into a JavaScript object.

Validating the request

The data then goes through a validation function which checks that the email is valid and that the honeypot field hasn’t been filled out.

module.exports.validate = function (data, callback) {return hasValidEmail(data, callback) && hasNoHoneypot(data, callback)}

function hasValidEmail (data, callback) {if (!('_to' in data)) {httpRoute.render('receive-no-email', data, callback)return false}if ('_to' in data && !httpValidation.isEmail(data['_to'])) {httpRoute.render('receive-bad-email', data, callback)return false}return true}

function hasNoHoneypot (data, callback) {if ('_honeypot' in data && data['_honeypot'] !== '') {httpRoute.render('receive-honeypot', data, callback)return false}return true}

The honeypot field is an optional spam prevention feature. Users can add the field to their form and hide it from visitors.

<input type="text" name="_honeypot" style="display:none">

If the form is submitted and the honeypot field isn’t empty the request will be ignored (because a spam bot has tried to enter a value there).

Invoking the send lambda

After validation the send lambda is invoked asynchronously with a payload of the data using the AWS-SDK npm package.

module.exports.send = function (data) {let event = {FunctionName: `formplug-${config.STAGE}-send`,InvocationType: 'Event',Payload: JSON.stringify(data)}return lambda.invoke(event).promise()}

It invokes it asynchronously because we don’t want visitors to be waiting after they’ve submitted the form for Amazon to send the email. Invoking it in this way also means that AWS will automatically retry it if it fails.

Generating a HTTP response

HTTP responses are made through the handler’s callback function. This is provided by the Serverless Framework. For successful responses the callback function should be called with the first parameter as null and the second parameter as the response object. For example, to generate a successful plain text HTTP response we could do the following.

callback(null, {statusCode: 200,headers: {'Content-Type': 'text/plain'},body: 'Form submission successfully made'})

But instead of a plain text response we want either a HTML, JSON or URL redirect response (depending on what was requested by the user). The application defaults to a HTML response.

HTML responses are generated by loading a local template file and replacing a {{ message }} variable with an appropriate message.

function buildHtmlResponse (statusCode, message, data) {let response = {statusCode: statusCode,headers: {'Content-Type': 'text/html'},body: generateView(message)}if (statusCode === 302) {response.headers.Location = data['_redirect']}return response}

function generateView (message) {try {let template = fs.readFileSync(path.resolve(__dirname, 'template.html')).toString()message = template.replace('{{ message }}', message)} catch (error) {utilityLog.error(error.message)}return message}

Users can also request a URL redirect after the form submission. This URL should be supplied as a hidden input in the form itself.

<input type="hidden" name="_redirect" value="http://google.com">

If a _redirect field is supplied then the application will set a 302 HTTP status code and the Location HTTP header. The visitor’s web browser will pick these up and handle the redirection.

Non-redirect response messages are defined in a routes file. Users can define custom messages in their config or use the service defaults. These are called through a route render function.

module.exports.render = function (type, data, callback) {let routeDetails = module.exports.getRouteDetails(type, data)callback(null, httpResponse.build(routeDetails.statusCode, routeDetails.message, data))}

module.exports.getRouteDetails = function (type, data) {let statusCode, messageswitch (type) {case 'encrypt-no-email':statusCode = 422message = config.MSG_ENCRYPT_NO_EMAIL || 'You need to provide an email address to encrypt.'breakcase 'encrypt-bad-email':statusCode = 422message = config.MSG_ENCRYPT_BAD_EMAIL || 'The supplied email address is not valid.'breakcase 'encrypt-success':statusCode = 200message = data['_encrypted']breakcase 'receive-honeypot':statusCode = 422message = config.MSG_RECEIVE_HONEYPOT || 'You shall not pass.'breakcase 'receive-no-email':statusCode = 422message = config.MSG_RECEIVE_NO_EMAIL || 'Form not sent, the admin has not set up a forwarding email address.'breakcase 'receive-bad-email':statusCode = 422message = config.MSG_RECEIVE_BAD_EMAIL || 'Form not sent, the admin email address is not valid.'breakcase 'receive-error':statusCode = 500message = config.MSG_RECEIVE_ERROR || 'Form not sent, an error occurred while sending.'breakcase 'receive-success':statusCode = httpValidation.hasRedirect(data) ? 302 : 200message = config.MSG_RECEIVE_SUCCESS || 'Form submission successfully made.'breakdefault:statusCode = 500message = 'An error has occurred.'}return {statusCode, message}}

Oftentimes the form forwarding service will be called with JavaScript, in which case a JSON response should be requested. This is done by appending _format=json to the receive endpoint URL querystring (the underscore is used to adhere to the same naming convention as other private fields).

https://apigatewayurl/to/[email protected]?_format=json

This returns the response with a Access-Control-Allow-Origin header set to *, to denote all domains. This CORS header is required as a security feature for cross-domain JavaScript calls.

function buildJsonResponse (statusCode, message, data) {let response = {statusCode: statusCode,headers: {'Access-Control-Allow-Origin': '*','Content-Type': 'application/json'},body: JSON.stringify({statusCode: statusCode,message: message})}return response}

Send lambda

AWS will trigger the send lambda handler after it receives an event invoked from the receive lambda.

module.exports.handle = (event, context, callback) => {let email = mailBuilder.build(event)mailService.send(email).then(function () {utilityLog.success('Successfully sent email')}).catch(function (error) {utilityLog.error(['Error sending email', event, error], callback)})}

The email body is built by looping over the event’s data which contains all of the user form fields.

function buildMessage (data) {let message = ''for (let field in data) {// Don't send private variables prefixed with an underscoreif (field.slice(0, 1) !== '_') {message += field.toUpperCase() + ': ' + data[field] + '\r\n'}}message += '---' + '\r\n'message += 'Sent with Formplug'return message}

The above buildMessage function is called in the exported build function which puts together the full email object for the send lambda handler.

module.exports.build = function (data) {return {Source: buildSource(),Destination: {ToAddresses: [data['_to']]},Message: {Subject: {Data: 'You have a form submission'},Body: {Text: {Data: buildMessage(data)}}}}}

The email object is sent via Amazon Simple Email Service, which has been encapsulated in a separate function (which allows it to be easily stubbed during testing). This function is called from the send lambda handler.

const aws = require('aws-sdk')const sesClient = new aws.SES()

module.exports.send = function (email) {return sesClient.sendEmail(email).promise()}

Encrypt lambda

Usage of encryption is optional; the service can be used without it. It gives users the opportunity to encrypt their email address in the receive endpoint.

https://apigatewayurl/to/1974d0cc894607de62f0581ec1334997

The encrypt handler grabs the email address from the URL, validates it and passes it to an encryption function.

module.exports.handle = (event, context, callback) => {let data = encryptRequest.getParams(event)if (encryptRequest.validate(data, callback)) {data['_encrypted'] = httpEncryption.encrypt(data['_email'])httpRoute.render('encrypt-success', data, callback)}}

Encryption is done through node’s built-in crypto module, using a cipher that is created using a custom encryption key environment variable.

const crypto = require('crypto')

const config = require('../../config.json')

module.exports.encrypt = function (str) {let cipher = crypto.createCipher('aes-256-ctr', config.ENCRYPTION_KEY)let crypted = cipher.update(str, 'utf8', 'hex')crypted += cipher.final('hex')return crypted}

module.exports.decrypt = function (str) {let decipher = crypto.createDecipher('aes-256-ctr', config.ENCRYPTION_KEY)let text = decipher.update(str, 'hex', 'utf8')text += decipher.final('utf8')return text}

The encrypted email address is returned to the user via the encrypt-success route which generates the HTTP response.

Wrap-up

In this post we went through the creation of a form forwarding service that can be used to handle contact form submissions for both static and dynamic websites. I walked through possible alternative solutions before going on to describe the technical architecture of my solution. The service uses the Serverless Framework to deploy to AWS Lambda, which allows it realise many of the benefits of self-hosting but with reduced complexity.

For the complete project code and for installation instructions, view the repository on Github.

danielireson/formplug-serverless_formplug-serverless — Formplug is a serverless form backend for AWS Lambda_github.com

If you found this post interesting you might enjoy an article I wrote a couple of weeks ago where I walked through another serverless project. I used the Serverless Framework to create a word count webhook for Github.

Creating a word count webhook for GitHub_A webhook is not a personal weapon, although it may sound like one. So if you’re looking to learn how to inflict damage…_hackernoon.com