Creating a form forwarding service for AWS Lambda

Since originally publishing this blog post I’ve . I’d recommend that you read that instead of continuing with this as some of the content below is now out of dat . written an updated article e _It’s estimated that approximately 269 billion emails are sent in a single day. Over 10 million were sent as you read…_hackernoon.com Introducing Formplug v1, a form forwarding service for AWS Lambda In this post I’ll be introducing a service I’ve created that makes setting up email form forwarding easy. It’s built in using Node.js and the Serverless Framework. It deploys to Lambda and uses Amazon Simple Email Service to send the emails. To check out the code see the Github repository below. Keep reading for an explanation of the problem and a walk-through of the technical solution. Javascript AWS _formplug-serverless - Form forwarding service for AWS Lambda_github.com danielireson/formplug-serverless Problem Recently I’ve been finding myself taking a preference towards static site hosting for my side projects. Github Pages works well because like most people I use Github to host my git repositories anyway. To deploy a site it’s as easy as a to the branch and in a few short minutes the website will be live. In many cases static hosting is perfectly fine but on a couple of occasions now I’ve struggled to set up a simple email-based contact form — something which wouldn’t usually be considered too challenging of a task. There’s a couple of ways of handling this. git push gh-pages Option 1: no contact form The obvious first option is to avoid contact forms altogether and instead just display an email address. This is the option that I’ve taken on many occasions because it’s the easiest but has a few downsides. It pushes extra work onto the users — they have to leave the site and open their email client. It’s also hard to specify what information is required when visitors first contact you, you might require qualifying information for a quote for example. You can have a message asking users for specific information but there’s no validation that can be done to confirm people are adhering to this request. Option 2: mailto links One solution for sending forms by email is to set the form’s action attribute to a link set to an email address. When the user submits the form it opens their default email client with the form data as the body, as shown in the screenshot below. The result isn’t a great user experience. Not all users will have a default email client setup, and even when they do they’ll get presented with a draft email. It’s unclear whether the form has be submitted and what actions need to be taken by them. mailto Option 3: use server-side code As a security feature you can’t send email client-side in the browser. If this were possible website visitors would be highly susceptible to malicious code sending email on their behalf. If you already host your apps on a server it’s painless to setup a contact form. Server-side scripting languages like PHP, Java or ASP have the capabilities that make it easy to send email. In PHP for example it can be achieved in two lines. $message = $_POST['message'];mail('johndoe@example.com', 'Form submission', $message); If you don’t already have a server however, having to maintain one just for a contact form seems a little wasteful. There’s time involved in maintaining it as well as financial costs, and it introduces other considerations around scalability and security. Option 4: use a third-party service Managed form backend services exist that range in functionality and scope. for example makes it easy to integrate forms into services such as Trello, Slack or Google Sheets, but can be expensive just for a simple contact form with plans start at $59/month. There’s also always a risk involved when utilising a third-party service; it can go offline at any moment’s notice and you have no control of it. If there service was to shutdown there would be no notification, contact forms would just suddenly stop working. FormKeep Solution A self-hosted form forwarding service is the optimal solution in my opinion. You get the benefits of control as well as no end-user experience degradation. Choosing to develop the backend for AWS Lambda further reduces many of the downsides of self-hosting. AWS Lambda lets you run code without provisioning or managing servers. You pay only for the compute time you consume — there is no charge when your code is not running. With Lambda, you can run code for virtually any type of application or backend service — all with zero administration. Just upload your code and Lambda takes care of everything required to run and scale your code with high availability. There’s multiple approaches to building applications on AWS Lambda. For simple services you can use the AWS Management Console to create functions and set up triggers. For more complex services it’s better to develop locally and deploy to AWS afterwards, where you can enjoy the familiarities of your favourite code editor. Getting the project structure set up initially can be challenging as there’s lots of things to consider. Which AWS region is the service going to be deployed to? How are AWS services going to be provisioned? How are environment variables going to be managed? How is security going to be managed? Because of these considerations I like to use an opinionated framework called the Serverless Framework when working with AWS Lambda. It provides sensible defaults and allows us to to provision services at a higher and easier to understand abstraction. Using the CLI, deploying and updating a service is as simple as running . serverless deploy If you’re looking to learn a bit more about the Serverless Framework I’d recommend you watch the following video below by its creator Austin Collins. The video is a couple of years old now but does a great job of introducing some of the core concepts. I’d also recommend going through the . Serverless docs Architecture overview The form forwarding service is composed of three lambda functions: encrypt, receive, send. The encrypt and receive functions are triggered through the Amazon API Gateway routes and respectively. (An API Gateway route is essentially a public URL.) The send lambda function is only ever invoked by the receive lambda function. The receive lambda is responsible for handling the form submission and invoking the send lambda to send the email. The encrypt lambda can optionally be used to get an encrypted representation of an email address to use in the receive endpoint URL. This is explained in more detail later. /encrypt /to Receive lambda The receive handler is triggered by the route which accepts one parameter, the email address to forward the form to. The endpoint is generated on the initial and returned in the terminal window. It takes the following structure. /to serverless deploy https://apigatewayurl/to/johndoe@example.com HTML forms should POST to this endpoint. This will trigger the receive lambda (shown below). It’s responsible for validating the request, getting the form data and invoking the send lambda. module.exports.handle = (event, context, callback) => {let data = receiveRequest.getParams(event)if (receiveRequest.validate(data, callback)) {eventInvoker.send(data).then(function () {httpRoute.render('receive-success', data, callback)}).catch(function (error) {httpRoute.render('receive-error', data, callback)})}} Getting the form data The form’s POST request will generate an API Gateway event which is automatically passed to the handler as the first parameter. An example of an event is shown below (with sensitive fields removed). {resource:'/to/{_to}',path:'/to/johndoe@example.com',httpMethod:'POST',headers: {Accept:'*/*','Accept-Encoding':'gzip, deflate','cache-control':'no-cache','CloudFront-Forwarded-Proto':'https','CloudFront-Is-Desktop-Viewer':'true','CloudFront-Is-Mobile-Viewer':'false','CloudFront-Is-SmartTV-Viewer':'false','CloudFront-Is-Tablet-Viewer':'false','CloudFront-Viewer-Country':'GB','content-type':'application/x-www-form-urlencoded',Host:'','User-Agent':'','X-Amz-Cf-Id':'','X-Amzn-Trace-Id':'','X-Forwarded-For':'','X-Forwarded-Port':'443','X-Forwarded-Proto':'https'},queryStringParameters:null,pathParameters: {_to:'johndoe@example.com'},stageVariables:null,requestContext: {path:'/dev/to/johndoe@example.com',accountId:'',resourceId:'',stage:'dev',requestId:'',identity:{cognitoIdentityPoolId:null,accountId:null,cognitoIdentityId:null,caller:null,apiKey:'',sourceIp:'',accessKey:null,cognitoAuthenticationType:null,cognitoAuthenticationProvider:null,userArn:null,userAgent:'',user:null},resourcePath:'/to/{_to}',httpMethod:'POST',apiId:''},body:'first=test&second=test&third=test',isBase64Encoded:false} We’re only interested in some of the event data and so we extract this through the function. Specifically we’re interested in getting the form data from the email address from and the query string from . getParams event.body, event.pathParameters event.queryStringParamters module.exports.getParams = function (event) {let data = Object.assign({}, querystring.parse(event.body), event.pathParameters, event.queryStringParameters)return data} The node querystring module is used to parse the POST form data from the request body. This takes the string and turns it into a JavaScript object. first=test&second=test&third=test Validating the request The data then goes through a validation function which checks that the email is valid and that the honeypot field hasn’t been filled out. module.exports.validate = function (data, callback) {return hasValidEmail(data, callback) && hasNoHoneypot(data, callback)} function hasValidEmail (data, callback) {if (!('_to' in data)) {httpRoute.render('receive-no-email', data, callback)return false}if ('_to' in data && !httpValidation.isEmail(data['_to'])) {httpRoute.render('receive-bad-email', data, callback)return false}return true} function hasNoHoneypot (data, callback) {if ('_honeypot' in data && data['_honeypot'] !== '') {httpRoute.render('receive-honeypot', data, callback)return false}return true} The honeypot field is an optional spam prevention feature. Users can add the field to their form and hide it from visitors. If the form is submitted and the honeypot field isn’t empty the request will be ignored (because a spam bot has tried to enter a value there). Invoking the send lambda After validation the send lambda is invoked asynchronously with a payload of the data using the AWS-SDK npm package. module.exports.send = function (data) {let event = {FunctionName: `formplug-${config.STAGE}-send`,InvocationType: 'Event',Payload: JSON.stringify(data)}return lambda.invoke(event).promise()} It invokes it asynchronously because we don’t want visitors to be waiting after they’ve submitted the form for Amazon to send the email. Invoking it in this way also means that AWS will s. automatically retry it if it fail Generating a HTTP response HTTP responses are made through the handler’s function. This is provided by the Serverless Framework. For successful responses the callback function should be called with the first parameter as null and the second parameter as the response object. For example, to generate a successful plain text HTTP response we could do the following. callback callback(null, {statusCode: 200,headers: {'Content-Type': 'text/plain'},body: 'Form submission successfully made'}) But instead of a plain text response we want either a HTML, JSON or URL redirect response (depending on what was requested by the user). The application defaults to a HTML response. HTML responses are generated by loading a local template file and replacing a variable with an appropriate message. {{ message }} function buildHtmlResponse (statusCode, message, data) {let response = {statusCode: statusCode,headers: {'Content-Type': 'text/html'},body: generateView(message)}if (statusCode === 302) {response.headers.Location = data['_redirect']}return response} function generateView (message) {try {let template = fs.readFileSync(path.resolve(__dirname, 'template.html')).toString()message = template.replace('{{ message }}', message)} catch (error) {utilityLog.error(error.message)}return message} Users can also request a URL redirect after the form submission. This URL should be supplied as a hidden input in the form itself. If a field is supplied then the application will set a 302 HTTP status code and the . The visitor’s web browser will pick these up and handle the redirection. _redirect HTTP header Location Non-redirect response messages are defined in a routes file. Users can define custom messages in their config or use the service defaults. These are called through a route function. render module.exports.render = function (type, data, callback) {let routeDetails = module.exports.getRouteDetails(type, data)callback(null, httpResponse.build(routeDetails.statusCode, routeDetails.message, data))} module.exports.getRouteDetails = function (type, data) {let statusCode, messageswitch (type) {case 'encrypt-no-email':statusCode = 422message = config.MSG_ENCRYPT_NO_EMAIL || 'You need to provide an email address to encrypt.'breakcase 'encrypt-bad-email':statusCode = 422message = config.MSG_ENCRYPT_BAD_EMAIL || 'The supplied email address is not valid.'breakcase 'encrypt-success':statusCode = 200message = data['_encrypted']breakcase 'receive-honeypot':statusCode = 422message = config.MSG_RECEIVE_HONEYPOT || 'You shall not pass.'breakcase 'receive-no-email':statusCode = 422message = config.MSG_RECEIVE_NO_EMAIL || 'Form not sent, the admin has not set up a forwarding email address.'breakcase 'receive-bad-email':statusCode = 422message = config.MSG_RECEIVE_BAD_EMAIL || 'Form not sent, the admin email address is not valid.'breakcase 'receive-error':statusCode = 500message = config.MSG_RECEIVE_ERROR || 'Form not sent, an error occurred while sending.'breakcase 'receive-success':statusCode = httpValidation.hasRedirect(data) ? 302 : 200message = config.MSG_RECEIVE_SUCCESS || 'Form submission successfully made.'breakdefault:statusCode = 500message = 'An error has occurred.'}return {statusCode, message}} Oftentimes the form forwarding service will be called with JavaScript, in which case a JSON response should be requested. This is done by appending to the receive endpoint URL querystring (the underscore is used to adhere to the same naming convention as other private fields). _format=json https://apigatewayurl/to/johndoe@example.com?_format=json This returns the response with a header set to to denote all domains. This is required as a security feature for cross-domain JavaScript calls. Access-Control-Allow-Origin *, CORS header function buildJsonResponse (statusCode, message, data) {let response = {statusCode: statusCode,headers: {'Access-Control-Allow-Origin': '*','Content-Type': 'application/json'},body: JSON.stringify({statusCode: statusCode,message: message})}return response} Send lambda AWS will trigger the send lambda handler after it receives an event invoked from the receive lambda. module.exports.handle = (event, context, callback) => {let email = mailBuilder.build(event)mailService.send(email).then(function () {utilityLog.success('Successfully sent email')}).catch(function (error) {utilityLog.error(['Error sending email', event, error], callback)})} The email body is built by looping over the event’s data which contains all of the user form fields. function buildMessage (data) {let message = ''for (let field in data) {// Don't send private variables prefixed with an underscoreif (field.slice(0, 1) !== '_') {message += field.toUpperCase() + ': ' + data[field] + '\r\n'}}message += '---' + '\r\n'message += 'Sent with Formplug'return message} The above function is called in the exported function which puts together the full email object for the send lambda handler. buildMessage build module.exports.build = function (data) {return {Source: buildSource(),Destination: {ToAddresses: [data['_to']]},Message: {Subject: {Data: 'You have a form submission'},Body: {Text: {Data: buildMessage(data)}}}}} The email object is sent via Amazon Simple Email Service, which has been encapsulated in a separate function (which allows it to be ). This function is called from the send lambda handler. easily stubbed during testing const aws = require('aws-sdk')const sesClient = new aws.SES() module.exports.send = function (email) {return sesClient.sendEmail(email).promise()} Encrypt lambda Usage of encryption is optional; the service can be used without it. It gives users the opportunity to encrypt their email address in the receive endpoint. https://apigatewayurl/to/1974d0cc894607de62f0581ec1334997 The encrypt handler grabs the email address from the URL, validates it and passes it to an encryption function. module.exports.handle = (event, context, callback) => {let data = encryptRequest.getParams(event)if (encryptRequest.validate(data, callback)) {data['_encrypted'] = httpEncryption.encrypt(data['_email'])httpRoute.render('encrypt-success', data, callback)}} Encryption is done through node’s built-in module, using a cipher that is created using a custom encryption key environment variable. crypto const crypto = require('crypto') const config = require('../../config.json') module.exports.encrypt = function (str) {let cipher = crypto.createCipher('aes-256-ctr', config.ENCRYPTION_KEY)let crypted = cipher.update(str, 'utf8', 'hex')crypted += cipher.final('hex')return crypted} module.exports.decrypt = function (str) {let decipher = crypto.createDecipher('aes-256-ctr', config.ENCRYPTION_KEY)let text = decipher.update(str, 'hex', 'utf8')text += decipher.final('utf8')return text} The encrypted email address is returned to the user via the route which generates the HTTP response. encrypt-success Wrap-up In this post we went through the creation of a form forwarding service that can be used to handle contact form submissions for both static and dynamic websites. I walked through possible alternative solutions before going on to describe the technical architecture of my solution. The service uses the Serverless Framework to deploy to AWS Lambda, which allows it realise many of the benefits of self-hosting but with reduced complexity. For the complete project code and for installation instructions, view the repository on Github. _formplug-serverless — Formplug is a serverless form backend for AWS Lambda_github.com danielireson/formplug-serverless If you found this post interesting you might enjoy I wrote a couple of weeks ago where I walked through another serverless project. I used the Serverless Framework to create a word count webhook for Github. an article _A webhook is not a personal weapon, although it may sound like one. So if you’re looking to learn how to inflict damage…_hackernoon.com Creating a word count webhook for GitHub