Recap In the of this series, we began the tale of the serverless microservice we created in to help us manage the configuration of our customers’ big-data pipelines. We discussed the huge benefits we get from working with Lambda and API Gateway in terms of availability and general peace of mind in terms of our production environment. But we also reviewed some pain points in the development process. These pain points set us on a course to to build an automated flow of deploying a micro-service. previous part ironSource atom AWS serverless In the first part of this trilogy (you are now reading “The Empire Strikes Back”) we discussed setting up proper unit-tests for our Lambda functions in Node.js, so we can have a good time developing on our own machine. In this part, we will describe some tools we adopted and practices we’ve developed for actually interacting with AWS Lambda, since working directly with Amazon’s UI can be cumbersome and error-prone. Introducing Apex First of all, several tools exist in the market today to help streamline the process of working with Lambda, ranging from small utility libraries to full blown solutions like “ ” (now in beta). When evaluating different solutions we chose . Its minimal approach was appealing to us, it was well documented and seemed to have a decent sized community of developers (currently over 3.5k stars on GitHub). Serverless Apex Apex is simple to use and well-documented, so I won’t go into details here on how to set up or get started with it. Instead, I’d rather just go over how we use it in our team. Apex relies on some pretty straightforward JSON files to describe configuration and on a pre-defined structure for your project. In this post, we’ll be working on a hypothetical microservice which allows others to search for repositories in GitHub. This service will expose a single endpoint: and receive input in the query string like . Working with Apex, our sample project structure would look something like this: “GET /repos” “/repos?q=rotemtam” Apex sample project structure (view the source ) here In the directory of our project we have a file describing to Apex general things about the project “src/” project.json We have a directory with a separate directory for each Lambda function. functions Inside each function’s directory, we have a file describing configuration for that specific function. function.json We’ll get into the Dockerfile and the “ directory in a bit. infra/” When we run Apex will create new versions of our functions in our AWS account and update the configuration. “apex deploy” Sharing dependencies between functions TJ Holowaychuk (one of Apex’s creators) recently published an article on Medium with some “ ”, one of which is very relevant to our topic: “Don’t substitute FaaS with writing good libraries”, which basically means that your actual Lambda functions should be very small. You should take advantage of the package/module system in your language to encapsulate whatever logic you need in an ordinary module and use it. A general rule of thumb would be: Handle input and high-level flow-control in the Lambda function, leave all the rest to your libraries. This way you can easily re-use them in other contexts. Do’s and Don’ts of AWS Lambda Remember our hypothetical microservice for searching GitHub? Well, assume it contained a Lambda function which would take a string as input and use that to search GitHub for repositories with that string in their name. There is no reason to include the call to the GitHub API in our Lambda function. Instead we could do something like this: q Which would rely on a “Repository” module like this: By default, Apex will take whatever is in the function directory, put it in a zip file and upload to Lambda. So basically, if you want to include any modules you wrote or 3rd party stuff from NPM, you need to include it there. That works fine for a project with only one function, but what if you have several functions that share modules? Would you have a Node.js for each function and manage the dependencies per function? Copy and paste your custom modules into each folder? package.json file Using Webpack and Apex hooks to create deployable artifacts Fortunately, Apex lets you customize your build process by specifying life-cycle hooks, namely custom and commands. We take advantage of this to make use of to create our deployable artifacts. At build time, webpack will intelligently create a single Javascript file which apex will zip up and deploy. Here’s an example Apex file taken from our : “build” “clean” Webpack project.json example repo The only caveat with this method is that the official AWS Javascript SDK doesn’t play nice with Webpack. Check out to learn more about it. Luckily for us, it turns out that the SDK comes pre-bundled in the Lambda context (super thanks to for pointing this out), so we can basically tell webpack to ignore it in our webpack config: this discussion Victor Delgado Hooking things up to a CI/CD service The next part of our process is to hook this up to a CI/CD service. We currently use and for our builds, but this could easily be done with any other service. Let’s list our requirements for a build job for our microservice: Jenkins Codefresh Install all required packages, install Apex. Run unit tests, exit if they fail. Run “apex deploy” to push our Lambda functions to production. (Build a Swagger file describing our API Gateway) — we’ll get to this in the next post (Redeploy our API Gateway) — coming soon! Building inside a Docker container No truly hip deployment process would be complete without using Docker somewhere, right? Seriously though, there are many advantages to using Docker in your build process. People usually consider Docker containers for running production tasks. Docker containers really are a great way to have complete control over the environment which your code runs in. With Docker you can have an exact replica of your production environment. But the same benefit is true for doing any other type of task, really. By running CI/CD tasks in Docker, you have complete control over the way things happen and you can use whatever tools you like. I usually find that it’s just easier (and faster) for me to run my builds inside Docker containers. Here’s a Dockerfile which I’d use for a task like we described above: Our final build script Our build script might look like this: Lets break it down: Depending on where your build job runs, you might need to deal with setting AWS environment variables. For example, if you run from an EC2 instance that has a role which allows it to interact with Lambda you won’t need it. Apex needs to know what region it’s deploying to, which we set using the environment variable $AWS_REGION We first run inside our container, if all unit tests pass we continue. “npm test” If all is good, we continue to run inside our container. “apex build” We take this build script and plug it into whatever system we’re using to build our code, and BAM! We have an automated CI/CD flow for working with AWS Lambda! Wrapping Up In the previous post of this series, we discussed how to setup unit-tests for Lambda functions. In this post, we’ve outlined how to create an automated workflow for deploying Lambda functions to production environments. In the third and last part of this trilogy (Return of the Jedi), we will review how we set up API Gateway to make our service available as a RESTful Web API and how we automate the grunt work of continuously deploying it with a cool tool we wrote. See you soon!
