Ufo — Easily Build Docker Images and Deploy Containers to AWS ECS

Update: Did a presentation on ufo, the slides are here: UFO Ship for AWS ECS.

Quick Introduction

Amazon EC2 Container Service, ECS, is an AWS service that provisions and manages Docker containers on a cluster of EC2 instances. As with most of AWS services, it is great and simply requires a little tooling wrapped around it to create a smooth flow. Ufo is a simple tool that makes building and shipping Docker containers to AWS ECS super easy.

Ufo provides a command called ufo ship that does the following:

1. builds a docker image.
2. generates and registers the ECS template definition.
3. deploys the ECS template definition to the specified service.

Ufo deploys a task definition that is created via a template generator which is fully controllable. We’ll go over a quick example to show what the template looks like and how it works.

Task Definition ERB Template and DSL Generator

The task definition is created from an ERB template in the ufo/templates folder. Here is an example: ufo/templates/main.json.erb.

{"family": "<%= @family %>","containerDefinitions": [{"name": "<%= @name %>","image": "<%= @image %>","cpu": <%= @cpu %>,<% if @memory %>"memory": <%= @memory %>,<% end %><% if @memory_reservation %>"memoryReservation": <%= @memory_reservation %>,<% end %><% if @container_port %>"portMappings": [{"containerPort": "<%= @container_port %>","protocol": "tcp"}],<% end %>"command": <%= @command.to_json %>,<% if @environment %>"environment": <%= @environment.to_json %>,<% end %>"essential": true}]}

The ERB template to use is specified in ufo/task_definitions.rb with the source method. Ufo loads the ERB template when it evaluates the task_definition blocks in ufo/task_definition.rb.

task_definition "hi-web" dosource "main" # will use ufo/templates/main.json.erbvariables(family: task_definition_name,# image: tongueroo/hi:ufo-[timestamp]-[sha]image: helper.full_image_name,environment: helper.env_file('.env.prod')name: "web",container_port: helper.dockerfile_port,command: ["bin/web"])end

As you can see above, the task_definitions.rb file has some special variables and helper methods available. These helper methods provide useful contextual information from the project so you don’t have to copy paste and update the code in multiple places. For example, one of the variable provides the exposed port in the Dockerfile of the project. Here is a list of the important ones:

• helper.full_image_name — The full docker image name that ufo builds. The “base” portion of the docker image name is defined in ufo/settings.yml. For example, the base portion is “tongueroo/hi” and the full image name is tongueroo/hi:ufo-[timestamp]-[sha]. So the base does not include the Docker tag and the full image name does include the tag.
• helper.dockerfile_port — Exposed port extracted from the Dockerfile of the project.
• helper.env_file — This method takes an .env file which contains a simple key value list of environment variables and converts the list to the proper task definition json format.

The 2 classes which provide these special helper methods are in ufo/dsl.rb and ufo/dsl/helper.rb. Refer to these classes for the full list of the special variables and methods.

Usage Example

An example will demonstrate how easy it is to use ufo. More details are provided on the project’s official documentation: http://ufoships.com.

This example is a demo rails app that returns the rails welcome page. I have also created bin/worker and bin/clock scripts that just run an infinite loop to mock out worker and clock processes for testing. The full source code for the demo project is available on GitHub: tongueroo/hi.

Test Demo Project Locally in Mac OSX

Let’s setup the app, install the dependencies and start up the web process.

$ git clone https://github.com/tongueroo/hi$ cd hi$ bundle$ bin/web # start the web server

Let’s curl for a 200 response code.

$ curl -svo /dev/null localhost:3000 2>&1 | grep ‘< HTTP’< HTTP/1.1 200 OK$

Build and Test Demo Project with Docker

Now let’s build the docker image with ufo and test that it works locally. I’m only showing some of the shell output to keep the paste small in size and useful in context.

$ ufo init --app hi --image tongueroo/hiSetting up ufo project…created: ./bin/deployexists: ./Dockerfilecreated: ./ufo/settings.ymlcreated: ./ufo/task_definitions.rbcreated: ./ufo/templates/main.json.erbcreated: ./.envStarter ufo files created.$ ufo docker build$ docker images | grep “tongueroo/hi”tongueroo/hi ufo-2016–11–30T16–25–26-e1d57ce e511ec8a328a About a minute ago 826.8 MB$ docker run -d -p 3000:3000 — name hi tongueroo/hi:ufo-2016–11–30T16–25–26-e1d57ce$ docker ps$ curl -svo /dev/null localhost:3000 2>&1 | grep ‘< HTTP’< HTTP/1.1 200 OK$ docker stop hi ; docker rm hi$

In the above snippet, I ran two ufo commands: ufo init and ufo docker build. Let’s review some of the files that ufo init created:

• bin/deploy — Wrapper deploy script that shows how to deploy to 3 common web, worker and clock processes as services at the same time with ufo.
• ufo/settings.yml — This is where you set the Docker image name to be built. There is a service_cluster mapping option, which you can use to avoid having to always provide the --cluster option in the CLI. It is quite handy.
• ufo/task_definitions.rb — This is where you define the variables to be substituted into the ERB template.
• ufo/templates/main.json.erb — This the task definition ERB template. You can modify this and have full control over the task definition template that gets registered to ECS.
• .env — A starter .env file is provided if one does not yet exist on your project. You can rename this to .env.prod and .env.stag if that makes more sense for your needs. Remember to update the env_file line in task_definitions.rb if you rename the .env file.

The ufo docker build command created a tongueroo/hi:ufo-2016–11–30T16–25–26-e1d57ce Docker image.

Ship the Docker Image to ECS

Let’s ship the web process as an ECS service. First, create an ECS Cluster called stag that we will use to ship the web service to. You will also need the ELB Target Group associated with the web service so that the app will be accessible from anywhere in the world. I have created a stag cluster, a “hi-elb” and “hi-target-group” for this example. You can grabbed the Target Group ARN from Load Balancing / Target Groups:

You only need to provide the ELB Target Group ARN the first time deploying with ufo since you cannot update the Target Group of the ECS Service afterwards. If you want to change the Target Group, you must create and deploy a new service instead. This is an AWS ECS design decision.

ufo ship

Now we can ship the docker container to ECS with ufo ship!

$ ufo ship hi-web --cluster stag --target-group=arn:aws:elasticloadbalancing:us-east-1:467446852200:targetgroup/hi-target-group/f61e87b3c4761922Building docker image with:docker build -t tongueroo/hi:ufo-2016-12-01T07-37-53-e1d57ce -f Dockerfile .Docker image tongueroo/hi:ufo-2016-12-01T07-38-32-e1d57ce built. Took 2s.Pushed tongueroo/hi:ufo-2016-12-01T07-38-32-e1d57ce docker image. Took 4s.Building Task Definitions...Generating Task Definitions:Generated task definition at: ./ufo/output/hi-web.jsonGenerated task definition at: ./ufo/output/hi-worker.jsonGenerated task definition at: ./ufo/output/hi-clock.jsonTask Definitions built in ufo/output.hi-web task definition registered.Shipping hi-web...hi-web service created on stag clusterWaiting for deployment of task definition hi-web:3 to complete...........Time waiting for ECS deployment: 58s.Software shipped!

Let’s inspect and review what the ufo ship command actually did. First it built the docker image with a name of tongueroo/hi:ufo-2016-12-01T07-37-53-ec1d57ce. You can check it out with docker images.

$ docker images ufo-2016-12-01T07-38-32-e1d57ce a9e97fa264ab 5 minutes ago 826.8 MB$

Second, ufo generated all task definitions in the ufo/output folder and registers only the one that is deployed: hi-web. Let’s check out one of the generated task definitions: ufo/output/hi-web.json.

{"family": "hi-web","containerDefinitions": [{"name": "web","image": "tongueroo/hi:ufo-2016-12-01T08-07-08-e1d57ce","cpu": 128,"memoryReservation": 256,"portMappings": [{"containerPort": "3000","protocol": "tcp"}],"command": ["bin/web"],"environment": [{"name": "ADMIN_PASSWORD","value": "secret"}],"essential": true}]}

Third, ufo deploys the newly registered hi-web:3 task definition to ECS.

Confirm It Works

Grab the ELB DNS endpoint from EC2 Console / Load Balancing / Load Balancers.

Confirm that the app is up with curl:

$ curl -svo /dev/null hi-elb-1381308520.us-east-1.elb.amazonaws.com 2>&1 | grep '< HTTP'< HTTP/1.1 200 OK$

That’s it! The web process for this tongueroo/hi app has been deployed to ECS.

Using bin/deploy

When we use ufo init at the beginning of this example, it generated a bin/deploy script. This script handles deploying common application processes like web, worker and clock all at once. These processes typically use the same codebase, ie: same docker image, but have slightly different run time settings. For example, the docker run command for a web process could be puma and the command for a worker process could be sidekiq. Environment variables are sometimes different also.

Let’s quickly test to make sure that the worker and clock process work locally first. The worker and clock process scripts are actually mocked out and are just simple infinite bash loops. That’s all we need to test things. Here’s a quick local test:

$ bin/worker+ true+ echo 'fake worker process running...'fake worker process running...+ sleep 5^C$ bin/clock+ true+ echo 'fake clock process running...'fake clock process running...+ sleep 5^C$

Now let’s check out the bin/deploy script.

#!/bin/bash -xe

ufo ships hi-{web,clock,worker}-stag --cluster stag

The bin/deploy wrapper script just calls the ufo ships command, which is designed to build 1 docker image and ship it to multiple ECS services. Now let’s ship all 3 processes as services to ECS!

bin/deploy # deploys clock, worker and web!

Summary

The ufo tool automates building the docker image, registering the ECS task definition, and deploying the container to the ECS service. The project page is available on GitHub at tongueroo/ufo. Try it out and let me know what you think!

Thanks for reading this far. If you found this post useful, I’d really appreciate it if you recommend this post (by clicking the clap button) so others can find it too! Also, connect with me on LinkedIn.