Hackernoon logoOpsWorks meets docker-compose by@fbueno

OpsWorks meets docker-compose

OpsWorks is a tool that makes it easy to provision and manage hosts (EC2 instances) and integrate them with EC2 ELBs. You create application stacks that have application layers (instances, load balancers) and configure them using custom chef cookbooks. You can also provide a “custom JSON” on stack creation time that will be passed to chef. The idea here is to have the YAML translated to JSON so we can use it inside the custom JSON field on the OpsWorks Stack’s creation.
image
Felippe Bueno Hacker Noon profile picture

@fbuenoFelippe Bueno

Infrastructure Engineer

I found AWS OpsWorks when I was looking for a way to orchestrate the applications containers that I had just created. At that moment I was using shell script and puppet.

Just after starting using OpsWorks, I realised it should be my key tool for containerisation because of its features, like auto-healing to EC2 instances, auto-scale, and time-based EC2 instances. Of course there are other solutions available out there, and I’m still trying them, and I have to say that I’m kind of tending to kubernetes or deis.

Anyway, I thought that I could use the custom json OpsWorks feature to translate the docker-compose.yml that our developers already use to tell the docker hosts which container images they should run and how.

OpsWorks is a tool that makes it easy to provision and manage hosts (EC2 instances) and integrate them with EC2 ELBs for instance. Basically, you create application stacks that have application layers (instances, load balancers, RDSs…) and configure them using custom chef cookbooks. You can also provide a “custom JSON” on stack creation time that will be passed to chef.

The custom JSON is the key feature here, as it can be used to configure, tag and set properties of EC2 instances that will run chef recipes.

Some of the things I did (with the contribution of @tnache) were:

  • Change the EC2 instances tags on boot time
  • Update a specific Route53 IN A entry based on the current public ip of an EC2 instance
  • Mount NFS exports

But the major goal of this article is to tell you how I did the docker 
container orchestration with docker-compose.

Docker-compose uses yaml files to declare how containers should run. And
yaml files can be easily translated to json.

So, things like this:

db:
image: postgres
web:
build: my-djangoapp
command: python manage.py runserver 0.0.0.0:8000
volumes:
- /data:/data
ports:
- "8000:8000"
links:
- db

When represented in json, should look like:

{
"db": {
"image": "postgres"
},
"web": {
"image": "my-djangoapp",
"command": "python manage.py runserver 0.0.0.0:8000",
"volumes": [
"/data:/data"
],
"ports": [
"8000:8000"
],
"links": [
"db"
]
}
}

Easy….

So, the idea here is to have the YAML translated to JSON so we can use 
it inside the custom JSON field on the OpsWorks Stack’s creation, 
allowing the json to be passed to chef.

Of course that docker-compose as I said uses yaml files, so we would 
have to translate it from json back to yaml using chef, and it is pretty 
easy using chef templates.

Because I wanted to use the custom JSON to things other than simply 
write the docker-compose.yml, I decided to put the yaml translated to 
json inside a new json key. So the json would look like this:

{
"applications":
{
"db": {
"image": "postgres"
},
"web": {
"image": "my-djangoapp",
"command": "python manage.py runserver 0.0.0.0:8000",
"volumes": [
"/data:/data"
],
"ports": [
"8000:8000"
],
"links": [
"db"
]
}
}
}

This way, I can use specific keys to specific chef recipes, for 
instance tags, route53, nfs…

One last thing. As the OpsWorks Stacks supports ‘Layers’ of EC2 
Instances, it would be nice if I could specify the applications json’s 
key to each layer.

{
"layers": {
"djangoapp":{
"applications":
{
"db": {
"image": "postgres"
},
"web": {
"image": "my-djangoapp",
"command": "python manage.py runserver 0.0.0.0:8000",
"volumes": [
"/data:/data"
],
"ports": [
"8000:8000"
],
"links": [
"db"
]
}
}
}
}
}

On the example above, the chef template file would translate to yaml to all the EC2 
Instances inside the OpsWorks Stacks’s layer called ‘djangoapp’ the json 
starting on the ‘applications’ key.

We can “translate” one docker-compose.yml file to each OpsWorks Stack’s
layers creating more keys in the “layers” json’s level. And also, we can 
create a “global” “applications” key which would be merged to the 
specific layer’s “applications”. In that case the more long json would 
be like this:

{
"applications":{
"postfix" {
"image": "postfix"
}
},
"layers": {
"djangoapp":{
"applications":
{
"db": {
"image": "postgres"
},
"web": {
"image": "my-djangoapp",
"command": "python manage.py runserver 0.0.0.0:8000",
"volumes": [
"/data:/data"
],
"ports": [
"8000:8000"
],
"links": [
"db"
]
}
}
}
}
}

With the json above, every single EC2 Instance on the OpsWorks Stack would run the container postfix, and only the EC2 Instances on the layer called ‘djangoapp’ would run the postgres, and 
the my-djangoapp images.

This “feature” also extends itself to the other recipes that I 
mentioned (changing tags, route53 entries, nfs mounts…), so a really 
complete custom JSON on the OpsWorks Stack’s would look like this:

{
"tags"{
"Product": "Django example"
},
"applications":{
"postfix" {
"image": "postfix"
}
},
"layers": {
"djangoapp":{
"tags": {
"Service": "the application"
},
"applications":
{
"db": {
"image": "postgres"
},
"web": {
"image": "my-djangoapp",
"command": "python manage.py runserver 0.0.0.0:8000",
"volumes": [
"/data:/data"
],
"ports": [
"8000:8000"
],
"links": [
"db"
]
}
}
}
}
}

I will soon prepare a Youtube video describing step by step 
the process to get this working for those that have no OpsWorks and Chef knowledge.

Meanwhile, Thiago Nache is making available the docker-compose recipes that we wrote on his bitbucket. You can also can find a working fork on mine.

Hacker Noon is how hackers start their afternoons. We’re a part of the @AMIfamily. We are now accepting submissions and happy to discuss advertising &sponsorship opportunities.
To learn more, read our about page, like/message us on Facebook, or simply, tweet/DM @HackerNoon.
If you enjoyed this story, we recommend reading our latest tech stories and trending tech stories. Until next time, don’t take the realities of the world for granted!

Tags

Join Hacker Noon

Create your free account to unlock your custom reading experience.