Someone recently asked me about options and pros/cons of different ways to run containers on AWS. Once I was done explaining I thought it might make sense to write it down, maybe it’s useful for more folks out there.
Two quick notes: this ain’t a comprehensive guide, I’m sure I’ve forgotten some options. Let me know. Also, in terms of orchestrator I’ve limited myself to Kubernetes. YMMV.
I. Hand-rolled vanilla containers on EC2
This option has been around for as long as EC2 and Docker have been around. You use EC2 to spin up one or more VMs, install Docker and—especially if you have multiple VMs where containers can be launched—use shell scripts, Fleet, Puppet, Chef, or the like to manage the lifecycle of your containers. The first wave of container-based apps (from late 2013 on) essentially used this approach.
Pros:
- full control
Cons:
- from a 2018 POV largely considered outdated
- a hand-rolled, ad-hoc orchestration layer is hard to maintain and extend
- you have to (manually) take care of upgrades
II. Installer-based vanilla Kubernetes on EC2
Conceptually building on top of the previous recipe, one would use an installer such as kubeadm, kops, or kubicorn to install vanilla Kubernetes on EC2.
Pros:
- fine-grained control
- fast and rather well-documented
Cons:
- requires upgrade management and not all paths might be equally well tested and/or documented
III. Templated Kubernetes
A number of templated solutions exist, such as Heptio’s Quick Start for Kubernetes, that allow you to spin up a Kubernetes cluster with a few clicks or commands.
Pros:
- very quick and easy
Cons:
- depending on the solution, usage might be best for dev & test and not necessarily for prod
IV. A Kubernetes distribution on EC2
A number of vendors are offering their Kubernetes distro on AWS. This is a great option if you don’t want to or can not make the necessary decisions (like which SDN to use, etc.) and want to benefit from additional, higher-level functionality such as build pipelines & registries or multi-tenancy. Some example are:
- Rancher (via Marketplace)
- Tectonic
- OpenShift Dedicated
Pros:
- end-to-end taken care of by the distro
- often: no need to for operations incl. upgrades
Cons:
- there may be limitations in terms of what and where you can use this setup
- price
V. Amazon Elastic Container Service (ECS)
Available since 2014, ECS was up until last year the primary way Amazon wanted you to run containers. ECS has a proprietary architecture and orchestrator and is used by a number of customers, from Ubisoft to TIBCO.
Pros:
- very deep integration into AWS
- well understood, battle-tested
Cons:
- unclear where it is heading, given new options (see below) exist
VI. AWS Fargate
Introduced at the most recent re:Invent in late 2017, Fargate seems to be the gateway drug. It has been called ‘cluster-less container offering’ and other things and has (besides Azure Container Instances) no direct equivalent in other cloud providers.
Pros:
- super easy to use
- nice on-ramp to more complex/complete offerings in the platform
Cons:
- pricing and optimizing the prizing for a certain use case can sometimes be hard to figure out
VII. Amazon Elastic Container Service for Kubernetes (EKS)
Same as Fargate, EKS was revealed at last re:Invent. Unlike Fargate, EKS is at time of writing in early 2018, still only available as part of a sign-up-only preview program. I can’t really say much about it—until I’ve tried it—other than it looks interesting and I’m happy that AWS made that move.
What do you think? Any stories to share about your way to do containers on AWS? Maybe some experience report or deeper insight what worked for you or what folks should be aware of running containers on AWS? Share here or ping me on Twitter (DMs are open).
UPDATE (2018–02–01): As pointed out by Mark Coleman that with https://hyper.sh there exists a Fargate equivalent and that since 2016!