Deploying on AWS Free Tire with Docker and Fabric

In this article i want to summarize all things that u will need to make good dev environment and deployment for a small application.To make this happen we will use and containers and orchestration and app as a typical project AWS Free Tire Docker Django Link on this project github: https://github.com/creotiv/aws-docker-example Before go farther please install Docker first: https://docs.docker.com/install/linux/docker-ce/ubuntu/ All code running under Python 3.6 Docker Docker is a container virtualization engine that gives u ability to create cheap and fast environments for production and development use. Containers are not virtual machines. Key idea of containers is to make them as thin as possible. So you cant run Windows container on Linus system. Here is good image to see the difference. Docker image & container Two main parts of Docker system is images and containers. Image — is a clear build of environment without any working state Container — is a running/stopped image with some state(DB records created, files uploaded, etc.) Awesome thing in Images is that they created with just one file (Dockerfile) and thus can be stored in Git/Docker Hub, simple transferred across the net. This makes CD/CI much simpler and lightweight. Docker compose In most cases docker-compose tool is used for running all containers on local dev/CI/CD machine with the same configuration as they run in production. But also it can be used for single-machine deploy scenario(but we will use Docker Swarm for this) Orchestration Docker Swarm — is a service for container orchestration, basically it control running of containers on different machines with different configs and manage all communication between them.Docker Swarm not only orchestration engine that exist, there are also Kubernets, AWS ECS, Azure Container Service, Google Container Engine, and else.For big production application i recommend to use Kubernets. But for this simple demo we will use Docker Swarm as it the simplest and good choice for small/medium projects. Testing things locally Before making any changes on remote machines it’s good to test things locally. Our application will use 3 containers for Django+Gunicorn, Nginx, PosgreSQL instance, and for each of them we need an image. Making images with Dockerfiles Before trying to write some big Dockerfile with plenty of installation code it good to search it on (aka github for docker images).We will search for Python, Nginx, PostgreSQL. For example here is python image For all our images we will use most latest Alpine Linus images. We will not write Dockerfile for PostgreSQL as we will use clear image without any modifications. Docker Hub https://hub.docker.com/_/python Dockerfile for App: https://medium.com/media/6fad06d5ee97ef80b99340e4fad84a72/href Dockerfile for Nginx https://medium.com/media/446b945d5924dea3277ff6213fc7d2d7/href Now let’s try to understand what happening here. — used to set image which we are using as a starting point of our modifications. In out first case we use python image with tag 3.7.2-alpine3.9.if we dont need previous image we cant set it from scratch “From scratch” FROM — set an environment variable. ENV — running a command. RUN — change base directory of image. if directory doesn’t exists it creates it. WORKDIR — copy data from local fs to image fs COPY —command that will be run on each container initialization from this image. ENTRYPOINT So for application container we installing some needed system libs, then creating app dir at /usr/src/app, copy our app requirements there, install them, copy our app dir. We use our entrypoint to control PostgreSQL initialization(because Swarm control only container initialization and not control services in it) and for Django updates.Here is another way of controlling services: https://docs.docker.com/compose/startup-order/ https://medium.com/media/7ef6ea72b1cdc1febeae0e3f88231cc6/href After we understand how this work, lets build our first imagesdocker build -t [TITLE][DIRECTORY WITH DOCKERFILE] docker build -t awsdemo-app appdocker build -t awsdemo-nginx nginx Now if we call docker images we will see our two built images :) Understanding docker-compose It good practices to use different version of docker-compose.yml files for different environments.So we have also two of them docker-compose.prod.yml for prod, and docker-compose.yml for dev enviroment (The difference only in DEBUG mode and code reload param for gunicorn). https://medium.com/media/bcddba8f543d82f0a64845d5b9dc2dbe/href First, there are many versions of docker and docker-compose syntax. We will using the latest 3rd version. — setting up containers — setting up persistent storage for our data(on local machine, cause we dont want lost DB or files after each container restart) — setting up network connections between containers(For our case we will use simplest network type — mesh network) Services Volumes Networks Each service have a name that also a domain in our network. — setting up image from which container will be created. First local images, then remote images(Docker Hub, etc) — command to run after container initialized. Running Gunicorn with code hot-reload on port 8000 — linking ports between docker and local mahcine — Linking up volumes to instance. Here ./app/:/usr/src/app/ we linking our local app dir to container app dir, so we can change the code in run-time without container restart — setting up container env variables — setting up dependencies(initialization order) — connection to networks Image Command Ports Volumes Environment Depends_on Networks More about docker compose syntax and params here: https://docs.docker.com/compose/overview/ Now lets run docker-compose up and open to see result. http://localhost/ If things are working we can go to deploy stage. Deploy AWS Free Tire By default AWS gives you 750h of EC T3.micro instance.We will run our project on Ubuntu Server 18.04 LTS (HVM), SSD Volume Type - ami-34c14f4a Click ‘Launch Instance’ Choose Ubuntu server 18.04 Choose t3.micro instance type Here you will need to add All Tcp to firewall rules, so you can access any tcp port from public(for production set only needed ports) Click next until security group page. Click next until security group page In the end it will create ssh key which we will use to access to the instance. Deploying with Fabric2 For devops thing i would recommend use Fabric2 for small/medium projects and Ansible for medium/big projects. But today we will use Fabric2 because it simple as a door. First let’s install it pip install fabric2 What Fabric do is basically connect to host over ssh and run some commands that can be defined as tasks. Entrypoint for it is and to run it just call fab2 COMMAND fabfile.py https://medium.com/media/5a52f2c25cde344e56a27aab3d114ff6/href So what we have 4 task here: install-instance, set-pass, deploy, pgdumpAfter we create instance on AWS we need to install many things on it before we can run docker. I used simple command per line approach, but it also can be inside of some shell file and we can just run it. Dont forget to change HOST and KEY_FILE params Your domain you can find in AWS EC dashboard in instance params (Public DNS (IPv4)) install-instance 19# check if docker is running if not run install ops. 21–28# install docker stuff 29# Initialize docker swarm main server 30–33# create ssh key to be able to pull from private github 34–36# waiting while user will add key to github ssh keys 6. 37# adding github domain to known host. in other case it will ask this on first pull 7. 38–41# Creating app dir and cloning it fro github 8. 42–43# Building images 9. 44# Add our containers to the swarm. Stack is a group of services that running together. 10. 45–51# waiting for web container and trying to set django super user password $(docker ps -q -f name=awsdemo_web) — used to get service container id. This will not work if we have replication with more then one container for service. deploy For deploy we just getting all changes from github and then running service update.Updates in a Swarm mode running without breaking connections. Old container will run until all old queries will not be proceeded. All new queries will go to the new container instance. pgdump Here we first run pg_dump on db container, then copy dumped data from it to remote machine fs, and then use Fabric2 Class Transfer to download file to our local machine. Now let’s install all this things. fab2 install-instance fab2 set-pass Now if you go to your AWS domain you should see Hope it was useful :)