Flawless Releases with Continuous Deployment and Docker

Docker is a container technology that enables developers to run entire applications as a unit. It offers all the benefits of virtual machines, without the high overhead: Production and development environments are equal. Consistency: fewer dependencies with the underlying OS; the same image can be deployed on any cloud provider. Portability: better performance than virtual machines. Low overhead: distribute services among different containers. Divide and conquer: However, Docker introduces a new variable to the equation: the app must be baked into the container image, then correctly deployed. In addition, setting up a test environment can prove more challenging. Here is where a CI/CD platform can be of great value to us: by automating all tasks and giving us a fast and reliable environment to work in. In this hands-on tutorial, we'll learn how Semaphore can help us achieve all this in a few minutes. The tutorial is broken down into two sections: build and test the app image. Continuous integration: send the image to Heroku to run online Continuous deployment: Enter the Application First, we will work with Semaphore's ready-to-use . The app consists of a plain task manager, written for the web micro-framework, with a acting as a database backend. To keep things nice and tidy, it has been split into two containers: one for the database and the other for the web server. Python Flask demo Flask MongoDB The will: CI/CD pipelines Build a Docker image with our application. Push the image to Docker Hub. Test the application inside the container. Deploy it to Heroku. To get started, fork the repository and clone it. semaphore-demo-python-flask Continuous Integration A well designed Continuous Integration setup will help us to: Spend less time testing and deploying. Get 100% automated testing. Avoid it-works-on-my-machine syndrome. The objective of this section is to bake the app into a container. Ready to get started? Docker Hub provides free storage for images, so go ahead and get a account. You'll also need a account; you can sign up with your GitHub account. The free plan is plenty for our purposes. Docker Hub Semaphore To add the project to Semaphore, click on the sign under Projects and select your repository. + (plus) One last thing: Semaphore needs to be able to access your Docker Hub repository. The best way of storing sensitive data is by using , which are automatically encrypted and made available to jobs when called: Secrets Under , go to . Configuration Secrets Click the button. Create New Secret Name your secret and type your Docker Hub credentials: pyflask-semaphore Integrate Now, everything is in place to start integrating. How about a trial run? Edit any file in your repository and push the update: $ touch some_file $ git add some_file $ git commit -m $ git push "first run of the integration pipeline" Head back to Semaphore. in a few seconds, you'll see the new workflow starting in your dashboard: Inside the workflow, you'll be able to see the pipeline: Don't mind the button; we'll get to it in a bit. The good news is that the integration pipeline is all green. If everything went according to plan you should have a new image in your . Promote Docker Repository We're halfway there. But... what happened? How did Semaphore do all that? Building an Image Creating a custom Docker image is easily achieved with the right Dockerfile. The project already ships with a working . Take a look at the contents: flask.Dockerfile FROM python:3.7 ADD . ./opt/ WORKDIR /opt/ EXPOSE 5000 RUN pip install --upgrade pip RUN pip install -r requirements.txt CMD [ , ] "python" "run.py" The first line defines the base image used as a starting point, in this case, a basic Debian image with Python 3.7. Next, we the app files into the /opt dir. invokes pip, Python's package manager, to install all the app dependencies. The final defines how the app starts. ADD RUN CMD The container setup is completed with docker-compose.yml: version: '3.5' services: mongodb: image: mongo:3.4.20 container_name: "mongodb" ports: - 27017 :27017 command: mongod --smallfiles --logpath=/dev/null flasksemaphore: image: pyflasksemaphore container_name: semaphore-pyflask-docker_flasksemaphore_1 build: context: . dockerfile: ./flask.Dockerfile ports: - "5000:5000" volumes: - .:/opt/ environment: - DB=mongodb://mongodb:27017/tasks - PORT=5000 depends_on: - mongodb With Compose, we define two services: mongodb: references a public MongoDB image. flasksemaphore: the custom-built app container. It only takes one command to build the image and start everything: docker-compose up The CI Pipeline Next, let's examine how is Semaphore implementing the integration pipeline. If you ever get lost, take a look at the . Semaphore's guided tour version: v1.0 name: Semaphore Python / Flask / Docker Example Pipeline agent: machine: type: e1-standard-2 os_image: ubuntu1804 blocks: - name: Build task: secrets: - name: pyflask-semaphore jobs: - name: Docker build commands: - echo "${DOCKER_PASSWORD}" | docker login -u "${DOCKER_USERNAME}" --password-stdin - checkout - docker-compose build - docker tag pyflasksemaphore:latest "$DOCKER_USERNAME"/pyflasksemaphore:latest - docker tag pyflasksemaphore:latest "$DOCKER_USERNAME"/pyflasksemaphore:$SEMAPHORE_WORKFLOW_ID - docker push "$DOCKER_USERNAME"/pyflasksemaphore:latest - docker push "$DOCKER_USERNAME"/pyflasksemaphore:$SEMAPHORE_WORKFLOW_ID - docker images - name: Run & Test Docker image task: secrets: - name: pyflask-semaphore prologue: commands: - echo "${DOCKER_PASSWORD}" | docker login -u "${DOCKER_USERNAME}" --password-stdin - checkout - docker pull "$DOCKER_USERNAME"/pyflasksemaphore - docker-compose up -d jobs: - name: Check Running Images commands: - docker ps - name: Run Unit test commands: - docker exec -it semaphore-pyflask-docker_flasksemaphore_1 python -m unittest promotions: - name: Deploy to Heroku pipeline_file: deploy-heroku.yml auto_promote_on: - result: passed Let's break it down in more digestible bits. : The sets the and the Operating System that drives the pipeline. Semaphore offers machines in several sizes. For our needs, the e1-standard-4 with its 4 CPUs and 8GB of RAM is powerful enough to build a Docker image in a few seconds. Agent agent machine type The super convenient image includes everything you need to get started right away, including Docker, Docker Compose, Python, and Heroku. No additional components required. Ubuntu 18.04 : Blocks define actions for the pipeline. Each block has a single task, and each task can have one or more jobs. Jobs within a block run concurrently, each one in its own fully isolated virtual machine. Once all jobs in a block complete, the next block begins. Build block The build block does exactly that; it builds the Docker image: secrets imports the Docker Hub variables. clones GitHub repository. checkout is required for pushing the image to Docker Hub. docker login docker-compose the image… builds …which is , and finally to the registry. tagged pushed : In this block, both containers are spun up to do integration tests. The is executed before every job in a block. In this case, the prologue pulls the image and starts the app with . Test block prologue docker-compose up We have two tests jobs: Run unit test: start a test script inside the container. Check running images: lists docker containers running. : The final part of .semaphore/semaphore.yml defines how the workflow continues. At this point, we can chain multiple pipelines with to create complex workflows. Promotions can be manual or automatic. In this case, we have a manual connection to the deployment pipeline: Promotion promotions promotions: - name: Deploy to Heroku pipeline_file: deploy-heroku.yml Continuous Delivery Once we have a working image, we're ready to enter the continuous delivery stage. Here we'll discuss how we can make deploy the app so our users can enjoy it. We're going to add two new services to our scheme: : to run the application. Heroku : to get a managed MongoDB database. MongoDB Atlas Sign up for a Heroku account and get an authorization token: Heroku Sign up for a account and get an authorization token: Heroku Click on your account profile, on the top right corner. Select Account Settings. Go to the Applications tab. Press the Create Authorization button. Set the description as: semaphore-demo-python-flask Finally, create an empty application. From your Heroku dashboard, click the button and select : New Create New App Set your application name, or leave blank for a random one. Select your preferred zone: or . US Europe In Semaphore, create a new secret called to store the authorization token: heroku MongoDB Atlas While Heroku has a MongoDB addon, it isn't included on the free plan. Meanwhile, offers a 500MB cluster for free. Not bad, not bad at all. However, the setup process is rather lengthy, so please bear with me: MongoDB Atlas Sign up for a Account MongoDB Atlas Select AWS as a provider. Choose the region that matches your Heroku app. For the US , and for Europe us-east-1 eu-west-1 On Cluster Tier, select the M0 Sandbox You may set a name to describe your cluster. You can leave the rest of the settings alone. Click on . Give it a few minutes to provision. Create cluster Now, for the database user: On the left side navigation bar, open . Security Click on the button. +Add New User Add the user . Choose a secure password and set the privileges to "Read and write any database" semaphore-demo-python-flask Back in Security, select the tab. IP Whitelist Click on the button. +Add IP Address Choose and Allow access from anywhere Confirm. To get the connection URI, go back to and click on the button. Under the "Connect your application" section choose "Python 3.6 or later". Copy the entire connection string. Clusters Connect The connection string is incomplete; replace with your actual password. If it has any special characters, you should first run it through . URL Encode Head back to Semaphore to create a secret: mongodb-atlas The CD Pipeline In this section, we're going to examine all the steps that the deployment pipeline goes through. Take a look at .semaphore/deploy-heroku.yml version: v1.0 name: Deploy to Heroku agent: machine: type: e1-standard-2 os_image: ubuntu1804 blocks: - name: Deploy to Heroku task: secrets: - name: mongodb-atlas - name: pyflask-semaphore - name: heroku env_vars: - name: HEROKU_APP value: jobs: - name: Deploy commands: - checkout - echo "${DOCKER_PASSWORD}" | docker login -u "${DOCKER_USERNAME}" --password-stdin - docker pull "$DOCKER_USERNAME"/pyflasksemaphore:$SEMAPHORE_WORKFLOW_ID - heroku container:login - docker tag "$DOCKER_USERNAME"/pyflasksemaphore:$SEMAPHORE_WORKFLOW_ID registry.heroku.com/$HEROKU_APP/web - docker push registry.heroku.com/$HEROKU_APP/web - heroku config:set DB="$MONGODB_URI" - heroku stack:set container --app $HEROKU_APP - heroku container:release web --app $HEROKU_APP : We only need to explicitly set a single variable, , which should point to your Heroku app name. Once you've done this, go ahead and replace the value with it. Environment and Secrets $HEROKU_APP The deployment block needs access to all services, and the variables are imported from secrets: mongodb-atlas,pyflask-semaphore, heroku. : The only new commands introduced in this block are related to : Deploy block Heroku Docker Tag the image with URL. Heroku Registry Send the URI variable for the MongoDB connection. Set the to container. stack mode : gets the app started. Release Deploy Only one more step to go! Commit the changes to get the CI/CD started: $ git add .semaphore $ git commit -m $ git push "ready to deploy!" After a few minutes, we should have the CI pipeline completed. Click on the button to launch the CD pipeline: Promote Finally, click on the button to access the live application. Happy task managing! Open app Summing it up We've explored how to run a Python application using Docker. After that, we learned how to use Semaphore to create a pipeline that automates running the tests and the necessary build commands, as well as deploying the application to Heroku. Previously published at: https://semaphoreci.com/community/tutorials/continuous-deployment-of-a-python-flask-application-with-docker-and-semaphore .