Flawless Releases with Continuous Deployment and Docker

Docker is a container technology that enables developers to run entire applications as a unit. It offers all the benefits of virtual machines, without the high overhead:

• Consistency: Production and development environments are equal.
• Portability: fewer dependencies with the underlying OS; the same image can be deployed on any cloud provider.
• Low overhead: better performance than virtual machines.
• Divide and conquer: distribute services among different containers.

However, Docker introduces a new variable to the equation: the app must be baked into the container image, then correctly deployed. In addition, setting up a test environment can prove more challenging.

Here is where a CI/CD platform can be of great value to us: by automating all tasks and giving us a fast and reliable environment to work in.

In this hands-on tutorial, we'll learn how Semaphore can help us achieve all this in a few minutes. The tutorial is broken down into two sections:

• Continuous integration: build and test the app image.
• Continuous deployment: send the image to Heroku to run online

Enter the Application

First, we will work with Semaphore's ready-to-use Python Flask demo. The app consists of a plain task manager, written for the Flask web micro-framework, with a MongoDB acting as a database backend. To keep things nice and tidy, it has been split into two containers: one for the database and the other for the web server.

The CI/CD pipelines will:

1. Build a Docker image with our application.
2. Push the image to Docker Hub.
3. Test the application inside the container.
4. Deploy it to Heroku.

To get started, fork the semaphore-demo-python-flask repository and clone it.

Continuous Integration

A well designed Continuous Integration setup will help us to:

• Spend less time testing and deploying.
• Get 100% automated testing.
• Avoid it-works-on-my-machine syndrome.

The objective of this section is to bake the app into a container. Ready to get started?

Docker Hub provides free storage for images, so go ahead and get a Docker Hub account. You'll also need a Semaphore account; you can sign up with your GitHub account. The free plan is plenty for our purposes.

To add the project to Semaphore, click on the + (plus) sign under Projects and select your repository.

One last thing: Semaphore needs to be able to access your Docker Hub
repository. The best way of storing sensitive data is by using Secrets, which are automatically encrypted and made available to jobs when called:

1. Under Configuration, go to Secrets.
2. Click the Create New Secret button.
3. Name your secret 

   pyflask-semaphore

    and type your Docker Hub credentials:

Integrate

Now, everything is in place to start integrating. How about a trial run? Edit any file in your repository and push the update:

$ touch some_file
$ git add some_file
$ git commit -m "first run of the integration pipeline"
$ git push

Head back to Semaphore. in a few seconds, you'll see the new workflow starting in your dashboard:



Inside the workflow, you'll be able to see the pipeline:



Don't mind the Promote button; we'll get to it in a bit. The
good news is that the integration pipeline is all green. If everything
went according to plan you should have a new image in your Docker Repository.

We're halfway there. But... what happened? How did Semaphore do all that?

Building an Image

Creating a custom Docker image is easily achieved with the right Dockerfile. The project already ships with a working 

flask.Dockerfile

. Take a look at the contents:

FROM python:3.7
ADD . ./opt/
WORKDIR /opt/
EXPOSE 5000
RUN pip install --upgrade pip
RUN pip install -r requirements.txt
CMD ["python","run.py"]

The first line defines the base image used as a starting point, in this case, a basic Debian image with Python 3.7. Next, we

ADD

the app files into the /opt dir. 

RUN

invokes pip, Python's package manager, to install all the app dependencies. The final

CMD

defines how the app starts.

The container setup is completed with docker-compose.yml:

version: '3.5'
services:
  mongodb:
    image: mongo:3.4.20
    container_name: "mongodb"
    ports:
      - 27017:27017
    command: mongod --smallfiles --logpath=/dev/null
  flasksemaphore:
    image: pyflasksemaphore
    container_name: semaphore-pyflask-docker_flasksemaphore_1
    build:
      context: .
      dockerfile: ./flask.Dockerfile
    ports:
      - "5000:5000"
    volumes:
      - .:/opt/
    environment:
      - DB=mongodb://mongodb:27017/tasks
      - PORT=5000
    depends_on:
      - mongodb

With Compose, we define two services:

• mongodb: references a public MongoDB image.
• flasksemaphore: the custom-built app container.

It only takes one command to build the image and start everything:

docker-compose up

The CI Pipeline

Next, let's examine how is Semaphore implementing the integration pipeline. If you ever get lost, take a look at the Semaphore's guided tour.

version: v1.0
name: Semaphore Python / Flask / Docker Example Pipeline
agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu1804
blocks:
  - name: Build
    task:
      secrets:
        - name: pyflask-semaphore
      jobs:
      - name: Docker build
        commands:
          - echo "${DOCKER_PASSWORD}" | docker login -u "${DOCKER_USERNAME}" --password-stdin
          - checkout
          - docker-compose build
          - docker tag pyflasksemaphore:latest "$DOCKER_USERNAME"/pyflasksemaphore:latest
          - docker tag pyflasksemaphore:latest "$DOCKER_USERNAME"/pyflasksemaphore:$SEMAPHORE_WORKFLOW_ID
          - docker push "$DOCKER_USERNAME"/pyflasksemaphore:latest
          - docker push "$DOCKER_USERNAME"/pyflasksemaphore:$SEMAPHORE_WORKFLOW_ID
          - docker images

  - name: Run & Test Docker image
    task:
      secrets:
        - name: pyflask-semaphore
      prologue:
        commands:
          - echo "${DOCKER_PASSWORD}" | docker login -u "${DOCKER_USERNAME}" --password-stdin
          - checkout
          - docker pull "$DOCKER_USERNAME"/pyflasksemaphore
          - docker-compose up -d
      jobs:
      - name: Check Running Images
        commands:
          - docker ps
      - name: Run Unit test
        commands:
          - docker exec -it semaphore-pyflask-docker_flasksemaphore_1 python -m unittest

promotions:
  - name: Deploy to Heroku
    pipeline_file: deploy-heroku.yml
    auto_promote_on:
      - result: passed

Let's break it down in more digestible bits.

Agent: The agent sets the machine type and the Operating System that drives the pipeline. Semaphore offers machines in several sizes. For our needs, the e1-standard-4 with its 4 CPUs and 8GB of RAM is powerful enough to build a Docker image in a few seconds.

The super convenient Ubuntu 18.04 image includes everything you need to get started right away, including Docker, Docker Compose, Python, and Heroku. No additional components required.

Build block: Blocks define actions for the pipeline. Each block has a single task, and each task can have one or more jobs. Jobs within a block run concurrently, each one in its own fully isolated virtual machine. Once all jobs in a block complete, the next block begins.

The build block does exactly that; it builds the Docker image:

1. secrets imports the Docker Hub variables.
2. checkout clones GitHub repository.
3. docker login is required for pushing the image to Docker Hub.
4. docker-compose builds the image…
5. …which is tagged, and finally pushed to the registry.

Test block: In this block, both containers are spun up to do integration tests. The prologue is executed before every job in a block. In this case, the prologue pulls the image and starts the app with docker-compose up.

We have two tests jobs:

• Run unit test: start a test script inside the container.
• Check running images: lists docker containers running.

Promotion: The final part of .semaphore/semaphore.yml defines how the workflow continues. At this point, we can chain multiple pipelines with promotions to create complex workflows. Promotions can be manual or automatic. In this case, we have a manual connection to the deployment pipeline:

promotions:
  - name: Deploy to Heroku
    pipeline_file: deploy-heroku.yml

Continuous Delivery

Once we have a working image, we're ready to enter the continuous delivery stage. Here we'll discuss how we can make deploy the app so our users can enjoy it.

We're going to add two new services to our scheme:

• Heroku: to run the application.
• MongoDB Atlas: to get a managed MongoDB database.

Sign up for a Heroku account and get an authorization token:

Heroku

Sign up for a Heroku account and get an authorization token:

1. Click on your account profile, on the top right corner.
2. Select Account Settings.
3. Go to the Applications tab.
4. Press the Create Authorization button.
5. Set the description as:

   semaphore-demo-python-flask

Finally, create an empty application. From your Heroku dashboard, click the New button and select Create New App:

• Set your application name, or leave blank for a random one.
• Select your preferred zone: US or Europe.

In Semaphore, create a new secret called 

heroku

 to store the authorization token:


MongoDB Atlas

While Heroku has a MongoDB addon, it isn't included on the free plan. 
Meanwhile, MongoDB Atlas offers a 500MB cluster for free. Not bad, not bad at all. However, the setup process is rather lengthy, so please bear with me:

1. Sign up for a MongoDB Atlas Account
2. Select AWS as a provider.
3. Choose the region that matches your Heroku app. For the US 

   us-east-1

   , and for Europe 

   eu-west-1

4. On Cluster Tier, select the M0 Sandbox
5. You may set a name to describe your cluster. You can leave the rest of the settings alone.
6. Click on Create cluster. Give it a few minutes to provision.



Now, for the database user:

1. On the left side navigation bar, open Security.
2. Click on the +Add New User button.
3. Add the user

   semaphore-demo-python-flask

   . Choose a secure password and set the privileges to "Read and write any database"
4. Back in Security, select the IP Whitelist tab.
5. Click on the +Add IP Address button.
6. Choose Allow access from anywhere and Confirm.
   

To get the connection URI, go back to Clusters and click on the Connect button. Under the "Connect your application" section choose "Python 3.6 or later". Copy the entire connection string.

The connection string is incomplete; replace <password> with your actual password. If it has any special characters, you should first run it through URL Encode.

Head back to Semaphore to create a 

mongodb-atlas

secret:


The CD Pipeline

In this section, we're going to examine all the steps that the deployment pipeline goes through. Take a look at 

.semaphore/deploy-heroku.yml

version: v1.0
name: Deploy to Heroku
agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu1804
blocks:
  - name: Deploy to Heroku
    task:
      secrets:
        - name: mongodb-atlas
        - name: pyflask-semaphore
        - name: heroku
      env_vars:
        - name: HEROKU_APP
          value: <YOUR_APP_NAME>
      jobs:
        - name: Deploy
          commands:
            - checkout
            - echo "${DOCKER_PASSWORD}" | docker login -u "${DOCKER_USERNAME}" --password-stdin
            - docker pull "$DOCKER_USERNAME"/pyflasksemaphore:$SEMAPHORE_WORKFLOW_ID
            - heroku container:login
            - docker tag "$DOCKER_USERNAME"/pyflasksemaphore:$SEMAPHORE_WORKFLOW_ID registry.heroku.com/$HEROKU_APP/web
            - docker push registry.heroku.com/$HEROKU_APP/web
            - heroku config:set DB="$MONGODB_URI"
            - heroku stack:set container --app $HEROKU_APP
            - heroku container:release web --app $HEROKU_APP

Environment and Secrets: We only need to explicitly set a single variable,

$HEROKU_APP

, which should point to your Heroku app name. Once you've done this, go ahead and replace the value with it.

The deployment block needs access to all services, and the variables are imported from secrets: mongodb-atlas,pyflask-semaphore, heroku.

Deploy block: The only new commands introduced in this block are related to Heroku Docker:

1. Tag the image with Heroku Registry URL.
2. Send the URI variable for the MongoDB connection.
3. Set the stack mode to container.
4. Release: gets the app started.

Deploy

Only one more step to go! Commit the changes to get the CI/CD started:

$ git add .semaphore
$ git commit -m "ready to deploy!"
$ git push

After a few minutes, we should have the CI pipeline completed. Click on the Promote button to launch the CD pipeline:



Finally, click on the Open app button to access the live application. Happy task managing!



Summing it up

We've explored how to run a Python application using Docker. After that, we learned how to use Semaphore to create a pipeline that automates running the tests and the necessary build commands, as well as deploying the application to Heroku.

Previously published at: https://semaphoreci.com/community/tutorials/continuous-deployment-of-a-python-flask-application-with-docker-and-semaphore.