# Introduction

In this article, we're going to create a basic CI/CD pipeline for a typical Python package. The text focuses on practical aspects and should be enough to get you started.

## What is CI/CD

Skim through this paragraph if you're a veteran.

\
*CI/CD* stands for *Continuous Integration / Continuous Deployment*. It is an approach to software development that aims to build and deploy the code with each change (aka continuously).

## Benefits of a CI/CD pipeline

While introducing an automatic workflow might be a technical challenge for an unaccustomed team, it brings enough benefits to provide a handsome return on investment. These include:

\
* You can automate boring, repetitive, manual stuff
* Quality can be maintained by introducing automatic checks (static analysis, linting, testing)
* Development can happen at a higher velocity when you have processes in place
* Engineers can focus on actual development

## What Should a Pipeline Contain?

 ![Steps in a good CI pipeline](https://cdn.hackernoon.com/images/ckzkdg-9-y-800-fv-0-bs-60-fyg-7-wvy.jpg)

I have conducted a survey among developers I worked with in order to create a reasonable consensus on what should be automated in a CI pipeline. We've based our standards on twenty five responses in the survey.

## CI/CD tools (free!)

Okay, that sounds great, but how much will I pay for all of this? Is this something I can use in a personal project? Will it be a costly choice for my company?

\
Luckily, in the age of SaaS, introducing an automated CI/CD pipeline is easier and cheaper than ever.

\
GitHub and Gitlab both maintain competitive workflow engines: GitHub Actions and Gitlab CI. This article focuses on using the former.

\
How much does it cost? It's free! Well, at least up to some point. In both products, pipelines for public projects are free. For private ones, there is a free CI-minutes quota. If you need more, you need to pay up or deploy a *runner* on your own infrastructure.

# Let's code a pipeline!

*Coding* is an overstatement here. Let's define our CI/CD steps in a yaml file.

\
If you decide to expand the pipeline, you may want to split its config to multiple workflow files. In our case, single file is enough.

\
You can examine the final code and the sample python package code that was used on [github](https://github.com/horosin/python-standard-ci).

## Secrets

We are going to define several secret variables in order to authenticate with external services. Whenever you see the following in the yaml below

\
```
secrets.SECRET_NAME
```

\
define the secret in your project setting. Just go to Secrets > Actions and click "New repository secret". You can also define secrets at the team level.

## Defining a workflow

Standard workflow configuration needs to include the name and conditions that trigger actions described in the file. Our `on` key specifies that the pipeline will be run on pushes to master branch.

\
```yaml
name: Python workflow

on:
  push:
    branches: [ master ]
```

## Job configuration

Next, we need to configure the job details. We call our job *python_ci*, and tell it to run under the latest version of ubuntu.

\
```yaml
jobs:
  python_ci:

    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: [3.7]
```

\
You can notice a *matrix* option in the config. It is used to specify the strategy to run the pipeline in different environments with different configurations. I am running the code only in python 3.7 but you can add more versions by just adding elements to the array - `[3.7, 3.8, 3.9]`.

## Basic setup

Firstly, we need to get our project's code and set up Python. We can do this using pre-made pipeline steps provided by GitHub. Whenever a step contains `uses` keyword, it is utilizing a step made by someone else. Relative repository paths are supplied to `uses` keyword and follow `{user}/{step-name}@{version}` format.

\
```yaml
    steps:
    - uses: actions/checkout@v2

    - name: Set up Python ${{ matrix.python-version }}
      uses: actions/setup-python@v2
      with:
        python-version: ${{ matrix.python-version }}
```

## Install dependencies

Then, we need to install dependencies. To make this config applicable to more projects, I am installing flake8 and pytest explicitly, in case they are not included in *requirements.txt* file. The file is going to be used if it exists.

\
```yaml
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        python -m pip install flake8 pytest
        if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
```

## Lint

Let's then lint our code. Flake8 will check for obvious errors, bad code style and suspicious code constructs. First invocation checks for critical errors. The second looks for non-breaking mistakes and only prints warnings.

\
```yaml
    - name: Lint with flake8
      run: |
        flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
        flake8 . --count --exit-zero --max-complexity=10 --max-line-length=120 --statistics
```

## Test

Another essential part of the pipeline is testing. In this case, I am assuming you are using pytest to run your checks. I am also saving the coverage report in xml to later upload it to the test result tracking service. Modify this accordingly if needed.

\
```yaml
    - name: Test with pytest
      run: |
        pytest --cov --cov-report=xml
```

## Coverage tracking with codecov

 ![A look at the Codecov control panel](https://cdn.hackernoon.com/images/ckzkdg-9-y-900-fw-0-bs-6-azwk-76-zq.jpg)

\
If you don't know Codecov, definitely [check it out!](https://about.codecov.io) It is a great tool for tracking your test results, monitoring quality improvements, and providing useful metrics. We are using a pre-made action template `codecov/codecov-action@v1` and supplying it with the required configuration.

\
```yaml
    - name: Upload coverage to Codecov
      env:
        CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
      if: ${{ env.CODECOV_TOKEN }}
      uses: codecov/codecov-action@v1
      with:
        token: ${{ secrets.CODECOV_TOKEN }}
        file: coverage.xml
        name: ${{ matrix.os }}, python ${{ matrix.python-version }}
```

## Check Types With mypy

Another check we're going to use is type check with mypy. Don't worry, we won't annotate our Python code with types. Mypy can help you even if your code is not annotated. It can detect many common errors or inconsistencies.

\
```yaml

    - name: Check types with mypy
      run: mypy src
```

\
Let's say you write a code that switches a variable type, which is widely considered a bad practice.

\
```python
if __name__ == '__main__':
  tst = 'omg'
  tst = 0
```

\
Mypy will let you know that it's an error.

\
```
$ mypy src
src/main.py:3: error: Incompatible types in assignment (expression has type "int", variable has type "str)
Found 1 error in 1 file (checked 1 source file)
```

## Build a package

If your project is a python package, you can build it using the following config. In other cases, use your project-specific commands.

\
```yaml

    - name: Install pypa/build
      run: python -m pip install build

    - name: Build a binary wheel and a source tarball
      run: >-
        python -m
        build
        --sdist
        --wheel
        --outdir dist/
        .
```

## Publish a package

In this case, we're going to publish a package to pypi. Again, we are using existing action template from pypa.

\
```yaml
    - name: Publish package
      uses: pypa/gh-action-pypi-publish@27b31702a0e7fc50959f5ad993c78deac1bdfc29
      with:
        user: __token__
        password: ${{ secrets.PYPI_API_TOKEN }}
```

## Send a notification to a Slack channel

If our build and publish went all well, we can notify ourselves or our team on Slack.

\
```yaml

    - uses: act10ns/slack@v1
        with:
          status: ${{ job.status }}
          steps: ${{ toJson(steps) }}
          channel: '#workflows'
        env:
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
```

## Build and publish documentation

If you are using a standard documentation tool like Sphinx, you can build and deploy the documentation in actions as well.  In this case we're using a standard `make html` command. It is followed by deployment to GitHub pages, using a pre-made action step.

\
Have a look at the original pages deploy step documentation if you want to know more:
<https://github.com/peaceiris/actions-gh-pages>

\
Note: `secrets.GITHUB_TOKEN` is automatically created by GitHub, you don't need to define it yourself.

\
```yaml

    - name: Documentation
      run: cd docs && make html

    - name: Deploy documentation
      uses: peaceiris/actions-gh-pages@v3
      with:
        github_token: ${{ secrets.GITHUB_TOKEN }}
        publish_dir: ./docs/_build/html
```

# Summary

That's it! Your repo now has a basic CI/CD setup. Keep in mind this is just an example and your mileage may vary. We skimmed through many implementation details and nuances so be sure to check out [official docs](https://docs.github.com/en/actions) and dive deeper.

## Complete code

<https://github.com/horosin/python-standard-ci>

## Discussion

Is there anything else you automate in your CI/CD? Would you like to automate something I didn't describe? Let's talk about it in the comments!

\
You can also join me at my new blog! <https://horosin.com>

\
*First published [here](https://horosin.com/python-project-cicd-with-github-actions)*

Slack

Velocity

2022 - HackerNoon Contributor of the Year - Cicd

How to Integrate GitHub Actions and CI/CD With Your Next Python Project

Too Long; Didn't Read

Companies Mentioned

@horosin

RELATED STORIES