How to Set Up GitHub Actions and PyPI Integration for Python Projects

How to Use GitHub Actions and Create Integration With pypi for Python Projects Introduction When I was creating my first Python package, dataclass-sqlalchemy-mixins (github or pypi), I encountered an interesting challenge: how to set up CI/CD on GitHub to ensure that nothing breaks when I push new changes and to automatically publish the code to PyPI. I discussed how to publish your own package using poetry in this article. Typically, to verify any commit made to the master branch through a pull request, it is necessary to run tests. Additionally, using linters to check code style is beneficial, especially when multiple developers are working on the project. *CI (Continuous Integration) - The practice of automatically building and testing code changes after they are merged into the repository* *CD (Continuous Delivery) - The automated delivery of code to development or production environments* Creating Workflows GitHub supports Actions, which are automated processes that can run one or more jobs, including CI/CD pipelines. You can read more about them in the GitHub documentation. Workflows are what GitHub will run according to your configuration. They should be located in the .github/workflows directory and be .yaml files. For example, if we want to run tests after each commit, we need to create a file named test.yml. I chose the name test for convenience and to indicate that this particular workflow is responsible for tests, but it can be anything. name: "Test" on: pull_request: types: - "opened" - "synchronize" - "reopened" push: branches: - '*' workflow_dispatch: jobs: test: runs-on: ubuntu-latest strategy: fail-fast: false steps: - uses: actions/checkout@v4 - name: Set up Python 3.12 id: setup-python uses: actions/setup-python@v5 with: python-version: 3.12 - name: Cache poetry install uses: actions/cache@v4 with: path: ~/.local key: poetry-${{ steps.setup-python.outputs.python-version }}-1.7.1-0 - uses: snok/install-poetry@v1 with: version: 1.7.1 virtualenvs-create: true virtualenvs-in-project: true - name: Load cached venv id: cached-poetry-dependencies uses: actions/cache@v4 with: path: .venv key: venv-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('**/poetry.lock') }} - name: Install dependencies if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true' run: poetry install --no-interaction - name: Run tests run: poetry run pytest --cov-report=xml Let me explain the key elements of the file: name - This is the name of the workflow. pull request - Specifies the types of pull requests that trigger the workflow. The recommended values by default are opened, synchronize, and reopened. You can read more about these types here. push - Specifies the branches that trigger the workflow. jobs - A workflow can consist of one or more jobs, which run in parallel by default but can also be configured to run sequentially. I’d like to discuss what happens during the execution of jobs with this code. First, the name of a job can be used to set up rulesets for the repository or specific branches. In this case, the only job that will run is test. The details of rulesets will be covered later. The runs-on key specifies which operating system the jobs will run on. Setting strategy: fail-fast: false means that GitHub will not cancel all in-progress and queued jobs in the matrix if any job in the matrix fails. The matrix parameter will be discussed later. As you can see, there are a lot of similar codes in the uses key in the steps. For example: actions/setup-python@v5 actions/cache@v4 snok/install-poetry@v1 At first glance, the names of the actions might be confusing, but they are actually the paths to the GitHub repositories for these actions, excluding the @v part. For example, you can view the source code of the first action by visiting https://github.com/actions/setup-python. The same can be done for the other actions. @v is used to set which version of the action is to use. The with key is used to pass parameters to an action. For example, if we want to install Python 3.12, we pass python-version: 3.12 in the Set up Python 3.12 step. The same approach applies to other steps. In short, the test job performs the following steps: Sets up Python with the required version. Installs the Poetry 1.7.1 dependency using GitHub’s cache. Creates a Python .venv in the root directory if a cache is not found. The virtual environment cache depends on the Python version and poetry.lock. If the cached virtual environment is not found, the project's dependencies are installed. Runs the tests. Using Services in Workflows The created workflow is quite simple and may not be suitable for testing more sophisticated applications or projects. This is because such projects often require services like databases or caching to function correctly. While it’s possible to write tests that mock requests to these services, I wouldn’t recommend it, as it can reduce the effectiveness of test coverage. I will demonstrate how to include a service in a workflow, using PostgreSQL as an example. jobs: test: services: postgres: image: postgres:latest env: POSTGRES_PASSWORD: postgres ports: - 5432:5432 options: >- --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5 Adding a services key with a required service name allows to set up it during a job. Moreover, you can pass settings to it similarly when using docker. Using Several Parameters During a Job Run Last but not least is the situation where you need to run a job using different parameters. For example, you may need to run a job with several specific Python versions, not just the latest, or with different dependencies. This can be particularly useful when you need to support backward compatibility with older dependencies. GitHub provides the strategy key to address this. The matrix parameter allows you to run a job with multiple values. jobs: test: runs-on: ubuntu-latest strategy: fail-fast: false matrix: python-version: ["3.11", "3.12"] sqlalchemy-version: ["1.4.52", "2.0.31"] A job with a matrix set up this way will run each value of a parameter’s list against every other value. As a result, there will be four runs of the job with the following combinations of Python and SQLAlchemy versions, respectively: 3.11, 1.4.52 3.11, 2.0.31 3.12, 1.4.52 3.12, 2.0.31 You can access the parameters in a job run using ${{ matrix.python-version }} or ${{ matrix.sqlalchemy-version }}. - name: Set up Python ${{ matrix.python-version }} id: setup-python uses: actions/setup-python@v5 with: python-version: ${{ matrix.python-version }} - name: Install sqlalchemy ${{ matrix.sqlalchemy-version }} run: pip install sqlalchemy==${{ matrix.sqlalchemy-version }} Using GitHub Rulesets But sometimes, simply running jobs for your project isn't enough. There are situations where you may need to restrict merging commits to the master branch. While this might not be necessary when you're working alone on a project, it’s highly recommended when multiple people are working on the same codebase. This helps prevent situations where someone accidentally merges a commit that breaks the code, even if the pipelines fail. GitHub provides a useful tool for this called rulesets. You can access it from the repository settings menu at https://github.com/{Author}/{Repository}/settings/rules. After creating a new ruleset, scroll down to Require status checks to pass, turn it on, and specify which jobs are required to pass. To find a check, type the name of the required job, and GitHub will provide suggestions. Note that each job in the matrix will have its own name, allowing you to select them individually. Publishing to pypi In the last part, I would like to talk about how to create an integration with pypi and publish new versions with release. In order to do that, a new Yaml file with publishing workflows should be created. name: Upload Python Package to PyPi on release on: release: types: [published] permissions: contents: read jobs: deploy: runs-on: ubuntu-latest environment: name: pypi url: https://pypi.org/project/dataclass-sqlalchemy-mixins/ steps: - uses: actions/checkout@v4 - name: Set up Python uses: actions/setup-python@v3 with: python-version: '3.x' - uses: snok/install-poetry@v1 with: version: 1.7.1 virtualenvs-create: true virtualenvs-in-project: true - name: Build package run: poetry build - name: Publish package uses: pypa/gh-action-pypi-publish@27b31702a0e7fc50959f5ad993c78deac1bdfc29 with: user: __token__ password: ${{ secrets.PYPI_API_TOKEN }} This workflow: Sets up Python. Installs the Poetry 1.7.1 dependency using GitHub’s cache. Build the source and wheel archives. Publishes the package to pypi Yes, it is as simple as it looks. You just need to add a PYPI_API_TOKEN to repository secrets(https://github.com/{Author}/{Repository}/settings/secrets/actions). Finally, create a new release (https://github.com/{Author}/{Repository}/releases), and the package will be uploaded. How to Use GitHub Actions and Create Integration With pypi for Python Projects Introduction When I was creating my first Python package, dataclass-sqlalchemy-mixins ( github or pypi ), I encountered an interesting challenge: how to set up CI/CD on GitHub to ensure that nothing breaks when I push new changes and to automatically publish the code to PyPI . I discussed how to publish your own package using poetry in this article . dataclass-sqlalchemy-mixins github pypi CI/CD PyPI poetry article Typically, to verify any commit made to the master branch through a pull request, it is necessary to run tests. Additionally, using linters to check code style is beneficial, especially when multiple developers are working on the project. *CI (Continuous Integration) - The practice of automatically building and testing code changes after they are merged into the repository * *CI (Continuous Integration) - The practice of automatically building and testing code changes after they are merged into the repository *CD (Continuous Delivery) - The automated delivery of code to development or production environments* *CD (Continuous Delivery) - The automated delivery of code to development or production environments* Creating Workflows GitHub supports Actions, which are automated processes that can run one or more jobs, including CI/CD pipelines. CI/CD You can read more about them in the GitHub documentation . GitHub documentation Workflows are what GitHub will run according to your configuration. They should be located in the .github/workflows directory and be .yaml files. .github/workflows .yaml For example, if we want to run tests after each commit, we need to create a file named test.yml . test.yml I chose the name test for convenience and to indicate that this particular workflow is responsible for tests, but it can be anything. test name: "Test" on: pull_request: types: - "opened" - "synchronize" - "reopened" push: branches: - '*' workflow_dispatch: jobs: test: runs-on: ubuntu-latest strategy: fail-fast: false steps: - uses: actions/checkout@v4 - name: Set up Python 3.12 id: setup-python uses: actions/setup-python@v5 with: python-version: 3.12 - name: Cache poetry install uses: actions/cache@v4 with: path: ~/.local key: poetry-${{ steps.setup-python.outputs.python-version }}-1.7.1-0 - uses: snok/install-poetry@v1 with: version: 1.7.1 virtualenvs-create: true virtualenvs-in-project: true - name: Load cached venv id: cached-poetry-dependencies uses: actions/cache@v4 with: path: .venv key: venv-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('**/poetry.lock') }} - name: Install dependencies if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true' run: poetry install --no-interaction - name: Run tests run: poetry run pytest --cov-report=xml name: "Test" on: pull_request: types: - "opened" - "synchronize" - "reopened" push: branches: - '*' workflow_dispatch: jobs: test: runs-on: ubuntu-latest strategy: fail-fast: false steps: - uses: actions/checkout@v4 - name: Set up Python 3.12 id: setup-python uses: actions/setup-python@v5 with: python-version: 3.12 - name: Cache poetry install uses: actions/cache@v4 with: path: ~/.local key: poetry-${{ steps.setup-python.outputs.python-version }}-1.7.1-0 - uses: snok/install-poetry@v1 with: version: 1.7.1 virtualenvs-create: true virtualenvs-in-project: true - name: Load cached venv id: cached-poetry-dependencies uses: actions/cache@v4 with: path: .venv key: venv-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('**/poetry.lock') }} - name: Install dependencies if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true' run: poetry install --no-interaction - name: Run tests run: poetry run pytest --cov-report=xml Let me explain the key elements of the file: name - This is the name of the workflow. pull request - Specifies the types of pull requests that trigger the workflow. The recommended values by default are opened, synchronize, and reopened. You can read more about these types here. push - Specifies the branches that trigger the workflow. jobs - A workflow can consist of one or more jobs, which run in parallel by default but can also be configured to run sequentially. name - This is the name of the workflow. name - This is the name of the workflow. name pull request - Specifies the types of pull requests that trigger the workflow. The recommended values by default are opened, synchronize, and reopened. You can read more about these types here. pull request - Specifies the types of pull requests that trigger the workflow. The recommended values by default are opened , synchronize , and reopened . pull request opened synchronize reopened You can read more about these types here . here push - Specifies the branches that trigger the workflow. push - Specifies the branches that trigger the workflow. push jobs - A workflow can consist of one or more jobs, which run in parallel by default but can also be configured to run sequentially. jobs - A workflow can consist of one or more jobs, which run in parallel by default but can also be configured to run sequentially. jobs I’d like to discuss what happens during the execution of jobs with this code. First, the name of a job can be used to set up rulesets for the repository or specific branches. In this case, the only job that will run is test. The details of rulesets will be covered later. The runs-on key specifies which operating system the jobs will run on. runs-on Setting strategy: fail-fast: false means that GitHub will not cancel all in-progress and queued jobs in the matrix if any job in the matrix fails. fail-fast: false matrix The matrix parameter will be discussed later. matrix As you can see, there are a lot of similar codes in the uses key in the steps. For example: uses actions/setup-python@v5 actions/cache@v4 snok/install-poetry@v1 actions/setup-python@v5 actions/setup-python@v5 actions/cache@v4 actions/cache@v4 snok/install-poetry@v1 snok/install-poetry@v1 At first glance, the names of the actions might be confusing, but they are actually the paths to the GitHub repositories for these actions, excluding the @v part. @v For example, you can view the source code of the first action by visiting https://github.com/actions/setup-python . https://github.com/actions/setup-python The same can be done for the other actions. @v is used to set which version of the action is to use. @v The with key is used to pass parameters to an action. For example, if we want to install Python 3.12 , we pass python-version: 3.12 in the Set up Python 3.12 step. The same approach applies to other steps. Python 3.12 python-version: 3.12 Set up Python 3.12 In short, the test job performs the following steps: Sets up Python with the required version. Installs the Poetry 1.7.1 dependency using GitHub’s cache. Creates a Python .venv in the root directory if a cache is not found. The virtual environment cache depends on the Python version and poetry.lock. If the cached virtual environment is not found, the project's dependencies are installed. Runs the tests. Sets up Python with the required version. Python Installs the Poetry 1.7.1 dependency using GitHub’s cache. Poetry Creates a Python .venv in the root directory if a cache is not found. The virtual environment cache depends on the Python version and poetry.lock . Python poetry.lock If the cached virtual environment is not found, the project's dependencies are installed. Runs the tests. Using Services in Workflows The created workflow is quite simple and may not be suitable for testing more sophisticated applications or projects. This is because such projects often require services like databases or caching to function correctly. While it’s possible to write tests that mock requests to these services, I wouldn’t recommend it, as it can reduce the effectiveness of test coverage. I will demonstrate how to include a service in a workflow, using PostgreSQL as an example. PostgreSQL jobs: test: services: postgres: image: postgres:latest env: POSTGRES_PASSWORD: postgres ports: - 5432:5432 options: >- --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5 jobs: test: services: postgres: image: postgres:latest env: POSTGRES_PASSWORD: postgres ports: - 5432:5432 options: >- --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5 Adding a services key with a required service name allows to set up it during a job. services Moreover, you can pass settings to it similarly when using docker . docker Using Several Parameters During a Job Run Last but not least is the situation where you need to run a job using different parameters. For example, you may need to run a job with several specific Python versions, not just the latest, or with different dependencies. Python This can be particularly useful when you need to support backward compatibility with older dependencies. GitHub provides the strategy key to address this. The matrix parameter allows you to run a job with multiple values. GitHub matrix jobs: test: runs-on: ubuntu-latest strategy: fail-fast: false matrix: python-version: ["3.11", "3.12"] sqlalchemy-version: ["1.4.52", "2.0.31"] jobs: test: runs-on: ubuntu-latest strategy: fail-fast: false matrix: python-version: ["3.11", "3.12"] sqlalchemy-version: ["1.4.52", "2.0.31"] A job with a matrix set up this way will run each value of a parameter’s list against every other value. As a result, there will be four runs of the job with the following combinations of Python and SQLAlchemy versions, respectively: Python SQLAlchemy 3.11, 1.4.52 3.11, 2.0.31 3.12, 1.4.52 3.12, 2.0.31 3.11 , 1.4.52 3.11 1.4.52 3.11 , 2.0.31 3.11 2.0.31 3.12 , 1.4.52 3.12 1.4.52 3.12 , 2.0.31 3.12 2.0.31 You can access the parameters in a job run using ${{ matrix.python-version }} or ${{ matrix.sqlalchemy-version }} . ${{ matrix.python-version }} ${{ matrix.sqlalchemy-version }} - name: Set up Python ${{ matrix.python-version }} id: setup-python uses: actions/setup-python@v5 with: python-version: ${{ matrix.python-version }} - name: Install sqlalchemy ${{ matrix.sqlalchemy-version }} run: pip install sqlalchemy==${{ matrix.sqlalchemy-version }} - name: Set up Python ${{ matrix.python-version }} id: setup-python uses: actions/setup-python@v5 with: python-version: ${{ matrix.python-version }} - name: Install sqlalchemy ${{ matrix.sqlalchemy-version }} run: pip install sqlalchemy==${{ matrix.sqlalchemy-version }} Using GitHub Rulesets But sometimes, simply running jobs for your project isn't enough. There are situations where you may need to restrict merging commits to the master branch. While this might not be necessary when you're working alone on a project, it’s highly recommended when multiple people are working on the same codebase. This helps prevent situations where someone accidentally merges a commit that breaks the code, even if the pipelines fail. GitHub provides a useful tool for this called rulesets. You can access it from the repository settings menu at https://github.com/{Author}/{Repository}/settings/rules . https://github.com/{Author}/{Repository}/settings/rules After creating a new ruleset, scroll down to Require status checks to pass , turn it on, and specify which jobs are required to pass. Require status checks to pass To find a check, type the name of the required job, and GitHub will provide suggestions. Note that each job in the matrix will have its own name, allowing you to select them individually. Publishing to pypi In the last part, I would like to talk about how to create an integration with pypi and publish new versions with release. In order to do that, a new Yaml file with publishing workflows should be created. name: Upload Python Package to PyPi on release on: release: types: [published] permissions: contents: read jobs: deploy: runs-on: ubuntu-latest environment: name: pypi url: https://pypi.org/project/dataclass-sqlalchemy-mixins/ steps: - uses: actions/checkout@v4 - name: Set up Python uses: actions/setup-python@v3 with: python-version: '3.x' - uses: snok/install-poetry@v1 with: version: 1.7.1 virtualenvs-create: true virtualenvs-in-project: true - name: Build package run: poetry build - name: Publish package uses: pypa/gh-action-pypi-publish@27b31702a0e7fc50959f5ad993c78deac1bdfc29 with: user: __token__ password: ${{ secrets.PYPI_API_TOKEN }} name: Upload Python Package to PyPi on release on: release: types: [published] permissions: contents: read jobs: deploy: runs-on: ubuntu-latest environment: name: pypi url: https://pypi.org/project/dataclass-sqlalchemy-mixins/ steps: - uses: actions/checkout@v4 - name: Set up Python uses: actions/setup-python@v3 with: python-version: '3.x' - uses: snok/install-poetry@v1 with: version: 1.7.1 virtualenvs-create: true virtualenvs-in-project: true - name: Build package run: poetry build - name: Publish package uses: pypa/gh-action-pypi-publish@27b31702a0e7fc50959f5ad993c78deac1bdfc29 with: user: __token__ password: ${{ secrets.PYPI_API_TOKEN }} This workflow: Sets up Python. Installs the Poetry 1.7.1 dependency using GitHub’s cache. Build the source and wheel archives. Publishes the package to pypi Sets up Python . Python Installs the Poetry 1.7.1 dependency using GitHub’s cache. Poetry Build the source and wheel archives. Publishes the package to pypi Yes, it is as simple as it looks. You just need to add a PYPI_API_TOKEN to repository secrets( https://github.com/{Author}/{Repository}/settings/secrets/actions ). PYPI_API_TOKEN https://github.com/{Author}/{Repository}/settings/secrets/actions Finally, create a new release ( https://github.com/{Author}/{Repository}/releases ), and the package will be uploaded. https://github.com/{Author}/{Repository}/releases