Semantic release with Python, Poetry & GitHub Actions 🚀

9 0

Dec 20 '20

I'm planning to add a few features to Dr. Sven thanks to some interest from my colleagues. Before doing so, I needed to set up some proper release management as I will be introducing some breaking changes as well as automated deployment.

To manage this automatic versioning, the obvious choice is semantic versioning/semantic release which automatically increments a version number based on commit conventions.

Here's how ...

What I want to achieve

Automatic version number increment of releases.
Version numbers based on conventional commit format (Semantic Versioning).
Automatically create a release object on GitHub.
Only allow merging to main when tests and linting are passing.
Perform a release following a merge to the main branch.

TL;DR

If you don't want to read the detail you should be able to implement this in under 10 minutes.

Install this GitHub bot to enforce semantic PRs: https://github.com/apps/semantic-pull-requests.
Set the same version number in pyproject.toml and __init__.py. For example:

./pyproject.toml
```
[tool.poetry]
name = "your-project"
version = "0.0.1"
```
src_folder/__init__.py
```
__version__ = '0.0.1'
```

Add Semantic Release section to pyproject.toml:

[tool.semantic_release]
version_variable = [
    "src_folder/__init__.py:__version__",
    "pyproject.toml:version"
]
branch = "main"
upload_to_pypi = false
upload_to_release = true
build_command = "pip install poetry && poetry build"

Create the file .github/workflows/ci.yml within your repo, and copy the contents of this file into it to set up your CI action: https://github.com/MeStrak/dr-sven/blob/main/.github/workflows/ci.yml.

Done! Read on if you would like a little more explanation.

Background

Before diving into the code, let's do a light review of the principles behind Semantic Versioning to understand how the version numbers are generated.

Conventional Commits

This is a specification designed to ensure that commit messages are human-readable. Developers using this convention use specific commit prefix text such as fix: for commits which fix a bug, and feat: for commits which add a new feature. The most common convention is the convention developed by the Angular team which includes the following keywords:

build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.

A scope can also be added in parentheses. For example: feat(blog): add code parsing.

BREAKING CHANGE can also be added into the commit text to indicate that the commit is not backwards compatible.

These commit conventions are then used in the Semantic Versioning specification to calculate the version number.

Have a look at the full specification here.

Also, here are two tools worth investigating to help enforce this convention in all commit messages:

Semantic Versioning

Semantic Versioning, or SemVer is a specification defining a clear, logical version numbering system for an API. In this case, I'm actually using it to version an application with no external API, but the specification still holds very well because I have a requirement to indicate whether a new release is backwards compatible with configuration files.

I would recommend reading the full specification, but here is an excellent summary taken directly from the specification:

Given a version number MAJOR.MINOR.PATCH, increment the:

MAJOR version when you make incompatible API changes,

MINOR version when you add functionality in a backwards compatible manner, and

PATCH version when you make backwards compatible bug fixes.

Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.

Take a look at the full specification here.

Various packages existing to calculate a SemVer version number based on conventional commit messages. We'll be using Semantic Release to do this, and perform a release at the same time.

Semantic Release

Semantic Release is a tool which automatically sets a version number in your repo, tags the code with the version number and creates a release (for example publishing it to NPM, or PyPi).

This is done using the contents of Conventional Commit style messages to generate a version number. For example, fix: will bump the PATCH version, feat: will bump the minor version, and a message containing BREAKING CHANGE will bump the major version to indicate that the release is not fully backwards compatible with the previous version.

The original (as far as I know) semantic-release package was created for node projects and can be found here.

We'll be using the Python implementation of that project, Python Semantic Release.

Implementation

Now that we have covered the basic concepts, let's look at how to implement them. We need to do the following things, as listed in the TL;DR section but this time with a bit more explanation:

Ensure Conventional Commits are used.
Configure Semantic Release in pyproject.toml.
Configure a basic quality CI step using GitHub Actions.
Add a release step using Python Semantic Release.

Ensure Conventional Commits are used

The use of Conventional Commits is central to our whole versioning strategy. If it's not used then nothing will work. To enforce use of this type of commit on PRs I'll be using a GitHub bot which is easy-peasy to install: https://github.com/apps/semantic-pull-requests.

Just install that bot for your GitHub project, and you'll automatically see a new check for any new PR created on your project. If the PR doesn't contain commits which comply with the Conventional Commit specification then the check will fail.

As a later enhancement, I will add a pre-commit hook to the repo to check the format of local commit messages because it can be pretty annoying for developers to discover issues like this only when a PR is raised.

The first step is done! Easy right?

Configure Semantic Release in pyproject.toml

Semantic Release can be configured in pyproject.toml according to your requirements. The main things to note here:

As we're using Poetry the version variable is managed in both __init.py__ and pyproject.toml.
Branch is set to main, as we're using the GitHub flow branching strategy. We want our releases to be performed using code on the main branch, which is where all code is merged to for production deployment. If you use a different strategy just change the name.
upload_to_pypi has been set to false because Dr. Sven is not published on PyPi, so this is irrelevant for me.

[tool.semantic_release]
version_variable = [
    "dr_sven/__init__.py:__version__",
    "pyproject.toml:version"
]
branch = "main"
upload_to_pypi = false
upload_to_release = true
build_command = "pip install poetry && poetry build"

Configure a basic quality CI step using GitHub Actions

If you're reading this, I assume you're already aware that GitHub Actions is GitHub's CICD workflow solution. Workflows are configured in YAML files stored in the .github/workflows folder in your repo.

I'm using the nice and simple GitHub flow branching strategy, where all bugfixes and features have their own separate branch, and when complete each branch is merged to main and deployed.

The Dr. Sven CI pipeline will consist of 2 main stages, known as jobs:

Quality: runs linting and automated tests on the code, preventing merging if any of these steps fail.
Release: runs following the successful quality job when code has been merged to the main branch.

The full workflow file can be found here. Although the structure and syntax are simple, let's break it down to make sure we understand each part.

The header defines that the CI workflow will run whenever there is a direct push to main, or a pull request to merge another branch to main.

name: CI
on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

The quality job will use an ubuntu runner instance. The runner is the name of the application which runs GitHub Actions jobs, such as unit tests in almost the same way as you would run them on your development PC.

The checkout action will checkout the code so that it is available for you to use in the executing runner.
setup-python will install Python on the runner.

jobs:
  Quality:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - uses: actions/setup-python@v2
      with:
        python-version: 3.8

Next we install Poetry, check the version (not required but sometimes useful to see in the logs), and then install the dependencies using Poetry.

    - name: Install Python Poetry
      uses: abatilo/actions-poetry@v2.1.0
      with:
        poetry-version: 1.1.2
    - name: Configure poetry
      shell: bash
      run: python -m poetry config virtualenvs.in-project true
    - name: View poetry version
      run: poetry --version
    - name: Install dependencies
      run: |
        python -m poetry install

Finally, run Flake8 linting and pytest unit tests on the Poetry virtualenv. Other linting or unit test frameworks could also be used here.

    - name: Lint with flake8
      run: |
        # stop the build if there are Python syntax errors or undefined names
        python -m poetry run flake8 . --exclude .venv --count --select=E9,F63,F7,F82 --show-source --statistics
        # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
        python -m poetry run flake8 . --exclude .venv --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
    - name: Test with pytest
      run: |
        python -m poetry run python -m pytest -v tests

That's the quality step ready, simple 🤓.

Add a release step using Python Semantic Release

Now that our tests have run and passed the quality stage, it's time to release our new version.

The release job will only run after a successful quality stage. We also check the following:

This is a push to main. This prevents a release from being performed on PRs before they are merged.
The commit message doesn't contain chore(release), the message used by Semantic Release when a release is performed. Without this we would have an endless loop of releases, each release triggered by the previous release.

  Release:
    needs: Quality
    # https://github.community/t/how-do-i-specify-job-dependency-running-in-another-workflow/16482
    if: github.event_name == 'push' && github.ref == 'refs/heads/main' && !contains(github.event.head_commit.message, 'chore(release):')

Now install Python on the runner, checkout the code, then install and run python-semantic-release publish, which will:

Bump the version number in both version files.
Tag the code with that version.
Create a GitHub release object.
Commit the updated files to the main branch.

    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-python@v2
        with:
          python-version: 3.8
      - name: Checkout code
        uses: actions/checkout@v2
      - name: Semantic Release
        run: |
          pip install python-semantic-release
          git config user.name github-actions
          git config user.email github-actions@github.com
          semantic-release publish

Following this simple step, our releases are now created and the workflow is complete. It could even be simplified further by using the Python Semantic Release GitHub Action.

Final result

That's it!

We can now see all Actions workflow runs from the GitHub actions page. Looking at a specific run we can see the successful quality and release steps:

A release has been created, and assets can be downloaded for that specific version:

Of course some tweaks can always be made to improve the workflow, but now we have a nice lightweight CI workflow in place to support automated testing and release as I develop Dr. Sven further.

Possible Improvements

As GitLab say: everything is in draft. In keeping with that, of course there are things that I can and will improve in this CI workflow. Here are some of them:

Lock the main branch: currently I can commit directly to the main branch, which is bad practice. As I'm currently the only project contributor it's OK for now, but I need to lock the branch and configure Semantic Release to use a specific token with permission to push to that branch when updating the version numbers.
CommitLint within the CI: currently I've used the Semantic Commit bot which is great to allow you to see that your PR contains compliant commits from the PR page. However, technically, I could bypass this and commit something without compliant commit messages directly to main, skipping the release. This can be prevented by adding a CommitLint step within the pipeline (also locking the main branch will help).
Integration testing within the CI: currently only unit tests are executed. With this testing strategy, all integration testing must be performed manually on my dev machine or in AWS. Not very DevOpsy, only really acceptable in my simple, single developer environment.
Pre-commit hook to ensure Conventional Commits are used: prevent any commits from being made unless the developer is using the specified commit style. This prevents developer frustration by avoiding the annoying discovery at PR time that you didn't follow the project conventions.

Thanks for reading! Please leave a comment, I'm happy to answer any questions you may have.

Fin.