Inside TeamCI's Check Code

Photo by Blake Connally on Unsplash

TeamCI’s core runs multiple static analysis tools against code and reports back to the GitHub Checks API. Teams use multiple languages, which means tools are written in different languages and released in different ways. TeamCI’s check builder must account for the variety while being flexible enough to adopt new tools and languages. I also wanted empowered users to improve their tools. That’s why the check builder is open source. Anyone can throw code on GitHub and label it open source, but that’s not enough context for people to start hacking away. This post walks through the code with examples. Beyond that, it’s just a fun example of what you can get done with elbow grease and spattering of Bash (long live the king).

High Level Design

I wrote about building the TeamCI in a previous post. It’s a good place to start if you’re unfamiliar with how TeamCI works.

Building a Faster MVP with GitHub Checks_How I saved time, effort, and built an MVP._medium.com

This post covers the technical details behind how TeamCI actually works.

Each check (rubocop, shellcheck, eslint, etc) is a Docker image. Using Docker makes supporting any language or runtime trivial. Docker Compose manages the configuration. Docker containers run with test code mounted at /code and shared configuration at /config. /code is also the working directory. Alpine based images are preferred.

TeamCI checks as seen in Github.

TeamCI uses TAP output to parse results and create Github Check Annotations. None of the tools output TAP that fits TeamCI’s requirements. Luckily, they all output JSON so each Docker image includes a program to convert JSON output to TeamCI’s TAP.

TAP annotations results in action on PR diff

Each Docker image includes a wrapper script. The wrapper script handles setting CLI options, parsing JSON to TAP, and determines the exit code. Example responsibilities are checking for files in /config and adding an appropriate --config FILE and --output options. Wrapper scripts are named with -tap postfix. So shellcheck's wrapper becomes shellcheck-tap. The wrapper scripts may do more than just set options, they may also specify files to test. They're also the Dockerfile's CMD. In a nutshell, they're responsible for correctly invoking the underlying tool, outputting TAP, and determining the exit code.

TeamCI uses three exit codes:

• 0 for success
• 1 for failure of any kind
• 7 for skips (say running a Ruby linter, but there are no Ruby files)

Skip results can only be determined if the tool communicates the result via an exit code or output. Not every tool does this, so exit code 0 is used in this case. Unfortunately this comes across as a "pass" instead of "neutral" in the Github PR UI.

A Check Suite as a Buildkite pipeline. Note that each check is step in the pipeline. Steps execute in parallel.

TeamCI executes checks via Buildkite. Using Buildkite removes the need to manage infrastructure. TeamCI uses the Buildkite Elastic stack which provides scalable infrastructure and a functioning Docker stack. GitHub Check Runs trigger a Buildkite build. Each check runs as a build step. TeamCI receives a webhook on each completed step and communicates the result back to GitHub.

Each check build step clones the test code and associated ORG/teamci config repo and exports environment variables. The pre-command hook does all this since it's shared between all build steps. This results in slim build scripts. The build scripts run the relevant check with docker-compose run and volumes mounted at /code and /config.

Acceptance tests cover all checks. Acceptance tests are written with bats. Tests cover the 0,, 1, and 7 exit cases along with custom config cases (e.g. a config file exists in /config). Tests run through a BuildKite emulation wrapper which exports relevant Buildkite environment variables and executes hooks. Commands are stubbed in tests by executables in test/stubs/bin and appending test/stubs/bin to $PATH, thus taking precedence over real executable. The test suite stubs the git command to use fixture code instead of actual git repositories. The test suite wouldn't work without it.

Check Code Walkthrough

Add stylelint by ahawkins · Pull Request #10 · teamci/builder_This PR adds stylelint. No prior configuration is required. The standard configuration file is used if…_github.com

Let’s walk through a specific check to see how this works in practice. The stylelint PR is a good introduction. Let’s begin with the tests:

@test "stylelint: invalid repo fails" {	buildkite-agent meta-data set 'teamci.repo.slug' 'stylelint/code'	buildkite-agent meta-data set 'teamci.head_branch' 'fail'	run test/emulate-buildkite script/stylelint	[ $status -eq 1 ]	[ -n "${output}" ]	[ "$(echo "${output}" | grep -cF -- '--- TAP')" -eq 2 ]	# Test for annotation keys	echo "${output}" | grep -qF 'filename:'	echo "${output}" | grep -qF 'blob_href:'	echo "${output}" | grep -qF 'start_line:'	echo "${output}" | grep -qF 'end_line:'	echo "${output}" | grep -qF 'warning_level:'	echo "${output}" | grep -qF 'message:'	echo "${output}" | grep -qF 'title:'	[ -n "$(buildkite-agent meta-data get 'teamci.stylelint.title')" ]}

The first two lines are setup methods. TeamCI passes the git repo, branch, and commit via build metadata so the check code knows which code to clone. The two lines set values that map to a git fixture. Fixtures live in test/fixtures/$REPO/$BRANCH. git is stubbed to implement the pattern.

The next line executes the check through the buildkite emulation wrapper, followed by assertions on exit code and output. This test asserts TAP output with correctly shaped annotations.

The test covers the remaining success and configuration file cases. Here’s the test for a user provided configuration file:

@test "stylelint: config file exists" {	buildkite-agent meta-data set 'teamci.repo.slug' 'stylelint/code'	buildkite-agent meta-data set 'teamci.head_branch' 'config_file'	buildkite-agent meta-data set 'teamci.config.repo' 'stylelint/config'	buildkite-agent meta-data set 'teamci.config.branch' 'config_file'	run test/emulate-buildkite script/stylelint	# The configured options should make the failing fixture pass	[ $status -eq 0 ]	[ -n "${output}" ]	[ -n "$(buildkite-agent meta-data get 'teamci.stylelint.title')" ]}

The structure is the same except the provided configuration repo fixture. These test use fixture that fail using the default config, but pass with custom configuration. Thus, the expected result is 0.

The PR includes the expected code changes:

• A stylelint Docker image built from /stylelint
• A stylelint-tap wrapper in /stylelint
• A tapify.rb for converting JSON to tap in /stylelint.
• Additions to docker-compose.yml
• Additions to Makefile
• Tests in test/acceptance with corresponding fixtures.

The “valid repo passed test” may also litter the fixture with irrelevant code files. Stylelint test stylesheets (e.g. **/*.css), so a stray Ruby file in the code directory should not cause a failure. Testing this depends on the tool. Stylelint requires an explicit lists of files, so a glob is used. However this isn't the case for something like PHP CodeSniffer which detects php files itself.

Wrap Up

Adding a new check is straight forward once you understand the structure. I start by copy and pasting from the most recent check, since they’re all similar enough. Then I tweak the tests, -tap wrapper, and tapify.rb. I start by testing exit code 1. This allows me to test the tools work as expected and to inspect the JSON output. It's easy to tweak tapify.rb afterwards. Then it's head down grunt work to create passing fixtures, config file cases, and the Docker image.

Adding a new tool is the easiest when:

• The tool automatically detects testable files
• An explicit config file may be provided
• The tool prints JSON to standard out
• Errors and extraneous information print to standard error
• The tool signals (via exit code or output) that no testable files were found
• There’s some way to exclude files

I had fun writing the builder. It’s a straight forward task, plus the acceptance test suite gives me plenty of confidence. The experience also surfaced my preferred semantics in these tools.

So what do you think? Want to add your own check? Feel free to send a PR and make TeamCI more useful for you. If not, then at least you learned that TDD with Bash (long live the king) is possible — and fun. Test TeamCI for free during the beta.

TeamCI - Enforce Team Code Standards_Automatically test team code repos against team coding standards._teamci.co