Photo by on Blake Connally Unsplash core runs multiple static analysis tools against code and reports back to the . Teams use multiple languages, which means tools are written in different languages and released in different ways. TeamCI’s check builder must account for the variety while being flexible enough to adopt new tools and languages. I also wanted empowered users to improve their tools. That’s why the is open source. Anyone can throw code on GitHub and label it open source, but that’s not enough context for people to start hacking away. This post walks through the code with examples. Beyond that, it’s just a fun example of what you can get done with elbow grease and spattering of Bash (long live the king). TeamCI’s GitHub Checks API check builder High Level Design I wrote about building the TeamCI in a previous post. It’s a good place to start if you’re unfamiliar with how TeamCI works. _How I saved time, effort, and built an MVP._medium.com Building a Faster MVP with GitHub Checks This post covers the technical details behind how TeamCI actually works. Each check ( , , , etc) is a Docker image. Using Docker makes supporting any language or runtime trivial. Docker Compose manages the configuration. Docker containers run with test code mounted at and shared configuration at . is also the working directory. Alpine based images are preferred. rubocop shellcheck eslint /code /config /code TeamCI checks as seen in Github. TeamCI uses to parse results and create . None of the tools output TAP that fits TeamCI’s requirements. Luckily, they all output JSON so each Docker image includes a program to convert JSON output to TeamCI’s TAP. TAP output Github Check Annotations TAP annotations results in action on PR diff Each Docker image includes a wrapper script. The wrapper script handles setting CLI options, parsing JSON to TAP, and determines the exit code. Example responsibilities are checking for files in and adding an appropriate and options. Wrapper scripts are named with postfix. So 's wrapper becomes . The wrapper scripts may do more than just set options, they may also specify files to test. They're also the Dockerfile's . In a nutshell, they're responsible for correctly invoking the underlying tool, outputting TAP, and determining the exit code. /config --config FILE --output -tap shellcheck shellcheck-tap CMD TeamCI uses three exit codes: for success 0 for failure of any kind 1 for skips (say running a Ruby linter, but there are no Ruby files) 7 Skip results can only be determined if the tool communicates the result via an exit code or output. Not every tool does this, so exit code is used in this case. Unfortunately this comes across as a "pass" instead of "neutral" in the Github PR UI. 0 A Check Suite as a Buildkite pipeline. Note that each check is step in the pipeline. Steps execute in parallel. TeamCI executes checks via . Using Buildkite removes the need to manage infrastructure. TeamCI uses the which provides scalable infrastructure and a functioning Docker stack. GitHub Check Runs trigger a Buildkite build. Each check runs as a build step. TeamCI receives a webhook on each completed step and communicates the result back to GitHub. Buildkite Buildkite Elastic stack Each check build step clones the test code and associated config repo and exports environment variables. The does all this since it's shared between all build steps. This results in slim build scripts. The build scripts run the relevant check with and volumes mounted at and . ORG/teamci pre-command hook docker-compose run /code /config Acceptance tests cover all checks. Acceptance tests are written with . Tests cover the , , and exit cases along with custom config cases (e.g. a config file exists in ). Tests run through a BuildKite which exports relevant Buildkite environment variables and executes hooks. Commands are stubbed in tests by executables in and appending to , thus taking precedence over real executable. The test suite to use fixture code instead of actual git repositories. The test suite wouldn't work without it. bats 0, 1 7 /config emulation wrapper test/stubs/bin test/stubs/bin $PATH stubs the git command Check Code Walkthrough _This PR adds stylelint. No prior configuration is required. The standard configuration file is used if…_github.com Add stylelint by ahawkins · Pull Request #10 · teamci/builder Let’s walk through a specific check to see how this works in practice. The stylelint PR is a good introduction. Let’s begin with the : tests @test "stylelint: invalid repo fails" { buildkite-agent meta-data set 'teamci.repo.slug' 'stylelint/code' buildkite-agent meta-data set 'teamci.head_branch' 'fail' run test/emulate-buildkite script/stylelint [ $status -eq 1 ] [ -n "${output}" ] [ "$(echo "${output}" | grep -cF -- '--- TAP')" -eq 2 ] # Test for annotation keys echo "${output}" | grep -qF 'filename:' echo "${output}" | grep -qF 'blob_href:' echo "${output}" | grep -qF 'start_line:' echo "${output}" | grep -qF 'end_line:' echo "${output}" | grep -qF 'warning_level:' echo "${output}" | grep -qF 'message:' echo "${output}" | grep -qF 'title:' [ -n "$(buildkite-agent meta-data get 'teamci.stylelint.title')" ]} The first two lines are setup methods. TeamCI passes the git repo, branch, and commit via build metadata so the check code knows which code to clone. The two lines set values that map to a git fixture. live in . to implement the pattern. Fixtures test/fixtures/$REPO/$BRANCH git is stubbed The next line executes the check through the buildkite emulation wrapper, followed by assertions on exit code and output. This test asserts TAP output with correctly shaped annotations. The test covers the remaining success and configuration file cases. Here’s the test for a user provided configuration file: @test "stylelint: config file exists" { buildkite-agent meta-data set 'teamci.repo.slug' 'stylelint/code' buildkite-agent meta-data set 'teamci.head_branch' 'config_file' buildkite-agent meta-data set 'teamci.config.repo' 'stylelint/config' buildkite-agent meta-data set 'teamci.config.branch' 'config_file' run test/emulate-buildkite script/stylelint # The configured options should make the failing fixture pass [ $status -eq 0 ] [ -n "${output}" ] [ -n "$(buildkite-agent meta-data get 'teamci.stylelint.title')" ]} The structure is the same except the provided configuration repo fixture. These test use fixture that fail using the default config, but pass with custom configuration. Thus, the expected result is . 0 The PR includes the expected code changes: A Docker image built from stylelint /stylelint A wrapper in stylelint-tap /stylelint A for converting JSON to in . tapify.rb tap /stylelint Additions to docker-compose.yml Additions to Makefile Tests in with corresponding fixtures. test/acceptance The “valid repo passed test” may also litter the fixture with irrelevant code files. Stylelint test stylesheets (e.g. ), so a stray Ruby file in the code directory should not cause a failure. Testing this depends on the tool. Stylelint requires an explicit lists of files, so a glob is used. However this isn't the case for something like PHP CodeSniffer which detects php files itself. **/*.css Wrap Up Adding a new check is straight forward once you understand the structure. I start by copy and pasting from the most recent check, since they’re all similar enough. Then I tweak the tests, wrapper, and . I start by testing exit code 1. This allows me to test the tools work as expected and to inspect the JSON output. It's easy to tweak afterwards. Then it's head down grunt work to create passing fixtures, config file cases, and the Docker image. -tap tapify.rb tapify.rb Adding a new tool is the easiest when: The tool automatically detects testable files An explicit config file may be provided The tool prints JSON to standard out Errors and extraneous information print to standard error The tool signals (via exit code or output) that no testable files were found There’s some way to exclude files I had fun writing the builder. It’s a straight forward task, plus the acceptance test suite gives me plenty of confidence. The experience also surfaced my preferred semantics in these tools. So what do you think? Want to add your own check? Feel free to send a PR and make TeamCI more useful for you. If not, then at least you learned that TDD with Bash (long live the king) is possible — and fun. during the beta. Test TeamCI for free _Automatically test team code repos against team coding standards._teamci.co TeamCI - Enforce Team Code Standards
