Which CI is Best For My Team?

At the beginning of developing and building a new application or product, before writing any business logic, it's necessary to make some difficult decisions. These decisions can include which cloud hosting platform to use, which continuous integration (CI) tool to use, and various development configuration choices. These decisions can dictate the speed and success of your engineering efforts. Therefore, choosing the right CI tool can make or break the velocity and quality of a team's code production.

Like most engineering problems, a discussion of the trade-offs and constraints as well as an understanding of your team’s specific requirements can help you to pick the best CI tool for you. 

In this article, I 'll discuss how to choose the best CI for your team.

Trade-offs and Constraints

Before I discuss the details of specific CI tools, here are some dimensions to help you to make an informed decision for your team. Some of these considerations may be very important to you and your team; others may not be important at all.

Speed

Does the tool/service offer parallelization? How does cost scale with speed? I expect most developers would agree that the speed of the test and iteration cycle is one of the most important considerations when evaluating a CI tool.

Features

An important feature is test/prod parity. Does the particular tool offer identical staging and production environments to ensure parity?

Ease of setup and configuration is a commonly requested feature of CI tooling. All the amazing features are great, but they are useless if they are locked behind a wall of incomplete or confusing documentation and complex configuration.

UI/UX

As per the point above, usability is another important aspect of the product. Will new developers on your team be able to learn and interact with your CI system? How steep is the learning curve?

Cost

Lower is better assuming feature parity. Some tools are prohibitively expensive, especially for teams working on open-source projects. Other tools are free or available at a reduced cost for these teams.

Support

How easy is it to contact a support representative? Some teams might find this invaluable while others may not. What is the quality of their documentation? Do they provide examples of common applications and operations? Are those examples available in multiple languages?

How widely used is the tool? Does it have a vibrant community of fellow developers trying to solve similar problems with the tool? 

Integrations

Does it play nice with Docker, AWS, or GCP services? In a modern dockerized development world, container support is vital for most teams. Does the service integrate with GitHub nicely?

Are fragile and complex hacks required to make your multi-faceted app behave and test as expected? How easy is it to attach dependent services?

CI Tool Options

Here are some CI tool options that are currently leading the market, as well as some pros and cons of each.

Circle CI

Pros

• Capabilities to construct complicated sequential build processes to handle complex use cases.
• Large community.
• Online examples and troubleshooting available.
• Github integration.
• Easy setup.
• Performant.

By using Circle CI, users can add an image that is in a predetermined state for testing via containerization.

Cons

• Poor UI/UX - Circle CI has a number of rough edges (For example, Circle CI does not automatically refresh the workflow page when a task in the pipeline completes or fails.) There are also a number of other small workflow issues that could be improved.
• Downtime/denial of service - I've personally experienced some very inconvenient service outages. While looking at the up-time records for the last year, I'm sure others have as well.
• Aggressive GitHub permissions - This is cited by Circle CI users as one of the major cons of using the product.

Travis CI

Pros

• Free for open-source projects.
• CI environment provides multiple run-times, data stores, and so on.
• Hosting projects on Travis CI means you can effortlessly test your library or applications against multiple run-times and data stores without installing them locally. 

Cons

• You get what you pay for - This is a great tool for hobby projects and small open-source teams but may fail to support the complex needs of a larger engineering team or project.
• You have to pay to add Travis CI to a private Github repository.
• Configuration for complex testing suites and workflows may be challenging.

AWS CodePipeline

Pros

• CodePipeline offers amazing integrations with existing AWS products with plugins similar to AWS Lambda development.
• No servers to provision or set up.
• Offers a fully-managed continuous delivery service that connects to your existing tools and systems.
• Parallel execution.
• Simple to set up and relatively easy to use.
• AWS IAM access controls and permissions in a large and dynamic team.
• Enormous array of features and customized workflows to fit virtually any development team.
• Option to pay by usage.
• No upfront fees.

Cons

• Cost - AWS services can be expensive.
• Platform lock-in.
• Lack of integration with source control providers other than GitHub or AWS CodeCommit.

GitHub Actions

Pros

• Separation of concerns and easy debugging - GitHub actions are just consecutive docker runs. This makes it easy to reason about and debug.
• The individual actions in a workflow are isolated by default, making it much easier to reason about and debug.
• Vibrant developer community available as a resource.
• GitHub Actions are a more generalized tool - they do many things other than run CI.

Cons

• Environmental configurations for testing staging/DB and prod/DB parity is challenging
• Due to security reasons there is no support for pull requests from forks. If you're using a workflow that involves forks, that makes GitHub actions largely unusable as CI/CD tool.
• Limited quality and breadth of published GitHub actions.
• Less actionable documentation with limited real-world examples. This is a subjective claim but supported by others in the developer community. Check for yourself by comparing GitHub Actions documentation with the more detailed documentation provided by Heroku or AWS.

Heroku Pipelines

Pros

• Amazing documentation and examples
• Natural integration with Heroku hosted apps and a Heroku Pipeline
• Consolidated Heroku billing
• Great UI and developer experience
• Lower level of complexity for attaching dependent services, assuming the dependency is in Heroku's Add-On catalog (e.g. Postgres) compared to the roll-your-own mentality of Circle CI
• No waiting in the queueBest-in-class pipeline integration

Cons

• It is not possible to use Heroku CI to test container builds.
• Cannot use Heroku CI without using Heroku Pipelines or dynos for hosting
• Heroku CI costs $10/month

Which is the Best CI For You?

I've discussed some trade-offs and constraints as well as some key details specific to common continuous integration services that engineering teams should consider before selecting a CI tool.

Here’s my opinion: My current favorite is Heroku CI because I’m already using the Heroku universe (hosting, Heroku Pipelines) at my firm. I like Heroku CI because it runs tests on Performance Dynos in a disposable Heroku app on each merge to master. So, I don't have to wait for other developers' apps to build or tests to run. It also seamlessly integrates with Github and Heroku Pipelines. Generally speaking, Heroku CI is a real pleasure to work with.

If you aren't using Heroku, then my second favorite is Circle CI. Despite some UI/UX and quality of life issues, it is versatile, quick to set up, and cloud-platform agnostic. You can configure complex workflows with environmental set-up, integration testing, tear-down, and parallelized tasks with plenty of detailed tutorials and a large community of users. 

As an engineering team lead, it's important for you to make an informed decision based on your team’s constraints and requirements, knowledge of the market, and trade-offs to pick the CI tool that is right for you.