Continuous Integration for a Monorepo on CircleCI Example

The problem

Microservices were very popular a couple of years ago. Now we are all running towards monorepos! With microservices setting up Continuous integration was pretty straightforward and when they run they did relatively fast, unless you had a massive application.

While monorepos are easy to set up using tools like Turborepo the problem shows as soon as you try to deploy you application and set up a CI/CD pipeline. If you merge a few bigger apps suddenly it takes ages (we are talking about billable ages) for the pipeline to complete.

How to avoid it?

CircleCI dynamic config to the rescue

Consider the below monorepo. It consists of a design system and two applications. All applications have linting, types tests, unit tests, cypress and they even use loki visual regression tests!

So it’s just slow to run everything if you made a tiny change to app2 for example!

.
├── .circleci
│   ├── config.yml
│   └── continue_config.yml
├── design-system
│   ├── more_directories
├── app1
│   ├── more_directories
├── app2
│   ├── more_directories

Note that we have two config files.

The first one config.yml is super easy, it just checks which paths changed using a build in Orb.

I added a few comments so it’s hopefully easy to read.

version: 2.1

# this allows you to use CircleCI's dynamic configuration feature
setup: true

# the path-filtering orb is required to continue a pipeline based on
# the path of an updated fileset
orbs:
  path-filtering: circleci/[email protected]

workflows:
  # the always-run workflow is always triggered, regardless of the pipeline parameters.
  always-run:
    jobs:
      - path-filtering/filter:
          name: check-updated-files

          # Test which path is updated and set the parameter for continue_config
            design-system/.* run-design-system-job true
            app1/.* run-app1-job true
            app2/.* run-app2-job true

          # Compare changes of the branch with main branch
          base-revision: main

          # this is the path of the configuration we should trigger once
          # path filtering and pipeline parameter value updates are
          # complete. In this case, we are using the parent dynamic
          # configuration itself.

          config-path: .circleci/continue_config.yml

All the fun starts in the below file, which will dynamically run only the jobs for the affected files. If you are using turborepo remote cache you might want to consider a single step build, which takes seconds if you have builds cached.

version: 2.1

orbs:
  maven: circleci/[email protected]

# the default pipeline parameters, which will be updated according to
# the results of the path-filtering orb
parameters:
  run-design-system-job:
    type: boolean
    default: false
  run-app1-job:
    type: boolean
    default: false
  run-app2-job:
    type: boolean
    default: false

# here we specify our workflows, most of which are conditionally
# executed based upon pipeline parameter values. Each workflow calls a
# specific job defined above, in the jobs section.

# jobs are omited for this purpose

workflows:
  # when pipeline parameter, run-design-system-job is true, the
  # jobs is triggered.
  run-design-system-job:
    when: << pipeline.parameters.run-design-system-job >>
    jobs:
      - install-dependencies
      - build
      - unit_tests
      - visuals

  run-app1-job:
    when: << pipeline.parameters.run-app1-job >>
    jobs:
      - install-dependencies
      - build
      - unit_tests

  run-app2-job:
    when: << pipeline.parameters.run-app2-job >>
    jobs:
      - install-dependencies
      - build
      - unit_tests

  run-integration-tests:
    when:
      or: [<< pipeline.parameters.run-app1-job >>, << pipeline.parameters.run-design-system-job >>]
    jobs:
      - cypress_integrations

So we sped up our pipeline by not running everything every time!

We just run cypress tests every time on the run-integration-tests workflow. When a change to the design system or app1 (which uses design-system) is made to make sure both play nicely with each other as you might have broken something accidentally changing a component.

To speed the last workflow and cypress we could attach the workspace but that is a separate topic :)