Continuous integration in projects using monorepo

At the beginning of every project, we have to commit to some important decisions. Among the correct decisions we made at Code:Nation I can cite the use of Go language, the adoption of Clean Architecture and JAMStack and our choice to store the code in a monorepo at Github. In this post, I will write about the latter, and how we solved a common challenge that the monorepo architecture brings.

With the increasing complexity of modern projects, with microservices and distinct user interfaces consuming resources, teams need to decide between splitting the code into multiple repositories or using a monorepo approach. Some companies like Google and DigitalOcean adopted the monorepo, as we can see in the following posts:

• Why Google Stores Billions of Lines of Code in a Single Repository
• Cthulhu: Organizing Go Code in a Scalable Repo

In my humble opinion, the main reasons to use a monorepo are the simplicity of repository management and the reuse of code across the team.

But one of the challenges that this choice brings to the project is the potential complexity of automated build and deploy. Considering that all the code of a complex project is stored in the same repository, a change in a single file could trigger the build that can last minutes (or maybe hours) to be done. And this can be a real pain to the team, slowing the day to day productivity. In the DigitalOcean’s post that I cited before, they developed an internal solution, called gta (Go Test Auto), that is not open source. To solve this problem we created a similar solution but using Shell Script.

Currently, this is our project’s directory structure:

api = API and documentation chatbots = telegram, facebook and slack chatbots cli = codenation cli, used by developers to run the challenges cmd = utils and fixtures core = Go core packages, used by all the project docs = source code of internal docs (hosted at Github Pages) frontend = Vue.js project and templates used by Sam infra = configuration files used by staging and production servers lambda = lambda functions research = Python notebooks and other research assets sam = cli tool used by us to generate pages, include challenges and other admin tasks scripts = shell scrips used by CI/CD and other admin tasks web = ReactJS project (Signin, Signup, Forgot password) - IN PROCESS OF DEPRECATION workers = workers that consume SQS queues .drone.yml = CI/CD configuration file .goreleaser.yml = Goreleaser configuration file. Used to deploy the codenation-cli to Github, Homebrew docker-compose.yml = Docker configuration used by local and staging environments Gopkg.* = Go dependencies configuration files Makefile = build and admin tasks

We are using Drone.io as our CI/CD solution, that can we consider another good choice that we made. Our build pipeline can be seen in the image below:

And this is a sample of our .drone.yml config file:

As you can see, the step golang-build-api executes the script drone_go_build_api.sh, that is:

In the variable watch we store the list of directories that need to be monitored to our pipeline decide if the build needs to be run. This decision is made by the script shouldIBuild.sh:

In this script, the first thing being tested is the variable DRONE_DEPLOY_TO, that defines if the current execution is a deploy or a build. If so, the step should run. Otherwise, the script will check if one of the directories listed in the “watch” variable has been altered by this commit. If so, the step should run. The code of detectChangedFolders.sh is:

The same configuration exists in all our scripts used by Drone. That way, a change in the frontend won’t trigger the build of the API or the chatbots part of the code. Using this approach we decrease our build time from more than five minutes to a few seconds, depending on the change being commited in the repository.

I believe that this approach can be used with other tools different than Drone and I hope this can help more teams to stick to the monorepo decision with confidence.