How managing source code became transparent This blog post is part of a series where I share our migration from monolithical applications (each with their own source repository) deployed on AWS to a distributed services architecture (with all source code hosted in a monorepo) deployed on Google Cloud Platform. Part 1: “ A monorepo, GitHub Flow and automation FTW ” Part 2 (this post): “One vs. many — Why we moved from multiple git repos to a monorepo and how we set it up” Part 3: “ A (mostly) automated release process ” Part 4: “ Our approach to software development consistency ” Part 5 (coming soon): “Debug microservices locally” Multiple repositories means multiple everything Let’s list some of the things we need to manage with a repository: Dependencies Test configuration Pull request templates Pull requests / labels ESLint Prettier Deployment and release scripts For some things, such as managing dependencies, services like may help. However, if a dependency releases a new major version, you have to manually apply that to all repositories and run the tests. Greenkeeper It became clear that none of us enjoyed any of these maintenance tasks and we rather spend the time to make our market research chatbots more valuable to our customers. Dependencies Lerna Our code is mainly written in Javascript, which brought us to look at . Lerna Lerna is a tool for managing JavaScript projects with multiple packages. We decided to take this one step further. Instead of managing our npm packages only, we configured Lerna to also manage our services, which live in the same monorepo. Our monorepo directory structure is as follows: .├── lerna.json├── package.json├── packages└── services The file is straight forward: lerna.json {"lerna": "2.4.0","npmClient": "yarn","useWorkspaces": true, // See "Yarn Workspaces" below"packages": ["packages/*", "services/*"],"version": "independent"} With this configuration, our services can depend on packages and Lerna takes care of symlinking them. For example, we can run within the directory and lerna symlinks properly. No more dealing with . yarn add package-z services/service-a package-z yarn link NPM scoped packages To Lerna, and are considered packages. Most lerna commands support the , but that only works if you follow a strict naming convention for your properties in the files. packages/* services/* --scope flag name package.json We decided to separate packages from services by using different . Since get deployed to NPM, they use the company default scope (e.g. ). Services in use a scope. Packages and services are further prefixed with vs to distinguish between different types of packages and services. scoped packages packages/* @my-company services/* @my-company-services web-* svr-* Yarn Workspaces Lerna is great at managing inter-dependencies and running npm scripts or even arbitrary commands across all packages or subsets thereof. However, each package and service by default gets their own folder. That is a lot of duplication… node_modules The fine folks who give us Yarn released “Workspaces” and kindly enough blogged how to use it with Lerna: https://yarnpkg.com/blog/2017/08/02/introducing-workspaces/ Besides the in the , you also have to add to your root file. That’s it. "useWorkspaces": true lerna.json "workspaces": ["packages/*", "services/*"] package.json Now when you run and , your root folder contains close to all npm packages you ever need. This saves both time and disk space. The following showcases the difference between not using Yarn Workspaces and using it in our monorepo. The stats are based on 20 packages managed by Lerna, run on a 2016 MacBook Pro. yarn lerna bootstrap node_modules Without Yarn Workspaces +-----------------+--------+| Command | Time |+-----------------+--------+| yarn install | 13.23s || lerna bootstrap | 72.33s |+-----------------+--------+ This adds 96,112 files at a total of 666.4mb to disk. With Yarn Workspaces +-----------------+--------+| Command | Time |+-----------------+--------+| yarn install | 17.26s || lerna bootstrap | 3.85s |+-----------------+--------+ This adds 32,008 files at a total of 267.1mb to disk. Conclusion Waiting an extra 4 seconds to install the root packages is worth the savings we get with . With a bit of caching on the continuous integration server, things look even better, but I’m getting ahead of myself. lerna bootstrap Test Configuration We use , but decided to let Lerna manage the test runner instances. (FYI, Jest comes with a that may be useful in your use case.) Jest multi-project-runner In our case, we like the flag Lerna provides to run commands in certain directories only.More importantly, we have a variety of packages and services, some can be used in Node.js, others in the browser and some are isomorphic. --scope To accommodate for that, we have the following Jest configuration setup: .├── jest.config.js├── packages│ ├── iso-package│ │ ├── jest.config.js│ ├── svr-package│ │ └── jest.config.js│ └── web-package│ └── jest.config.js├── services│ ├── svr-service│ │ └── jest.config.js│ ├── web-service│ └── jest.config.js└── tests-setup├── polyfill.js└── setup.js The root-level contains the base Jest configuration we apply across all packages and services. It looks something like that: jest.config.js // jest.config.jsmodule.exports = {collectCoverageFrom: ['**/*.js'],resetMocks: true,verbose: true} Web packages and services A package or service uses the following within its root directory: web-* jest.config.js // packages/web-*/jest.config.js or services/web-*/jest.config.jsconst jestBase = require('../../jest.config.js')module.exports = {...jestBase,coverageThreshold: {global: {statements: 100,branches: 100,functions: 100,lines: 100}},browser: true,setupFiles: ['<rootDir>/../../tests-setup/polyfill.js','<rootDir>/../../tests-setup/setup.js']} Node.js / isomorphic packages and services A or package or service uses the following within its root directory: iso-* svr-* jest.config.js const jestBase = require('../../jest.config.js')module.exports = {...jestBase,coverageThreshold: {global: {statements: 100,branches: 100,functions: 100,lines: 100}},testEnvironment: 'node'} Notice how we configure the on a per package / service level? This allows individual teams to set their own thresholds. Managing that per package / service is significantly simpler than at the monorepo root level. coverageThreshold Test Execution The root file contains a script. Each package and service has its own script that simply invokes Jest: . The pattern applies to as well. package.json "test": "lerna exec yarn test" test "test: jest" test:coverage We can now use Lerna’s to do all sorts of nice things: flags Run tests for all services: . yarn test --scope @my-company-services/* Run test coverage for all web packages: . yarn test:coverage --scope @my-company/web-* Run tests for the package and all packages and services that depend on it: . @my-company/iso-package yarn test --scope @my-company/iso-package --include-filtered-dependencies Why use **lerna exec** to execute a npm script when **lerna run** does exactly that? From what we encountered, swallows the output of the npm scripts. With the flag, we get the output but it’s neither formatted nor does it have coloured console output. lerna run --stream Conclusion While I could imagine Jest’s multi-project-runner to be more performant than our solution, we like Lerna’s powerful flags and decided to forgo Jest’s approach. This may very well change as more and more tests get added to the monorepo. (Happy to chat about that if anyone has some thoughts) ESLint & Prettier No special consideration was necessary. Simply add your config files to the repository root and it works as expected. Pull requests, templates & labels The pull request template is configured once in the file. It applies across all packages and services. .github/PULL_REQUEST_TEMPLATE.md Compared to multiple repositories, managing pull requests in a monorepo requires a bit more thinking. At the time of this writing, we have not yet decided how we will deal with that. A few notes from initial discussions include: Use a labels with unique colours per package / service (we’ll run out of distinguishable colours quickly though) Use green labels for new features, red for bugs. Create green and red labels per package / service and add the package / service name as the label’s name. Conclusion The benefits of a monorepo immediately were apparent to the team. Prior to that, we used to deal with a small SDK we use to integrate with the backend API. It works, if you’re careful and don’t deal with Docker as we do for our local development. Regardless though, it is still a mental burden on each individual developer who works on the SDK. yarn link Getting everything configured took time, I am not going to sugarcoat that. Thanks to an amazing and curious team who showed patience throughout that transition period, we’re now in a place to spend more time building software rather than maintaining source repositories. Thank you!
