How our production deployments went from 🙀 to a piece of 🍰
This blog post is part of a series where I share our migration from monolithical applications (each with their own source repository) deployed on AWS to a distributed services architecture (with all source code hosted in a monorepo) deployed on Google Cloud Platform.
- Part 1 (this post): “A monorepo, GitHub Flow and automation FTW”
- Part 2: “One vs. many — Why we moved from multiple git repos to a monorepo and how we set it up”
- Part 3: “A (mostly) automated release process”
- Part 4: “Our approach to software development consistency”
- Part 5 (coming soon): “Debug microservices locally”
I think the simplest way to reduce risk in a new project is to stick to what you know. Sometimes though, it’s better to step outside your comfort zone in order to make progress.
Background (a 3-tier architecture)
Earlier in 2017 when I joined a new startup to lead the engineering team, I stuck to what I knew. Quickly, it became apparent that a bit of discomfort and exploration is what we needed. First though, a note on what we do: We build chatbots for market researchers to engage with their communities. The high-level requirements were:
- an admin web interface to author, deploy and monitor chatbots
- a backend system to deal with the business logic, validations and to process messages sent to the chatbots
- a (relational) database to persist all data
This screamed “3-tier architecture” all over, so a 3-tier architecture we built.
How do you deploy all that code?
Answer: You stick to what you know. In my case, I previously used the Gitflow workflow.
Gitflow Workflow (Source: Atlassian)
Every time a feature gets merged into the develop branch, the develop branch gets deployed to a staging environment.Periodically (more on that in a moment), a release branch gets created off of the develop branch, reviewed and merged into master. As soon as that happens, master gets deployed to the production environment.
We had multiple git repositories (frontend, backend, jobs service for long-running tasks and a SDK to talk to the backend’s REST API). The Gitflow workflow applied to all repositories.
So, what does “periodically” mean?
This is where the Gitflow workflow becomes problematic. At what point in time do you cut a release branch? Every Monday morning at 9am? What if someone merges a feature into develop at 8:30am? Do you create a release branch on Monday at 9am, test the release branch thoroughly and merge the release branch into master on Wednesday / Thursday / ???. Who is responsible / accountable for the production deployment?
In the worst case, creating the release branch, testing it and merging to master is a tedious process. Before you know it, it’s been a month or more since you last released to production. 🙀
We tried all sorts of approaches, experimented with more / less automation, etc.
More modularity, all code in a monorepo
As our struggle with the above approach increased, we gathered in front of a whiteboard and redefined how we want to deploy to production.
The main goal was to deploy to production much more frequently. In a best case scenario, we release each pull request to production as soon as it is tested in a staging environment.
Step One: Bring all code into a monorepo to better deal with dependencies. (This is worth its own blog post.)
Step Two: Get rid of the develop branch and only have a master and feature branches. Wait a minute… That sounds familiar! Of course, it’s the GitHub Flow 💡.
Step Three: Automate, automate, automate! (Worth another blog post…)
Lots of whiteboarding, experimenting, and many failed deployments later, we now have a monorepo, follow the GitHub workflow and deploy to production multiple times per day.
Release Process with a monorepo and the GitHub workflow
The green rectangles are fully automated, the blue rectangles require manual approval in order for the workflow to continue.
Benefits
With the above approach, we have a number of benefits:
- Deployments to production happen up to a few times per day. Each release is a fraction in size of what releases previously used to be.
- The responsibility to deploy code to production is shared among all team members. In our case, the reviewer of a pull request deploys to staging and production.
- With the GitHub flow approach, we have less branches to deal with and overall less process to get code released.
In a follow-up post, I will share the CircleCI configuration and a few other tools that help us automate most of the release process.
