Early in 2018 our engineering team embarked on a journey to revamp our infrastructure and tooling. In essence our developer experience was far from were we needed it to be.
Having a poor developer experience led to: engineers making workarounds to sync their code changes to EC2’s directly bypassing the pipelines. Unit tests barely used as the pipeline took a long time to give feedback. Debugging wasn’t easy. If a critical bug was found on production, good luck getting the fix out in anything under 1 hour on a good day.
We wanted to do better, our goal was to…
Move with speed and have pipelines deliver software in sub 5 minutes
…and that is what started our kubernetes journey.
TLDR: Want to save $1 million dollars AND have a happy development team? Provide the correct tooling.
So did we really save $1 million dollars?
Apparent click bait title aside, we can save that amount of money if we look at our delivery pipeline alone and forgetting our reduced infrastructure costs. We had all of our delivery pipelines running on Bamboo (an old Atlassian product). It looked something like this:
- Code push
- Manually go and hit Run Build in Bamboo 🐌
- Bamboo built an EC2 AMI 🐌
- Spun up a load balancer, EC2, RDS database, Memcache, etc ($ and 🐌)
The process took around 45 minutes on average (if the developer was lucky not to be queued up behind others builds).
We moved to kubernetes/circleci and it looks like this:
- Code push
- Automatic build triggered
- Unit tests run (1 minute) ⚡
- Build a docker image and push (2 minutes) ⚡
- Deploy to kubernetes cluster (1 minute) ⚡
The difference, no extra EC2’s, no extra load balancers and a 4 minute timeline.
What about that $1 million dollars
Our team is approximately 28 engineers, lets say 25 of them do 5 builds per day, 265 days a year. Our builds are running 40 minutes faster. Quick calculation:
5 builds x 40 minutes saving = 200 minutes x 25 engineers = 5,000 minutes x 265 days a year = 1,325,000 minutes / 60 = 22,083 hours x avg $50 per hour staff cost = $1.1 million dollars
I hear you say, “but engineers are not sitting waiting on builds to happen, they multi task”.
They do but looking back; It took a full 45 minutes to find out if a build worked when running on AWS, we didn’t have docker back then. Now engineers can be confident their tests pass, code works on a standardised pipeline and share development URLs with other teams within 4 minutes of pushing code.
Migrating to Circle CI/Kubernetes has given us a modern pipeline that the team can be proud off. The team can fix critical production issues in minutes, not hours.