Production is not sacred

by Mike Cook

By now, I have worked on a fairly significant number of projects, and, for the most part, the success of those projects has correlated pretty tightly with how confident I am in production deployments. The reason why that’s the case is probably a big enough topic for a separate blog post, but generally speaking I think some major barriers to confidently deploying code are:

1. The developer is new to the codebase
2. The change being deployed is too large, probably because…
3. The codebase is too complex
4. The deployment process is too complex

Or any combination of the above.

Obviously point 1 is unavoidable, but each of the other points are entirely avoidable, and points 3 and 4 have an all-too-real impact on how long it takes a new developer to start deploying confidently. By ruthlessly reducing the complexity of our systems and providing the simplest possible method of deployment, the barrier to having your changes hit production can be very low indeed (for example, at TravelNest, all of our core services can be independently deployed, taking ~3 minutes from merge to being live in production; we continually try to reduce this figure).

This is all well and good, but microservices and continuous integration/deployment aren’t new ideas. For me, the problem often extends further than the technical hindrances of releasing changes, to an ingrained cultural fear of making those changes to live systems. No doubt this is present due to years of collective exposure to projects that suffer from the issues mentioned above — I’m sure we’ve all had moments of trembly-fingered red-button pushing, followed by uncertain panic that gradually subsides but never really goes away.

Nobody wants to live in that world, and I feel that any team can start taking large steps forward in efficiency of delivery by adopting a production-first attitude to development. This can be done by sticking to a few basic principles:

• Code is written to run in production only
• Code can only be considered truly tested once it is in production
• Breaking production isn’t scary, because…
• Monitoring and alerting can tell us immediately if a change has introduced an issue, and…
• Changes can be reverted as quickly as they can be deployed

This is all much easier said than done, but I would urge anyone that isn’t in this position already, or at least close to it, to fight like hell for the time and resources to be able to achieve it. This isn’t just tech debt, or developer enablement — it’s the tools your business uses to stay alive.

I’d even argue that if you never break production, your team probably isn’t moving as quickly and confidently as it could be. So instead of reaching for the safety net of a staging environment or a drawn-out testing strategy, ask what is actively blocking you from releasing directly into production. If the answer is not obvious, then default to production first, because production isn’t scary or wild — by the numbers, it’s practically the only place our code will ever run anyway.

About the Author

Software engineer based in Edinburgh, UK. Trying to find out how to build things quickly and effectively.