Don’t Push to Production on Friday. Yet.

This has become a well-known practice or even a meme. When someone pushes code to production on Friday, there is a significant outage of all services, customers are angry, ratings are going down, and revenue is dipping as fast as one can imagine. Eventually, everyone will be brought back to their laptops instead of spending time with family and friends over the weekend. But how did we end up in this situation?

If you look at this from a holistic approach, that makes sense. If a team publishes their changes to production, therefore exposing them to the customers and potentially causing a high load, many things can go wrong. All the bugs that haven’t been identified during the testing phase could come out, and if no one is around - no one will be able to fix or revert changes that caused an outage.

It is now a rule in any organization that implements Agile at least somehow. This means that releases should be minor, frequent, automated, allow backup, and even be exposed to a small number of customers at first. But why are we still so scared to release it to production? We prefer code freezes, no-release Fridays, and other innovations many of us have seen in our workplaces.

While all the practices mentioned already, in theory, do allow a particular state of calmness in one's mind, there is always this feeling that something might go wrong and it will fail.

Of course, there is nothing wrong with introducing code freezes for Christmas before Friday. The stakes might be so high that even the most sophisticated guardrails do not guarantee safe space. Ordinary releases should be allowed at any point and eventually by anyone who has done their induction training.

Are We Agile Enough?

Most organizations are still developing in Waterfalls with Sprints - they deliver frequently, but in chunks big enough to be called Epic or even Initiative, with a couple million budgets and anything from one month to almost a year's worth of development. Yes, they call themselves agile. They have all the definitions and automation tests, but no one believes in it somewhere deep in the complex.

Most of the releases are still re-tested manually by developers, all the functionalities have to be approved by the risk department before release, and no one collects the metrics that can identify the significance of the released value.

It is hard to say, but in most cases, it is true. We are stuck in a constant circle of meetings where we understand what must be done. But if not for that one Agile coach or COO passionate about Agile, that is still waterfall with slightly changed titles and an iterative cadence.

Unless the shift of values is successful and the organization truly believes that what it implements and enforces has some meaning, it doesn’t matter. There might be some improvements and even significant ones, but the general course is the same: we imitate the standards they leave.

Quality Control Practices

Most of the organizations are still focused on pre-release testing. There are some tests. There is a manual testing process. Then there is testing on the almost-like-production environment, sometimes on-like-production environment, or even safeguarded from the customer’s real production environment.

However, it is still pre-release and is not treated as a must for most companies to implement the same testing after release. It will likely be manual or automated testing for critical functionality, and only to the extent that quality engineers from specific teams can do it in hours or days.

Ask any software developer on the interview - they will assure you they have 100% code coverage and are doing TDD. Ask any leader in the organization, and they will assure you that all the testing is automated and that only 10% of the functionality that is too costly to automate is still being tested manually. This might be true, but most importantly, it is not what we seek. Bugs will exist no matter how rigorous we are in testing. It is essential to identify them before customers start to be annoyed, and the money will be lost.

What Is the Release Process?

There might be situations when the irreversible changes happen. There is a migration from the vulnerable software, a change in the database scheme, and a migration to the new infrastructure. You can name it. In this case, there might be no way of adding additional safeguarding for your operations. However, 99% of the release organizations are different from this. They are minor tweaks in maybe not that critical features that are not used by 100% of the customers, and even if something goes wrong - that’s tolerable.

CI/CD has become almost synonymous with the release process. It is rare for organizations now to have a system admin who will go to the physical server, plug in a USB drive, and run a command on a server. Most likely, there is a pipeline in one of the known orchestration tools, and after the button is pressed, everything goes live.

All release processes have to be automated, with clear and straightforward options to identify problems and back out if the result provides more harm to the business than benefits. Of course, some decisions must be made, but one must have data to base them on to make those decisions.

When to Push to Production?

Some may argue that implementing all of the practices we have described is too expensive. And they are not wrong - it will take long hours, result in some failures, and potentially annoy customers.

However, when you get the right mindset, implement the proper quality controls, and build an error-prone pipeline, there will be nothing to stop you from pushing to production whenever it is right to do so - on Friday, during Christmas, Easter, Black Friday, and before the summer social party.