The Evolution of DevOps: Lessons for Organizations of all Sizes

When I started practicing DevOps in 2015, the term DevOps had been around for several years already. However, there weren't any definitive answers on how to handle cloud infrastructure and release developers' code to production.

Many companies didn't realize they needed to think this way since they had their developers and operations handle the lifecycle of their applications.

The developers would build the code, QA would test it, and operations would deploy the code to production using checklists, scheduling hours of downtime windows, and running scripts or replacing the new code manually on the servers.

Over time, organizations realized the benefits of adopting methodologies to reduce the time it takes for new code to reach production safely.

How DevOps evolved

Fast track to 2022, We can identify a few major shifts in thought.

DevOps is not a role, but rather a mindset, culture, and a methodology

Here is a short explanation of how DevOps evolved, but in a funny way:

https://www.youtube.com/watch?v=kBV8gPVZNEE

Simply hiring a DevOps expert will increase the release time by a small percentage but won’t solve the main pain points. Once the best practices are implemented, the investment will bear fruit in release time, deployment standardization, security, and microservices management.

Many organizations found that separating the huge monolith into smaller microservices reduced the release time and increased fault tolerance.

There is no need to reinvent the wheel (most of the time)

The easiest way to solve something was by writing a script to handle some kind of a repeatable task. Over time the scripts would grow and often break which resulted in maintaining scripts that only a few knew and understood how they operate.

Now there are many open-source and commercial solutions for most repeatable tasks. Occasionally teams would write wrappers to compliment and ready-made solutions to suit the organization's needs.

Switching to a Microservices architecture

When having everything in one place inside the monolith, it's easy to handle the communication with each other and adjust the dependencies between them.

Although the Monolith has its advantages, many organizations found that separating the huge monolith into smaller microservices reduced the release time and increased fault tolerance. By doing that teams could ship faster, be more agile and iterate consistently.

Why Microservices introduce complexity

Things get complex when having separated microservices on different servers with different runtimes, scaling, and security concerns.

There are 10 main questions to answer

1. How to handle the different software runtimes
2. How to make communications easier between services on different servers
3. How to operate all of these servers
4. How to harden the security of the communications between and inside the services
5. How to handle compliances, secure authorization, and authentication to the cloud providers and the services
6. How to properly scale your servers, and services and optimize cloud costs as your traffic grows
7. How to simplify and shorten the release cycles
8. How to control the different phases of the release cycles, from build to test to release
9. How to visualize and monitor the microservices?
10. How to educate the teams and enable them with the proper guidelines and toolsets

Who is responsible for answering these questions?

It depends on the size of the company, but the general guideline is to have owners for each pillar. If there are many owners, they should coordinate on a regular cadence to accommodate the business goals. It’s easier when there is a budget for building separate teams with a clear goal, and harder for small companies as their main focus is to grow.

When the company understands that they need to start adopting the DevOps methodologies they either set to hire a DevOps Engineer or use an outsource company.

Large organizations

Large organizations that have already adopted the DevOps ways have different teams who can address those concerns:

• IT operations take responsibility for facilitating the different enablers for the R&D groups (i.e., codebase, build environments, artifact storage, secret management, etc.)
• The security team is in charge of setting security policies for the entire business logic. (i.e., internal network, codebase, cloud environments, etc.)
• Developer/SRE teams are handling the release flow by using the stack provided to them from the IT operations and following the guidelines set by the security teams.
• Software/Cloud architects work closely with the different teams and advise, design, and assist with the implementation of best practices.

Startups

In short, the CTO and the developers.

Most of the companies that are between the pre-seed and round A phase (or before product-market fit) don't take most, or all of the 10 questions asked into consideration.

The results of focusing only on building the application could be:

• Lack of building cloud infrastructure-as-code increases the creation time of cloud environments, while building manually increases the chances of human error
• There is little consideration for security on any level
• The release cycles are either manual or written in a way that cannot be scaled as the business grows. They are frequently failing and are taking a long time to complete which holds up the cycle deadlines
• When best practices are overlooked, and things are built in a hurry, the result is a costly refactoring

When the company understands that they need to start adopting the DevOps methodologies, they either set to hire a DevOps Engineer or use an outsource company to assist them.

Choosing either hire will solve the problem only momentarily

Outsource company

There are great companies that can take the entire refactoring project off of the company's hands from designing and planning with experienced cloud architects to professional DevOps practitioners who will build and sometimes even operate the entire cloud infrastructure, release flows, etc.

Most of the time, they will assist with answering the ten questions.

While this can suit many companies, the costs are rather high for a Startup to handle at an early stage, and there is a significant overhead with managing a 3rd party contractor.

DevOps Engineer

It's becoming harder and harder to hire an experienced DevOps Engineer, but even when found - the chance that they will solve all of the issues is not high since it's too much to handle for a single person who is pressured of making poor decisions.

As a result, the company can end up hiring additional Engineers and eventually someone to lead the entire culture.

Many Startup companies after the round A phase are looking for a "one-man show" to avoid the mistakes of a never-ending hiring spree.

Finding someone to instill the DevOps culture inside the company with the proper skills of a cloud architect, systems administrator, security, testing, and automation is near impossible to find. This could get costly with high demanding salaries and the time to find one.

Once found and held for dear life, they are tasked to fix and refactor all of the accumulated issues that were neglected or poorly implemented, and over time they churn as sometimes they are chasing their own tail in order to fix production and development environments.

They wish to be involved during the decision-making process and to build production-grade environments with best practices.

Invest the time to educate the development team on the proper microservices best practices on the cloud as they do with software development best practices.

What should Startups do instead?

In short, “strike while the iron is hot”. They need to address the issues even before they become issues while having the knowledge for setting up the proper foundations at every milestone.

Reduce or avoid major refactoring entirely.

1. The CTO or any equivalent has to find the time to address the 10 questions so a plan can be formulated
2. The company needs to invest the time to educate the development team on the proper microservices best practices on the cloud as they do with software development best practices
3. Hire someone to initiate the DevOps methodologies and the entire culture early on when there are only 1 or 2 services. This way foundations are built stronger and will last. The person doesn't have to be the "one-man show" since there isn't too much to handle early on and the CTO has already laid out a portion of the plan

Benefits of enabling the DevOps culture from inception

1. Cloud environments are built fast and more reliable with best practices using infrastructure-as-code so developers can benefit from testing their code with production like resources in minutes
2. The release cycles are blazingly fast and the development team can ship new code reliably to production several times per day
3. Security policies are introduced at an early stage which keeps user data safe and private
4. As the traffic increases, the proper scaling of the various services is done in a cost-effective way
5. All the service logs are aggregated and all the monitors are easily visualized to assist the development team to find and troubleshoot issues fast when they occur
6. The teams are collaborating more efficiently during the various stages of the cycle
7. It's easier to educate the company early on when no bad habits were formed yet

Conclusion

We saw that only focusing on the application development can result in delays in deadlines, and companies understand they need to invest in DevOps methodologies at a later stage when the cost of refactoring is very high, both in terms of actual money and time.

Having someone to lead the DevOps space early on and eventually becoming a high-performing team will increase the company’s time to market by 10x, if not more.

Also published here.