Applying and scaling DevOps in Enterprise Organizations
Motivation
As technology continues to grow in importance in our daily life, Enterprises of all types see themselves forced to become their own “digital versions”.
Either because of pressure of their super connected consumers or pushed by new competitors, they soon realize that their current processes are no longer able to keep up the pace of business innovation and faster feedback cycles.
While most of these organizations have already shifted their development methodologies towards Agile, this agility is usually confined into the development teams silos and have little, if any, impact on the overall organization time to market.
DevOps is definitely one of the key practices that are being used to accelerate end to end delivery and innovation, but large companies usually face many challenges to scale this culture, compared to startup or pure technology companies.
Below, I go through a few ideas that can help you during this journey.
1. Automation Squads
Everybody knows how a typical Enterprise IT looks like: several teams, vendors spread in multiple locations, communication challenges, multiple projects and programs going on in parallel.
How companies usually try to tame this dragon? The standard approach: Standards.
Get people to create documents describing the DOs and DONTs, hand it over to the people involved, define manual verification and approval gates between each phase.
Results: delivery delays, dependency hells, engineers spending too much time on paperwork, no optimization on the long run.
A better (DevOps) approach: (technical) Contracts and Tools.
That is, digitalize your processes, automate whatever is possible (Building, Testing, Delivery, Monitoring). Focus on agility on the whole delivery stream, put automated speed over (human and error prone) control.
In a perfect world (you may wonder), engineering teams would be able to identify, get buy-in (from multiple areas and stakeholders…) and implement these tools themselves.
In real enterprise scenario (with both internal and external players), you will probably need to put in place a focused team (an Automation Squad) with focus and empowerment to do so.
This should not be a new “Change Management team”. This group must create tools that enable other teams to develop and deliver their software autonomously.
Jenkins (an Open Source Automation Tool) can probably be a good base platform for this team and is probably already being used by many of your development teams anyway
2. Application Analytics
Not everyone understand that Agility is not only about “moving fast”, but also about “changing your direction” as you need.
And this need usually comes in two flavors: user feedback and technical reasons.
My experience is that the first type is usually already in place. There is already very strong business pressure around it, and it is something companies have being doing for many years already. Tools like Google Analytics give Product Owners and business stakeholder access to key insights on how the application is performing at the user level (page views, conversion rates, …).
The good ol’ Google Analytics
It does not mean that there is no room for improvements. There are many new User Analytics tools that can be of great value, specially for Designers. If you take a look on HotJar and similar ones you will have a sense of what is possible with this newer bread of tools.
HotJar Heatmap example
Now, in order to support DevOps teams you need a tool with a totally different perspective, something that can provide things like:
- Issue debugging.
- Performance analysis.
- Security and audit logs.
And for that, you usually need access to all the generated logs, not only samples and summarized data (that is usually what you get from the previous category of tools).
Off course there are many Enterprise grade tools with this capability, but if you want to start with the basics, the number one (hands down) option is ELK (ElasticSearch+LogStash+Kibana).
Sample ELK Dashboard (analysis of HTTP/Apache logs).
You can setup your own infrastructure, or you can search for SaaS options (there are several available).
The important thing is, you need to have a tool that enables productive communication and interaction between the multiple roles in a DevOps organization.
If any incidents you have on production takes several days (and dozens of e-mail threads) to be analyzed and solved, you are still running on the traditional “Dev vs Ops” model.
3. A New Security Mindset
Differently from the Automation topic, where I think a centralized approach can work best in enterprise organizations, I think the opposite must be applied for security.
You can put in place tools to check system vulnerabilities and perform penetration tests, but people (all of them) must be really the center of your security strategy.
Why?
Most of the hard to catch (functional) bugs I have troubleshoot in the last years were caused by accident, and by that I mean: skilled people doing something (exceptionally) wrong.
But when I think about the level of security being delivered in many projects, it is totally the opposite. The security flaws and breaches were created by people that did not have a clue of the mistakes they were making. The success would be the exceptional case.
Go ask your team a few questions:
- Do they know what OWASP is all about?
- Can they give you some examples of XSS?
- Do they understand the different files are sit on their ~/.ssh folder? If so, are they still connecting on the servers using passwords?
- Are their hard drives encrypted? What about the ones on the server they are working on?
- What password manager are they using?
- What type of SSL/TLS certificates they are deploying on their web applications?
They will probably not be able to answer some of them. Security is unfortunately still not a very important topic on most of computing/development courses. You have to put in place training and knowledge sharing practices to foster this skill in your teams.
The fastest (but not the best…) way to make companies invest on security training and tools is to get hacked — sad but true.
The next topic can give you another, much better way, to bring this to everyone’s attention.
4. War Games
DevOps being defined as a culture of integration of different teams, a key thing to do is to put together some team building activities to make it happen for real.
One way of doing that is performing some “War Games”, simulations and exercises involving different goals and people. Some ideas:
- Single line change: test your whole development and deployment flow by changing a single line of code.
- Internal Hacking/Penetration test: Try to hack your own system and use this to create security awareness (see topic above).
- Disaster Recovery: Simulate different disaster (single server, single AZ, complete region). You can do that using a backup of your system. You do not need to be a Netflix company running chaos monkey for that.
- Peak demand: What happens to your application/infrastructure during a severe demand peak. Will it be able to scale as designed?
Each of these exercises will put different skills to the test, and definitely will help your teams to handle real life situations with much more confidence.
—
Let me know if you have any DevOps implementation experiences to share!