Applying and scaling DevOps in Enterprise Organizations ------------------------------------------------------- ### Motivation As [technology](https://hackernoon.com/tagged/technology) continues to grow in importance in our daily life, Enterprises of all types see themselves forced to become their own “**digital** versions”. Either because of pressure of their super connected consumers or pushed by new competitors, they soon realize that their current processes are no longer able to keep up the pace of business **innovation** and faster **feedback** cycles. While most of these organizations have already shifted their development methodologies towards Agile, this agility is usually confined into the development teams silos and have little, if any, impact on the overall organization **time to market**. [**DevOps**](https://hackernoon.com/tagged/devops) is definitely one of the key practices that are being used to **accelerate** end to end delivery and innovation, but **large** **companies** usually face many **challenges** to **scale** this culture, compared to startup or pure [technology](https://hackernoon.com/tagged/technology) companies. Below, I go through a few ideas that can **help** you during this **journey**. ### 1\. Automation Squads Everybody knows how a typical Enterprise IT looks like: several **teams**, **vendors** spread in multiple locations, **communication** **challenges**, multiple **projects** and programs going on in **parallel**. How companies usually try to tame this _dragon_? The standard approach: _Standards_. Get people to create documents describing the **DOs** and **DONTs**, hand it over to the people involved, define manual verification and approval gates between each phase. Results: delivery **delays**, **dependency** **hells**, engineers spending too much time on paperwork, no optimization on the long run. A better ([DevOps](https://hackernoon.com/tagged/devops)) approach: (technical) **Contracts** and **Tools**. That is, _digitalize_ your processes, automate whatever is possible (Building, Testing, Delivery, Monitoring). Focus on agility on the whole delivery stream, put automated speed over (human and error prone) control. In a **perfect** **world** (you may wonder), engineering teams would be able to **identify**, **get buy-in** (from multiple areas and stakeholders…) and implement these tools themselves. In real **enterprise** scenario (with both internal and external players), you will probably need to put in place a focused team (an **Automation** **Squad**) with **focus** and **empowerment** to do so. This should not be a new “**Change** **Management** **team**”. This group must create **tools** that **enable** other **teams** to develop and deliver their software autonomously.  [_Jenkins_](https://jenkins.io/) _(an Open Source Automation Tool) can probably be a good base platform for this team and is probably already being used by many of your development teams anyway_ ### 2\. Application Analytics Not everyone understand that Agility is not only about “moving **fast**”, but also about “**changing** your **direction**” as you **need**. And this **need** usually comes in two flavors: **user feedback** and **technical reasons**. My experience is that the first type is usually already in place. There is already very strong business pressure around it, and it is something companies have being doing for many years already. Tools like **Google Analytics** give Product Owners and business stakeholder access to key insights on how the application is performing at the **user level** (page views, conversion rates, …).  _The good ol’ Google Analytics_ It does not mean that there is no room for improvements. There are many new **User Analytics** tools that can be of great value, specially for **Designers**. If you take a look on [HotJar](https://alternativeto.net/software/hotjar-insights/) and similar ones you will have a sense of what is possible with this newer bread of tools.  [_HotJar_](https://www.hotjar.com/) _Heatmap example_ Now, in order to support DevOps teams you need a tool with a **totally different perspective**, something that can provide things like: * Issue **debugging.** * **Performance** analysis. * **Security** and audit logs. And for that, you usually need access to all the generated **logs**, not only samples and summarized data (that is usually what you get from the previous category of tools). Off course there are many Enterprise grade tools with this capability, but if you want to start with the basics, the number one (hands down) option is [**ELK**](https://www.elastic.co/webinars/introduction-elk-stack) (ElasticSearch+LogStash+Kibana).  _Sample ELK Dashboard (analysis of HTTP/Apache logs)._ You can setup your own infrastructure, or you can search for SaaS options (there are several available). The important thing is, you need to have a tool that enables **productive** **communication** and **interaction** between the multiple roles in a DevOps organization. If any **incidents** you have on production takes **several days** (and dozens of e-mail threads) to be analyzed and solved, you are still running on the traditional “Dev vs Ops” model. ### 3\. A New Security Mindset Differently from the Automation topic, where I think a **centralized** approach can work best in enterprise organizations, I think the **opposite** must be applied for **security**. You can put in place **tools** to check system vulnerabilities and perform **penetration** **tests**, but people (all of them) must be really the center of your security strategy. **Why?** Most of the hard to catch (**functional)** bugs I have troubleshoot in the last years were caused by **accident**, and by that I mean: **_skilled_** **_people_** _doing something (exceptionally)_ **_wrong_**. But when I think about the level of **security** being delivered in many projects, it is totally the opposite. The security **flaws** and **breaches** were created by people that did not have a clue of the **mistakes** they were making. The success would be the exceptional case.  Go ask your team a few questions: * Do they know what **OWASP** is all about? * Can they give you some examples of **XSS**? * Do they understand the different files are sit on their ~/.**ssh** folder? If so, are they still connecting on the servers using passwords? * Are their hard drives **encrypted**? What about the ones on the server they are working on? * What **password** **manager** are they using? * What type of SSL/TLS **certificates** they are deploying on their web applications? They will probably not be able to answer some of them. Security is unfortunately still not a very important topic on most of computing/development courses. You have to put in place **training** and **knowledge** **sharing** practices to foster this skill in your teams. The fastest (but not the best…) way to make companies invest on security training and tools is to get **hacked** — _sad but true_. The next topic can give you another, much better way, to bring this to everyone’s attention. ### 4\. War Games DevOps being defined as a **culture** of integration of **different** **teams**, a key thing to do is to put together some **team** **building** activities to make it happen for real. One way of doing that is performing some “**War Games**”, simulations and exercises involving different goals and people. Some ideas: * **Single** **line** change: test your whole development and deployment flow by changing a single line of code. * Internal **Hacking**/Penetration test: Try to hack your own system and use this to create security awareness (see topic above). * **Disaster** **Recovery**: Simulate different disaster (single server, single AZ, complete region). You can do that using a backup of your system. You do not need to be a Netflix company running [chaos monkey](https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey) for that. * **Peak** **demand**: What happens to your application/infrastructure during a severe demand peak. Will it be able to scale as designed?  Each of these exercises will put different skills to the test, and definitely will help your teams to handle real life situations with much more confidence. — _Let me know if you have any DevOps implementation experiences to share!_