DevOps and Behind the Scenes

First of all, this is not an article for telling everyone what DevOps is. There are dozens of those out there. This, however, is the bitter truth of the IT business and what DevOps approach can provide to your business.

Background story

I have been traveling around the world for the last decade, mostly for business purposes. Most of these visits were for field engineering; solution deployment/design, project management or high-risk change management during nightly maintenance windows.

In most cases, a business traveler is on a tight schedule and runs from meeting to meeting, therefore does not have the time to discover the city itself. However, I find myself very lucky to have had such an experience. It is not only because I had a chance to meet with many people and got to taste nice food but also because I saw how a similar task is done differently in different regions/companies. Moreover seeing how a product is being used at a customer site was enlightening. This is because developers generally do not see how really the product is being deployed in the field where the customers’ expectations might be different than the original design purpose.

BEST EFFORT != BEST PRACTICE

Deploying a project might take a couple of months and requires a good plan. A Very high-level project plan can include such steps:

• Identify the technical requirements/use cases
• Check the feasibility of the use cases in a lab environment. Prepare the configuration.
• Meanwhile, check the release guide (if applicable) and see if there are any known bugs that might affect your setup.
• Prepare a high-level design document (HLD) before moving forward. This helps the team to be on the same page.
• Prepare a test plan (functionality and smoke testing).
• Execute the test plan in a lab environment and document the results.
• Plan a maintenance window.
• After getting approval for the maintenance window, prepare a MoP (Method of Procedure). The MoP should include every step/command that will be executed during the live deployment. It also includes a plan for a possible rollback.
• Monitor systems for a day or two.
• Prepare Low-Level Design document(LLD) which is suppose to include every small detail of the deployment. (specs, ports used by the services, diagrams, where to contact in case of an issue, used firmware, supported features, system management tool details, configuration details etc) This is to handover the project to the client so it can be maintained.

Now I would like to tell you how both customers and vendors are failing to follow up such a checklist, or even if they do, why things get complicated.

• Identify the technical requirements/use cases.

Things are generally over-promised. Presales sells a use case together with the relevant hardware with it and a field engineer is supposed to deploy it. Sometimes they say: “yes” for a non-existing feature and letting field engineer and the project manager find a workaround for it. OR asking Research & Development team to quickly develop that feature not to jeopardize the whole deal. This makes everything very complicated and existing priority list for the feature requests becomes completely invalid. Not to mention that it also demoralizes the team.

• Check the feasibility of the use cases in a lab environment. Prepare the configuration.

This is the one I love talking about the most; Best effort is not the best practice. The same use case can be deployed in many different versions. Especially if it requires some coding or complex configurations, the customer is depended on the field engineer’s skill-sets. Most of the vendors out there, unfortunately, does not have the best of class internal training materials for their engineers. Some of their field engineers pick up much faster or does the job in a more optimal way while others’ configuration may not be the best. Make no mistake, both can fulfill the goal however at the long term, the non-optimal one can cause some issues such as when scaling is on the table or additional hardware is being added. Things can get broken and generally they escalate pretty fast.

• Prepare a test plan (functionality and smoke testing).
• Execute the test plan in a lab environment and document the results.

Sometimes the timeline is too tight and the customer is under pressure because for example, high-level management is expecting a faster deployment, or maybe it is a request from the regulator and they need to implement the changes as soon as possible. Test phase might be skipped and generally, the end result is devastating enough to spend even more time to fix the configuration and go for the new maintenance windows. Meanwhile, damage is already done and it can behard to get your customer’s trust back.

• Prepare Low-Level Design document(LLD) which is suppose to include every small details for the project. This is to handover the project so your client can maintain it.

I think we engineers, in general, lack the skills to document what we do. We generally forget to prepare a low-level design document and hand it over to the customer. Documentation is the most boring part and people don’t bother to ask for it at that phase since the project is deployed and everyone is happy. A lack of documentation makes a possible issue very hard to troubleshoot, or even for vendor technical support team it is hard to understand what the root cause might be at a first glance.

Each of these failures can possibly delay the completion of a project for a couple of weeks or even months.

What is THE Root-Cause?

Many of those failures are because departments within a company are very isolated. It might sound like an easy job, however for different development teams together with Quality Assurance, Documentation and Sales departments, moving forward to a common goal is a real challenge. Nowadays enterprise products are the combination of multiple technologies and the components, and larger the company, the greater the lack of collaboration for these different groups. DevOps approach is a must for business owners and the development, operations, and quality assurance departments to deliver the software in a continuous manner that enables the businesses to adapt to the market requirements faster.

Adapting DevOps

Adapting DevOps is not putting the commonly used tools to the table such as Ansible, Terraform, Kubernetes, Puppet, Gerrit etc.

It starts with a real communication. Moving towards a common goal requires social engineering. How can a development team be the most productive? What code collaboration tool fulfills their expectation. How should a ticket for an issue or a feature request be created? What would be the workflows of these tickets and how does the issue tracker react when relevant code is pushed to master branch? Who evaluates the feature request? What are the existing in-house tools that can possibly be replaced with open source ones? How is a code snippet being tested and deployed on the lab environment? Is the test being initiated automatically? If yes, how it really works and does it put more burden to the development team to test their code or make their life easier which can result with a better product quality?

DevOps teams are here to provide you the ability to identify business needs and adopt the widely known DevOps practices such as;

• Continuous Integration,
• Continuous Testing
• Continuous Delivery
• Continuous Monitoring

CI,CD practices allows developers to work together even if there are a large number of cross-functional teams. Continuous Integration also requires Continuous Testing where each new feature or a bug fix can be tested by using test automation approach and revert back to theteam if need be, or integrate the ready-code with a specific branch and fire up another automated test to verify the build.

Continuous Monitoring, however, is my personal favorite, this approach can be used in test or and the production systems. Figuring out how a product behaves after being deployed in production environment is another challenge. My experience shows that even at the companies where an IT Operations team is allocated to monitor the systems don’t really know before an issue becomes a problem. When things are broken, everyone will be involved to solve this, and possibly a small blame game is around the corner. With Continuous Monitoring approach, you can check the metrics and provide real data to everyone including shareholders. Does your network pipes are being saturated? What is the utilization rate and do you need to invest for a possible expansion at the close future? How many times or how often a certain error occurs and can we visualize this? Can those metrics be used for different goals rather than troubleshooting? (marketing maybe?)

Data provided by monitoring solutions can help businesses to react in an agile manner and change their business plans if need be.

Bonus: Career Advice — How to Become the ONE?

It is all about connecting the dots. Unfortunately, there is no book out there that will make you a DevOps Engineer(if that title ever exists, that is another discussion though).

It is all about becoming curious and learning how to reason the behavior. I do like to talk to people a lot. This teaches me lots of things and at the end of the day it takes a real communication skill within a team to have one good product.

1- Learn thoroughly

I remember talking to one of my friend during an office lunchtime break. I asked Philip about his future plans. what does he want to do 2 years later? He told me that he wants to be a better programmer. What about 5 years later? become even a better one! This struck me suddenly, many people living at the tech-savvy countries do share a similar goal. Have a passion for something and learn it thoroughly, where on some other countries however, it is, learn it enough and jump to a higher position, eventually become a manager. When this example applies to big part of community, then you become a maintainer or user level developer rather than the technology founder. Sure some of us need to manage projects, teams etc, however this should not become a rat race but instead a passion. If someone is better at managing a product life-cycle rather than doing something else, surely his time shall be well invested in that direction. Titles or positions are not here to superior the others, therefore should not be the goal itself.

2- Be Curious

Often people are so into what they are doing and cannot find the opportunity to even become curious about anything else they see in their daily life.

I remember asking a candidate about the difference between HTTP and HTTPS on a job interview. This might sound like a cheesy question, but it is also something we see everyday at our URL bar when visiting websites, and it tells me a lot about his/her learning cravings. Many say that one is more secure than other, however, most do not have an idea what is more secure there yet alone knowing how it works on a high level.

Being curious and eager to learn such small details will impact your success huge. You are going to realize that they are all connected at the end.

3- Talk to others

Coffee breaks, kitchen halls, stairs, anywhere actually. You get to know your colleagues when chitchatting. How are you? What project are you working on? Where are you going for lunch? Such simple questions can open up way more interesting topics along the way and helps you to broaden your knowledge as well as social skillsets.

4- Don’t be afraid to tell what you think but also be open to the opposing ones

There will be many occasions where you will find yourself in a meeting room with many people and possibly in the middle of a “nonsense” discussion. You might have a better plan than others or maybe overall topic is not worth to spend so much time to discuss about. I realized that more I learn more I found out that how little I know. This always helps me to question myself more often before giving an advice, or perhaps what others are saying is not nonsense at all?? Nevertheless, always be gentle and graceful, being too pushy and giving advice without listening is not a real communication but more like a one way conversation.

5- Be humble

Do not be an a.hole. What you might think you know the best, might not be the case at all. There are silent but very knowledgeable people around there who listens everyone and filters the popping up ideas thoroughly. I am sorry to say that but you cannot be the best. IT is a very broad domain, even if you think you are the best at your own field, which is probably not true, not even for Linus Torvalds or Elon Musk. You might be super good on software development but lacking the presentation skills or vice versa. My point is, don’t be an arrogant person. There is always better than better, use this as a benefit to yourself and learn from others.

6- Never give up, this is a long distance running not a sprint one!

There are some jobs out there that you can stop thinking or be doing it after business hours. However, there are some others that are more like a lifestyle. A landscape photographer rarely says he cannot take a photo after 5 pm. IT is a lifestyle and the learning curve is for the lifetime. There will always be new tools, languages, frameworks to learn about, if you can make it your passion rather than your job, then you will enjoy more of what you are doing.

Now Connect the dots: a person who likes talking to others, graceful and gentle, open-minded and always learning new skill sets, solid at Computer Science fundamentals is my favorite candidate for the future DevOps Engineer.

one of my favorite from xkcd.com