How I became a better developer by being an Operations team member.
This is a brief talk I gave to the interns who joined us recently. If you are in the know, this might sound like an over simplification. It is intended to be so.
Software
For the purpose of this write up, let’s assume that any tool that addresses a particular need by executing a set of instructions(code) in a computer is considered as Software.
How does it come into existence?
Jane finds it repetitive to compute her daily expenses over and over again, John finds himself wondering if only there was an easier way to keep track of all the bills that he needs to pay — they are intuitively thinking of either a machine or piece of software to do the mundane, repetitive stuff and make their life easier.
It’s hard to build a hardware machine (screws, nuts and bolts) to keep track of all the things you need to do on a daily basis. But hey, we have a dumb machine that can execute instructions we give it to them. Can it execute any instruction? Bill Bryson said:
For a long time it puzzled me how something so expensive, so leading edge, could be so useless. And then it occurred to me that a computer is a stupid machine with the ability to do incredibly smart things, while computer programmers are smart people with the ability to do incredibly stupid things. They are, in short, a perfect match.
It shouldn’t be that hard to write a set of instructions to a computer to do something for you. Or so someone thinks and writes up a bunch of instructions(code) for the computer and thus a _soft_ware is born. I say shouldn’t be because if it indeed is hard, why would there be 100s of ToDo apps for Android alone? Ever wondered even though there are 100s of apps, how it’s hard to find something that meets your needs? Coding isn’t hard but developing software is.
Types of application software
There are several types of software built by different people and organisations and are sold, subscribed to or given for free. Broadly speaking, there are 4 types (there are multiple ways to classify, but this is just a simplification of application software to an uninitiated user):
- COTS — commercial off the shelf software. These are applications like Microsoft Office, iMovie editor for Mac, Adobe PhotoShop, Oracle database, etc. You get a pre-built application that can use however way you please.
- SaaS — software as a service. This is when you don’t own the software but you pay for your usage. For example, Gmail for business is an application that you use via a browser but don’t install it using a CD (an installable) like MS Office. You pay for the usage, and can customise it to your business but you don’t own the piece of Gmail software, nor do you maintain it (bring the application up if it’s not running or fix issues or buy the servers yourself).
- Web applications — these are software built by companies that you use to access the company’s services. A bank website that helps you make online transactions is one example. An e-commerce site like Amazon or eBay is another. These are maintained by the companies themselves, and you as a user don’t pay for using the application itself.
- Mobile apps — the lifeline for most of us after smartphones became popular. Tiny bits of software that lives on your phone that either stand alone on the phone like the calculator or talk to a Web application to store and retrieve information like Uber.
- Lastly we have tools, utilities and modular pieces (called libraries) of software that helps us run these other software listed above.
How are they built?
To build a decent software(or an application, we will use them interchangeably), a lot of things has to happen.
- First, the idea for the application (mostly driven by need, sometimes by greed and sometimes by competition), for e.g — i would like an application that helps me manage my ToDo list.
- Second, what things should our application do. In technical parlance, this is called the backlog of features which is managed by a Product Owner — for e.g., I would like my ToDo application to remind me at a set date and time about a ToDo item and so on.
- Thirdly, how should the application look like — this used to be categorised as UI design but it is more broader and covers UX, user experience, managed by a Designer — for e.g., my ToDo application should have a prominent green + image which, when I touch it, should open up a form where I can add a new ToDo item.
- And lastly, the actual code that does all the features in our backlog — things like the application should store my to do list somewhere so that I can look at it later or I should be able to mark an item as done and it should be shown with a green tick mark next to it, etc.
Have you built an airplane?
Building a software application is very similar to building an airplane. A bunch of engineers sit, design and build the engine, wings, seats, cockpit, toilet, all the way down to the last screw and nut that goes in an airplane. Either a company like Airbus builds the A380 and sells it to an airline like Emirates (COTS) or you can rent an airplane for business or pleasure (SaaS) or you fly an A380 operated by someone and pay for getting from A to B (web/mobile applications).
There are decades and decades of work and literature on how to write better code (or build airplanes). This post is not about that. Let’s fast forward a few months to the point where we have the airplane is built and is ready to take its virgin flight. In other words, the code is written and our ToDo list application is ready for use. I’m going to continue the airplane example but you can draw the analogy to our ToDo list application.
Okay, we built the airplane. But who flies it and keeps it flying?
After this point, some or all of the following things happen:
- There are problems in what we built— the seat belt of one seat is faulty— a.k.a a bug.
- User wanted the airplane to have more things like ability to order food from their seats by touching a button — a.k.a enhancements.
- Other concerns with the airplane— it takes too much fuel to fly, it can’t go fast enough, it needs frequent check ups to see if anything looks like it’s going to break — a.k.a performance issues, downtimes, etc.
In most companies, the above things are left to be handled dedicated teams called the Operations/Run/Maintenance* teams(pilots, stewardesses and ground crew in our A380). Their job is to make sure the flight is up in the air, fly smoothly, none of the users face any discomfort, tighten a few screws here, and in some serious cases, land the airplane safely even the landing gear fails — software parlance: maintain uptime, provide minor bug fixes and enhancements.
At times, Ops teams is not equipped to handle big issues like engine failures (a wrong formula or logic in code), in which case they call the people who built the engine to take a look (the developers who wrote the code). By the way, these developers, once their first [release](https://en.wikipedia.org/wiki/Software_release_life_cycle) is done, either:
- Enhance the A380 with feedback from users — this happens in SaaS, COTS, mobile apps and a lot of web apps — and release the next
[version](https://en.wikipedia.org/wiki/Software_versioning), the A380.1.
OR
- As is usually the case with large corporations (a.k.a enterprises), the budget is allotted for the first version. Next year, some other team gets their budget and these developers are probably moved to other teams. The A380 they built (probably with bugs) flies till the time the issues/enhancement needs became so numerous that it’s no longer economical to keep it. So they end up buying the next awesome version A381 (ironically, built by the same team that had the buggy A380 and much in the same manner).
This is what happens to your code (A380)
Your code gets patched, updated, abused and pressure tested all the time after it [goes live](https://en.wikipedia.org/wiki/Software_release_life_cycle#General_availability_.28GA.29) by inexperienced people, experienced people under pressure, people who are just coding while they wait to get into grad school. In fact, any decent application spends more time in maintenance phase than in development phase.
[Even if the application is in high demand that users are constantly requesting new features, the number of things that an application can do (and do well) are rather limited. At some point, we will either run out of features to build or the money to build them or the technology would be outdated that we have to rebuild the application again. While this might not apply for technology companies/giants like Google/Facebook/Twitter/Amazon, it usually happens to Mom & Pop store websites, Enterprises web applications, etc.]
People, who are unaware of what you were thinking when you wrote the code, will modify it in ways that would make any developer cower. [I’m not demeaning the work of maintenance teams but they just have to patch it up and keep the application up under tighter SLAs which means compromises have to be made and more often at the cost of code quality]. Commercial software is usually built under so much constraints, people start expecting it to have bugs. Which means, at any given time and altitude, our airplane has problems waiting to happen.
But why should you, as a developer, care?
Every time, you forget the common sense aspect of code, you make the life of a user and an ops/support person miserable. When you are on weekend support and a Platinum customer with an 1-hour SLA calls you at 3 AM on a Sunday, you better pick up. And if the logs and code doesn’t help me to figure out what’s going on, may God help you — I’m going to curse you, your mentors and teachers, parents, grandparents and will pray for your children.
Early on in my career, I wrote some bad code. I never bothered to think what happens after I am done. Until, after 2 years, on a Friday evening when I was about to board a train, I got a call from our support team (I have put my name as author name in the code, best practice you see) asking me to help with an issue because they couldn’t see any logs and the code is 100+ lines of nested loops and if-else conditions. I looked at the code and couldn’t figure out what I was trying to do in the code. My own code. I managed to f**k myself in the future. Then I realised how many people’s Friday evenings were screwed up because of my code all this while. I took the job of an Ops person for a year to understand how developers look like through that lens. We ain’t pretty looking.
What then, as developers, are we to do?
Follow best practices and apply our common sense. Think about the passenger and the crew when building the airplane and all that goes in it.
Optimize for good User Experience — I’d rather be comfortable than anything else(other than safety).Things like cramped leg room because we wanted to have more seats stuffed in, we can have only one door next to toilet because that’s how we have always built planes or because we built the seats first and now they can’t be moved are not adding up to good flying experience.
Modular code — If we, as developers, don’t build airplanes well enough, a crew member might have to spend 30 minutes to figure out where exactly both ends of the seat belt goes. Because there could be plenty of other issues, 30 minutes of an Ops team’s time is too valuable. And you don’t want to remove the seat to fix the tray table or the toilet door.
Logs — Issues are unavoidable, or so they say. So when an issue occurs, Give enough information so that it becomes easy for the flight crew or the passenger to figure out where the beeeeep sound is coming from. Instead of saying something went wrong, seat belt on 27c is not workingwould be much more helpful.
_Tested code —_This is like a checklist. When the pilot fixes the oil leak which was the reason for the beeeeep, it would make his life easy if he has a checklist to make sure what a properly sealed oil compartment looks like.
Readable code — Make sure the thing we built is easy to understand and change for others and under tight deadlines.
Design Principles — Follow conventions, standards and principles in the area you are working on. Don’t have different types of seat belts for Boeing and an AirBus A380 just because we should think differently. I expect the Logout button to log me out and not open up a form for feedback. Build things that do one thing well. Your airplane doesn’t need a rocking option in the seat, though it would be cool.
There are plenty of other best practices which are common no matter what programming language or tech stack you choose. Follow them. I don’t care if you write the most intelligent piece of code and do a billion operations in under x lines. If I can’t fix the plane while I’m flying it, the code is a liability no matter how beautiful.
But we got DevOps, the silver bullet. All our problems will be vanquished!
I can claim No, you didn’t get DevOps right! and be right roughly 95% of the time. While most consultants and vendors tell you DevOps is going to fix your problems, remember there is no silver bullet. If you
- rotate your developers across budgets (or products, whichever available) or
- you have a team of people who specialise in one thing (Linux experts vs Angular experts vs Java/C# experts) and are siloed, or
- you have people whose job is making sure processes are followed to the last letter without questions, reduce risks at any cost and charting graphs about developer productivity and optimise processes for a fixed story point burn down every sprint or
- a million other things that run counter to DevOps principles (if you Google DevOps principles, people have 4 or 3 or 5 principles. Seems like we can’t come to an agreement on what basic principles we agree on).
chances are your problems are much worse that can be solved by implementing DevOps.
TLDR;
Your code spends most of its time in maintenance than under development. When you code, be empathetic of the other person(users, testers, ops, support, fellow developers, etc). The other person could be you. No, DevOps is not the silver bullet — either you are not ready, or you don’t need it yet.
Notes:
*Operations team is a blanket statement I used for all including monitoring, infrastructure, small bug fixes, emergency response teams, etc. Technically, there are huge chasms of difference when you get specific, but this article is an oversimplification.