Hype Driven Development: A Case For Punishing This Abomination

Disclaimer: long read. About the painful stuff, what I want to share for more than a month. Also does not claim to be complete, it is more about how I try to deal with it.

Here, I want to share a few thoughts on how to convince the team/manager/product owner to not use the "hype" technology or make proof that this technology fits good in your project.

It is well known that representatives of different professions often invent ways to solve a particular problem. For instance, which approach and constituent materials to use and positions, the choice of a specialist often determines the success of achieving the goals.

For example, would the cafe be profitable, and would the bridge withstand the load successfully, would the operated heartbeat for many years. And, unfortunately, one wrong decision can lead to fatal consequences and suffer great losses, and even worse – harm the health and life of people.

Engineers, software developers, web developers, and other technical guys are no exception. Moreover, the final software products are extremely dependent on real business, with real people and finances. And the harm that can be caused by badly designed and implemented software can potentially be huge and irreparable damage to the entire business.

Let's look at one of the most common crimes committed by various teams that somehow participate in the design and development of various types of IT projects, named Hype Driven Development (hereinafter referred to as HDD).

This concept is not new, but it does not go back to the past century, and means a decision about a technological stack or software architecture that is taken without technical justification, research, based only on loud and popular words from the Internet or other sources, "buzzwords ", unverified rumors.

It's a development based on fashion and trends, without explanation of your choice from a technical point of view and not confirming by checking or evaluating the proposed solution.

What I have most often seen: blockchain, data science, machine / deep / any other learning, MongoDB, GraphQL, Serverless, Clouds, Kubernetes, and more.

These "hype" technologies and much more I've seen in different proposals of various technical requirements, in RFPs (Request for proposal), spreadsheets with estimates, and so on.

Don't get me wrong, I have a deep respect for the above-listed technologies, and, as an example, I often use MongoDB and Serverless. The problem is that these technologies were proposed to be used and added in the post-production phase, and their connection with business logic seemed as far-fetched as possible.

HDD definition as a crime

Why I started with such a long prelude at the beginning - you can imagine when a builder chooses from what material to build a skyscraper "because it is so fashionable".

Or maybe a doctor who prescribes pills according to the principle "well, those people take similar pills abroad, and this may help you too." Personally, I have not, I have never seen this, I wish the same to you. 

For me, such an approach and attitude to work looks like a crime. Therefore, I propose to consider HDD a crime, at least in the article.

To begin with, I think it's worth remembering what constituent elements of a crime are. Without the presence of corpus delicti, an act cannot be considered criminal.

It's a quite extensive and complex definition, but let's summarize and simplify it a little, coz after all, this is not a jurisprudence article, especially since the laws of countries and states are different enough. We are interested in the following components of constituent elements of a crime:

the object - is the part to which the damage was caused;

the objective side- is the external manifestations of the crime, action or inaction, the consequences of the crime, as well as the connection between them or, for example, the time, method, means of committing;

subject - the one who committed the crime;

the subjective side - is a person's attitude to a crime that he has committed. Was it intent (direct or indirect) or it was negligence (frivolity or negligence), there is a motive and what is the purpose of the crime.

Let's start with the subject. In our case, this is one of the members of the development team in the broad sense of the word: engineers, managers, product owners. That is all those people who in one way or another influence the development process.

But they can be divided into two categories. Some of them work on business logic, tasks, and functional requirements - usually the manager, business analyst, and product owner.

On the other hand, we have technical specialists - test engineers, developers, architects, team leads. Of course, they can work at the junctions of the technical and non-technical parts of the project, but now it is important that both the former and the latter can offer, perhaps even insist on using the hype technology.

And here we smoothly move on to the subjective side of the HDD crime - did anyone want to harm when choosing an HDD technology? I think nobody would choose technology or design architecture to the detriment of their project or product. Then why?

Based on personal experience, I can say, that typically, customers (meaning people or companies which order development of product, not product users ), product owners choose one of these technologies to attract the attention of investors and potential users as sources of funds and income.

The reason is more than clear and transparent - often investors are not technical specialists, so, many of them are being led by beautiful marketing slogans, that promise many profitable things for the product. But at the same time, they usually don't understand the real essence of things. As a result, we get a technical solution based on marketing. There are also cases when a customer already comes with technical requirements for a future product.

When asked "why does he want such a stack of technologies?", we usually get one of the options: "one of my projects uses such a stack and everything is fine", "my partner's project uses such a stack - and he is happy with everything".

People who say this usually do not think about the fact that if functional and non-functional requirements are different, then the technologies used do not have to be the same.

Sometimes it happens that someone from the technical part of the team initiates the use of "hype" technology. Usually, this is less experienced teammates (juniors), which still cannot always assess the risks and make appropriate time estimations.

For them, it all comes down to the fact that they want to try a new technology, which is given in a beautiful wrapper. And also usually they do not look at the license of the solution (let's say it is a library or a framework), or at its maturity.

Objective side and consequences of the HDD crime

The objects of the HDD crime can also be different. It depends on what side of the project we are considering - technical or business. If we are talking about the technical side, there are codebases in general, its architecture, the infrastructure of deployments.

On the other side, the business operated by other concepts, like include cash flows, investments, profits, and other business processes.

So, let's say a "hype" technology penetrates the project, seeps into the code base, a lot of logic, additional code and configurations begin to rise around the technology. In this case, what and who will suffer from the introduction of "hype" technology, and how (we are talking about the objective side) will the HDD affect the project and the business as a whole (the object of the crime)?

There are a lot of things, consider that you are playing dice - it all depends on how hype and not properly researched technology is deeply integrated into the business logic of your application, code and development processes in general. The list below presents the most obvious consequences.

🚫 Building a business model around technology, not adopting technology to business. This happens relatively rarely, but extremely unfortunate. It happens when a "hype" technology is at the very core of the business logic and, accordingly, the code base, when the very idea of ​​the application is to build something over the technology.

Accordingly, if the technology breaks down in one way or another (for example, an unobvious bug pops up) - with a high probability, the product that is based on it breaks too.

🚫 Low conversion from "hype" technologies - that is, the discrepancy between the expected profit from the technology and the real one, as they often do not take into account that needs to pay for additional infrastructure along with the support of such a solution, as well as the work of specialists who will configure this solution.

🚫 The black box problem. Usually, the external dependency is perceived as something atomic, as an integral component within our system, with its public API (interfaces, methods, endpoints).

But in fact, this is the same program code as the one in which we integrated this component, with its dependencies and bugs, performance, and security problems. You probably say: "But often the code of such components is open source!". Now think, how often do you have enough time, desire, and immediate need to look into the program code of one of your libraries or database (which is also a software product with its source code)? I think not often (but usually it tends to never).

For us, a bug (this is how it can harm your project) in any external dependency is a kind of analog of "Schrodinger's cat in a box", which is both there and not. We often find out about the presence of bugs only when this external bug affects the state of our system.

🚫 Lack of professionalism and experience in the introduced technology. It's simple - the devil is in the details. Even if you have over-qualified specialists in your team who love work and are ready to delve into everything new on their way, hardly anyone can comprehend the depth of this or that technology in a short time.

🚫 Increasing the entropy of the program code and the system as a whole. Hmm, okay, but what is entropy? It is a very broad concept, rooted in natural sciences, thermodynamics, and later mechanics and software. The initial idea is that in an isolated system the degree of chaos and uncertainty will not decrease over time but will remain the same or increase. On the other hand, entropy is a measure of the unknown about the system as a whole.

The concept first appeared in the context of software in the book Object-Oriented Software Engineering (by Ivar Jacobson, Patrik Jonsson, ACM Press Staff, Magnus Christerson, Gunnar Overgaard). In general, think of entropy as a measure of the dishonesty of the system. The more disorder, uncertainty, the higher the probability of breakage.

Compare a hammer and a car, which is a hundred times more complex in its structure - which one is broken down more often and faster? A car has many more constituent components with its level of entropy, and therefore the total entropy is higher.

Returning to HDD, the more complex and more unknown technology we involve in the process, the higher the likelihood that the project entropy will increase, and hence the number of bugs and Mean time between failures (MTBF).

🚫 Life cycle and aging of software. Are you sure that the hype framework, the server, will exist, developed, and be supported for a long time, at least as long as the life of your project/product? And that in the next version the public API will not change completely - you need to either translate the API to a new one or live with the deprecated API. Have you assessed these risks correctly? Let's discuss this further.

🚫 Increased risks. New technologies usually mean new risks:

1. Going out of bounds by time estimation.
2. Risk of unsuccessful completion of the project.
3. The additional overhead for infrastructure, development, and support.
4. Additional vulnerabilities of the system crash.
5. Nobody guarantees that critical security updates will be issued and fixed on time.

How to prevent HDD in project

Ok, it seems that we have considered all the components of this crime. But how to prevent and avoid the use of HDD on a project without realizing full awareness of what technology the team will live with on the project? Or you can prove the validity of the implementation and evaluate all the pros and cons.

Once again, just in case, if the technology is hype, this does not mean at all that it is full of bugs, has not matured, or does not apply to your project. It's just that there should be common sense in everything, namely in the analysis of technology from a technical and business point of view, a sound assessment of risks and additional costs.

Here are some ideas and approaches, the use of which can influence the views and awareness of the team regarding the choice of hype technology.

💡 The opinion of experts, both in the technical part and in the subject area of business - "do we need it?" Before getting into Data Science, Serverless, and the like, consult a specialist with experience with this technology. Probably, you will not give up the technology, but completely rethink its use in the project. Moreover, it has a positive effect on grades.

💡 Proof of Concept creation is as old as the world - before doing a full-scale implementation and implementation, you should make the simplest demo version tied to any technology. Most likely, you will not notice any deep and hidden problems, but you can throw away the technology at the start if you see that it does not suit you in any way.

💡 A collective brainstorm "for" and "against" - decisions should never be made alone or in a narrow circle of employees. It is worth discussing and listening to all opinions, perhaps a sensible thought will come from an unexpected side. Also, interesting is the Three amigos approach, which is rooted in BDD and Agile, and means that there are three parties involved in the process: Business, Development, Testing. Thus, you can try to identify the largest number of problems and corner cases associated with the introduced technology. More details here.

💡 "5 Whys" - and again about how not to drag unnecessary things to your project. Try to apply this technique to the next technology adoption issue. Ask yourself and your colleagues - "why are we doing this," and so on, 5 times according to the list (see the article). It may well be that simpler steps are sufficient to solve the original problem.

Also, consider whether you can achieve the desired results without using this technology. How much more difficult and costly it will be.

💡 KISS (keep it simple stupid) And YAGNI (you ain't gonna need it). A huge number of articles have been written about these principles, and I see no need to retell them. Just make sure you don't break them when you introduce the next technology to your project.

Conclusion

We're finally here, and I'll try to make the ending shorter! There is a huge difference between "adding and usage of new technology" and "deliberately adding and usage of new technology". It would seem that the phrases differ by one word, but there is a huge gap between them. Hype technologies are often interesting, progressive, the main thing is to choose and apply them deliberately, without risking the project and business, not forgetting about your technical debt.