How To Accept Over-Engineering For What It Really Is

The devil is in the details (or in the lack of it)

The picture of many swiss knives stacked in a single crazy object

Discussions are a fundamental part of software development, they are a necessary friction to increase scalability. A few patterns can emerge from those discussions that allow us to spot problems early when those problems can have an impact on the quality of the outcome. One of them is Over-Engineering.

According to Wikipedia, Over-Engineering is:

… the designing of a product to be more robust or complicated than is necessary for its application, either to ensure sufficient factor of safety, sufficient functionality, or because of design errors.

Wikipedia also goes further to say that Over-Engineering can be desirable when:

… safety or performance on a particular criterion is critical, or when extremely broad functionality is required, but it is generally criticized from the point of view of value engineering as wasteful.

In the developer community, we can find additional thoughts. According to StackOverflow user Jeff Sternal, the common symptom of Over-Engineering is:

Code that solves problems you don’t have.

Another user, 0scar, provides a visual representation of what Over-Engineering looks like in the context of Software Development:

A graph with three columns where the first one contains stacked blocks named "User" at the top, "Design", "Engineering" and "Hardware" at the bottom with the title "With balanced engineers". The second one entitled "With self absorbed cave dwelling engineers" shows that the "Engineering" starts going off track removing the balance of "Design" and starting to invade their space as cancer, the "Engineering" block was renamed to "just bad engineering". In the last column, entitled "With narcissistic egocentric socio-pathic engineers", "Engineering" have taken "Design" out of the column, invaded its space completely and started to surround "User", now it's called "Over-engineering". Over-engineering always go up, the only thing that never changes is the "Hardware" block at the bottom.

The Second-System Effect states that "small, elegant, and successful systems tend to be plagued with feature creep due to inflated expectations". In software development, the Second-System Effect can also be observed when any project stakeholder (which also include engineers) decides to build more than necessary in hope that it will add intrinsic value to the project. The difference is that the expectations are inflated through speculation of the future instead of actual facts. The problem is maximized the farthest the speculation goes from user requirements to engineering and vice-versa.

A famous comic using the attempt of building a tree swing as the analogy of what is wrong in each part of the traditional development lifecycle

Over-Engineering is when someone decides to build more than what is really necessary based on speculation

Still, we need to understand what exactly is "really necessary based on speculation". It probably boils down to the same dilemma of the Proper Solution and the Legibility Argument, when the answer depends not on an objective criterion but on subjective arguments.

Let's take a small naive example for changing the class of an element on click_._ Here are the requirements:

Build 3 clickable images where clicking on each one of them will cause the one being clicked to become bigger double the original size and after 1 second to become smaller again to the original size.

The code for the operation on the clickable images

In the example above, the implementation doesn't use any big pattern or framework. When we click on the element, it becomes bigger and then after 1 second it goes back to the original size.

There is clearly some duplication there and we can create a single function to remove that duplication:

The code after the duplication is removed

Is this solution enough? Maybe.

If we take a step back we may notice that we have 3 common steps in the click lifecycle: the inclusion of the class, the 1-second delay and the removal of the class. If we need to add any operation on any of the steps we would have to alter that function instead of altering the attributes of each step. There's a hidden feature in jQuery that allows us to create a queue of operations with a delay. We can leverage that to separate the functionality in steps (I know, library internals do not exist and no, this feature is not internal to jQuery):

The code using jQuery built-in queue

The example above builds the operation in 3 steps. Each step function can evolve separately by altering the contents of those functions. The delay can be implemented anywhere between any of the steps.

The problem now is that we are calling next() a lot of times on each function. Since the only thing we care is when the steps change and we don't use any async flow, we don't even need to explicitly call next() anywhere by abstracting that into a function:

The code using a function to abstract the creation of the step

At this point we have much less code than the first example, the steps are declared as one-liners and we repeat only what we know that may change.

One could even argue that jQuery is not enough of a solution for this and we should start our application with Angular, React, VueJS, etc. because it’s more likely that the application will evolve and the cost of changing it later will be much bigger...

Wait! Why are we talking about frameworks, queues, and steps?

Let's go back to the original requirements:

Build 3 clickable images where clicking on each one of them will cause the one being clicked to become bigger double the original size and after 1 second to become smaller again to the original size.

Was this Over-Engineered? If so, at which point exactly?

A comic with a person eating on a table and asking for the second one "Can you pass the salt?". After some time, given that there's no response, the first person asks again "I said-" and is interrupted by the second person with "I know! I'm developing a system to pass you arbitrary condiments.". The first person says "It's been 20 minutes!" and the second one "It'll save time in the long run".

I think the problem is not about the proposed solution, it is about the requirements. Instead of start implementing the solution and coding it, the engineer should have taken a step back and looked after the root of the problem by asking "why?"

To reduce the chances of considering something as Over-Engineering we need to understand the problem and to understand the problem we need to clarify the requirements as much as possible by asking "why"?

Of course, as a general rule, the usage of software principles that were already discovered should not be considered Over-Engineering. However, even if the requirement were just to make two buttons bigger and then smaller, one could argue that making it extensible and decoupled would be valuable, others can argue that just if the requirement were part of a bigger system then that would justify the effort of being developed using a framework or jQuery, otherwise the first example was enough.

The point is that Over-Engineering is subjective and the damage of its subjectiveness increase as the requirements fail to present the full picture of the problem the engineer is supposed to solve. However, even when the full picture is available, someone could argue that building a better foundation is worth more than a naive solution that can have more problems in the long term than an apparent over-engineered one.

Over-Engineering is subjective and the damage of its subjectiveness increase as the requirements fail to present the full picture of the problem

Don’t disregard something for being "Over-Engineered" if you can’t understand the purpose and the implications in the long term and don't try to do more than necessary before taking a step back and understanding the full picture of the problem.

At the end of the day, the definition of Over-Engineering will vary according to each individual and context. Those involved in the project will dictate what it is and what it isn't based on their opinions.

Unfortunately, there's no universal answer for what always constitutes Over-Engineering and what doesn't.

And we need to live with that…

Edit October 27, 2016:

Improved the requirements used as an example according to the feedback of MoreMoschops.

Thanks for reading. If you have some feedback, reach out to me on Twitter, Facebook or Github.

Wanna chat in person? You can find me in the Sydney Software Crafters meetup.