Understanding Application Architecture that Scales

This article aims to give a fundamental analysis of the applications development life-cycle, identify the main contradiction of software development, and suggest a solution.

I started learning programming back in 2006 when my older brother had his informatics class. Back in those days, in an average Ukrainian school, they were learning programming on Pascal, running the “IDE” in DOS mode on Windows XP. I quickly understood the syntax and started solving easy tasks, like finding a square of rectangles, or length of hypotenuses. The pattern was pretty simple: you ask for the input data, do a calculation and print the result.

It was the first feeling of power, you could do whatever you want, at least I thought so at that moment.

Later, when I switched to PHP to build simple websites, I started learning new concepts: loops, functions, arrays, and eventually classes. That was about the time when PHP5 by Dmitry Koterov

was released and it brought a new killer feature — support of OOP.

For every concept, I remember my confusion “Why does someone want that?”, “What is this for?”. Why do we need arrays, if we can define multiple variables? Why do we need functions if we can write inline?

Really, if your task solution fits a screen, why do you need to break it into multiple functions? You start breaking into files and classes only if a solution exceeds a dozen functions. I eventually realized the usefulness of those concepts when I encountered a bigger problem.

As development projects progress, the complexity of code tends to expand, leading to a constant struggle between writing more code and maintaining manageable context. Starting with screens, functions, and then breaking them down into classes, files, and beyond, the expansion of complexity challenges the limitations of the human brain.

While the industry has provided clear abstractions up to classes, the path forward becomes less defined. Traditional solutions like the Model-View-Controller (MVC) pattern have been widely adopted but can result in a multitude of flat files within the app/models folder. Alternatively, microservices have gained popularity, but implementing and coordinating numerous isolated services can introduce significant overhead.

Finding the right balance between code organization and context management becomes crucial as the industry navigates this evolving landscape of abstractions and scalability.

In the pursuit of managing complexity and maintaining manageable context, I have developed an approach that combines various renowned techniques such as Test-Driven Development (TDD), Domain-Driven Design (DDD), and SOLID principles into a cohesive structure. While not entirely novel, this compilation aims to provide a comprehensive resource that organizes my experiences and offers practical value to you, the reader.

We can start by defining a couple of invariants to make the process more formal:

• complexity constantly grows with codebase size
• the human attention context is constant and limited with 5–7 entities simultaneously

We can reach the conclusion that the only solution for this situation is composition over abstractions. The whole development process can be illustrated by the good old game “2048".

How many software projects died because of growing complexity? It starts slowly by increasing the time it takes to implement a feature, the number of unexpected bugs, and 500 errors on production. And it always ends up with the same result: a feature that typically takes days can’t be implemented in a month; when a bugfix introduces three new bugs; when everybody is afraid to make a structural change because nobody understands how the system works. It’s a doom spiral where efforts to continue development makes things even worse.

The approach I'm describing involves a recursive modular structure with limited entities on each level, assuming constant refactoring. Similar to the "2048" game, we begin with a small module containing a few classes or functions. As we expand, we divide larger modules into submodules, and so on.

Each module must possess its own logical interface, inputs, and outputs, allowing independent operation within the application. Think of it as a collection of micro-services, but without the explicit division, all functioning seamlessly together.

Consider the application as a car. ChatGPT informed me that a modern car consists of approximately 30,000 parts. Can one comprehend them all at once? Certainly not. However, you can likely name the main components: engine, transmission, steering, exterior, interior, and suspension. They maintain a distinct, well-defined interface with each other. The interactions between the engine and exterior, for example, are simply facilitated by the three engine mounts in between.

You can definitely easily tell whether it’s an engine problem or a cracked bumper. If you’re working with an engine, you don’t need to think about the steering wheel or about the rear door. But at the same time, you expand the engine context — now you need to think about intake, exhaust, fuel injection, cooling, and electronics. It’s an example of a great interface that was established for a well-known thing within the last hundreds of years. But as developers, we work with more specific and unique products, in a young industry. We are the ones who got to define modules and interfaces between them, even if GPT will take our coding jobs tomorrow.

In the next articles, I’m going to talk a little more specifically about modules in practice and show how DDD, TDD, and SOLID techniques can help you drive the architecture.

I hope you enjoyed it. Thank you for your reading.