Complexity is the enemy of software engineering

About a year ago, I was at a client's office to help them with a project that they were working on. Now, this client had hired one of the high-end consulting firms for this project. A real top-of-the-line group, the kind that uses pair programming on everything.

Obviously, I can't name names here, but the red flags began for me before I even got a chance to actually look at the project.

As the hiring manager was explaining things to me, he told me how it was a really cool project and they were using X, Y, and Z technologies. Then he told me that this was "definitely one of the most complicated projects" that he'd ever worked on.

This was the first red flag for me.

Some people have the bad habit of assuming that just because a project is complicated, that must mean that it is also good. They assume that as complexity rises, so does value.

This is not the case.

In fact, the truth is usually exactly the opposite. As complexity rises, the quality of the project usually goes down.

The fact of the matter is this: complexity is the enemy of successful software projects.

There are a number of patterns that get a bad rap because they are perceived as overly complicated. If you think about what a pattern is, it's something that we put in place to help us eliminate complexity. By placing the hard-to-understand pieces into abstractions, the patterns deal with complexity so that we don't have to.

A lot of people have had the experience of starting up a project where, in the beginning, everything is really quick and fast.

You get a boilerplate. You add a couple of features. You have an initial burst of energy and creativity around the project. Everything is going quickly and smoothly. Everything is super-high velocity and you're just crushing it.

Over time, though, more features get added onto this project. With each one, you begin to pay a complexity tax.

What began as a simple prototype using something like CRUD operations becomes overly complicated. Especially if you're not paying enough attention to the actions or events occurring.

That is, if you don't have good patterns.

This is how you end up with something referred to as Anemic Domain Design, where the logic is around the model rather than part of it. You modify things and insert the states, and all you end up saving at the end of the day is the current state.

There are big problems with an Anemic Domain Design. Martin Fowler, author of Refactoring: Improving the Design of Existing Code describes it as an "anti-pattern" that only looks like design on "first blush." The problem? As you dig deeper, you see there is "hardly any behavior" attached to the objects within it.

What's the practical upshot of this anti-pattern and its minimal behavior? It's a nightmare for developers when they need to change something.

This does not make for the kind of long-lived and easily maintained software that we should be aiming for. Why would we want to make things complicated, when we can make them easy?

For the alternative to this kind of principle, we turn to Eric Evans, the author of Domain-Driven Design: Tackling Complexity in the Heart of Software and father of Domain-Driven Design (DDD).

Specifically, we're going to look at microservices.

In simple terms, this is something that inputs messages and then outputs messages. (You can check out his excellent presentation on DDD and microservices here, if you're interested for a more in-depth explanation.)

There are two types of messages that are really important for these systems. The first is commands, and the second is events.

Keeping these in mind allows us to use a pattern called a servicebus. Think of it as basically an event-emitter for your whole system. There are multiple ways to use it, and advantages and disadvantages to each one.

The main takeaway is that by using this pattern, you're abstracting away the tools that you are using to implement this functionality. What would otherwise be spread out and potentially difficult for a dev to locate is contained within consistent boundaries.

Another thing that happens without a servicebus is that, because your microservices have to communicate in some way, many end up making REST calls. And if you're dealing with REST, then REST isn't as reliable as a persistent message queue.

To fix that, you end up bringing in other things like Istio or other service meshes. Instead of just, you know, using a reliable messaging protocol in the first place.

These are just some of the tools that we have at our disposal in what must be a constant battle against complexity. A lot of patterns may have complicated names, but the reality is that they are not actually that complicated when looked at from a high level.

Of course, all this is moot if you don't have the right abstractions in place when you go to implement them.

The lesson that we should take away from all this? If you hear something is complicated, that should be your red flag that something has gone wrong.

A lot of patterns, when presented by themselves, are actually quite simple, even if they appear complex. Event sourcing, for example, is just merging together the payloads of previous events.

While the learning curve can be difficult to climb at the beginning when you're unfamiliar with the patterns, over time you will develop small, self-contained services which are only a couple hundred lines of code each. Sometimes even less.

The good part of something like that?

This also will increase the reusability of your ideas. It eliminates noisy communication patterns.

So, what is the key to long-lived, maintainable software?

Otherwise, KISS (Keep it simple stupid).