When Over-Engineering Is A Good Idea

How often have you worked on a project that was REQUIRED to over-engineer and be “internet-scale”?

These projects bring a ton of assumptions – handling millions of users, hundreds of requests per second, and massive storage needs. It takes months to build these high-scale features.

Delays will happen. Stakeholders get frustrated. And yet, your team gets it done.

Unfortunately, the only user is Steve in Marketing. He needed to email a fancy report once a week.

I’ve encountered this situation TOO MANY times in my career. Sometimes I caused the problem. Other times I had to talk people off a ledge.

Over the years, I’ve learned that the right time to over-engineer a project depends on one thing:

Understanding the problem you’re trying to solve.

What problem are you ACTUALLY solving when you over-engineer?

Engineers are enamored with the idea of “What’s possible?“. We love building things and thinking about the future. Unfortunately, that excitement leads to problems from above.

Thinking about “what’s possible?” adds scope.

To get around this, you have to focus your energy. Don’t take the requirements for granted. Pay attention to the assumptions your team is making. Ask the critical question to your stakeholder:

What outcome is the customer looking to achieve?

But your curiosity should not end there. You need to understand the context you’re working in. Look at how your broader company works.

• What are the goals of the business this year?
• Does your company sell to small businesses or enterprises?
• What resources does your team have at its disposal?

This information will show you the trade-offs you need to make and where to invest your time.

So when exactly should you over-engineer a project?

When should I ship a software project as fast as possible?

I optimize for shipping a project as fast as possible when the goal is to learn.

Start-ups are the best example of this. The early days of a start-up business prioritizes product-market fit. The goal is to find customers and learn quickly.

It’s OK to write just enough code if it means you ship.

We can always make the system more resilient later.

So if your company is exploring a new domain, optimize for shipping over robust systems.

When should I over-engineer my software project?

I recommend over-engineering a project when it meets two requirements:

1. The system is needed to achieve long-term business goals.
2. You have a deep understanding of the problem space.

Over-engineering is perfect for systems that are core to the business. These are old systems that are used by most customers and keep getting new features. Examples include Google Search, Facebook’s News Feed, or even the user permission settings in your web app.

Each of these systems has had some sort of re-write over the years. While risky, their success is due to understanding the codebases and aligning a rewrite with the company’s long term goals.

I’ve personally encountered this in my day job. Over the past six years, my team migrated our app to three different database technologies. Each migration was worth it until we hit a mixture of traffic scale and use cases that pushed the system to the limit.

Each re-write was a success thanks to great planning, aligned goals, and healthy project estimation.

Using Jobs to Be Done to know when to over-engineer a Project

The Jobs to Be Done framework is a great tool to evaluate when to over-engineer a project. This framework asks you to think about the specific job customers hire your product to perform.

My favorite example of this comes from Competing Against Luck when the author tries to answer the question:

“Why do people by McDonalds milk shakes in the morning?”

…what these milk shake buyers had in common had nothing to do with indvidual demographics. Rather, they all shared a common job they needed to get done in the morning.

“Help me stay awake and occupied while I make my morning commute more fun.”

Christensen, Clayton M., et al. Competing against Luck the Story of Innovation and Customer Choice. New York, N.Y. Harper Business, An Imprint Of Harpercollins Publishers, 2016.

These people weren’t just buying milkshakes as desserts to their meal. They wanted something with high calories, easy to consume while driving, and quick to buy on their way to work.

Understanding the specific outcomes your customer is trying to achieve will help you know when to over-engineer a project.

When the outcome is to learn, forego quality and ship fast.

But when the goal is to scale, then over-engineer the hell out of your software.

How do you figure out when to over-engineer a project? Let me know on LinkedIn or Twitter.

Further Reading

• Competing Against Luck by Clayton Christensen

• The Critical Path Episode 19

• The Critical Path Episode 134

• The Critical Path Episode 146