If you've spent any time on engineering Twitter or reading system design interview prep, you've probably internalized this idea: Complexity is progress.
Microservices, Kafka, sharding, event sourcing... these are the words of the gods, the hallmarks of a real senior engineer, right?
Well, let me tell you a story based on the four phases of an engineer’s system design journey.
The Four Stages of Architectural Grief (and Wisdom)
This isn't about skill level; it's about scars.
1. The Junior SWE: The Single Service Dream
"Let's just build one backend and one DB for everything."
The Junior engineer chooses simplicity because they just want the damn thing to work. One codebase, one database, one deploy command. The mental model is small, clear, and predictable. They are optimizing for shipping velocity and debugging ease. They don't have the scars yet, so they don't see the limits.
2. The Mid-level SWE: The Pattern Enthusiast
"We should move this into microservices with separate databases."
This is the engineer who just read Designing Data-Intensive Applications. They’ve learned about scalability, domain-driven design, queues, and the "Monolith is Bad" mantra. They are eager to apply every pattern they know. They are optimizing for resume bullet points and theoretical scalability. This is the most dangerous phase, where you create problems to use your newly acquired solutions.
3. The Senior SWE: The Anticipator of Doom
"We need sharding, Kafka, CQRS, Redis, S3, and a global cache."
The Senior engineer has seen the Monolith break. They are anticipating actual failures, potential spikes in traffic, and geographical distribution. They are optimizing for worst-case scenario reliability and preemptive scale. Their design is often correct for the scale they might reach, but not the scale they have today. This complexity debt starts to slow the team down.
4. The Principal SWE: The Complexity Survivor
"Can this just be a single service with a read replica?"
The Principal engineer has paid the price for every unnecessary piece of complexity. They have lived through the 3 AM on-call pages caused by a Kafka broker failing, the subtle race conditions between two separate microservice databases, and the bug where three different components were slow to deploy together. They are optimizing for on-call happiness and team velocity. They understand the cost of complexity.
The Real Shift: From Tools to Constraints
The fundamental difference between these four stages isn't a better knowledge of tools. It’s an evolving sense of constraints.
- Junior: Constraint is getting it to work.
- Mid-Level: Constraint is using the right pattern.
- Senior: Constraint is handling future scale.
- Principal: Constraint is organizational cognitive load and operational pain.
The Principal knows that the most expensive part of any system is the people managing the complexity, not the servers running the code.
My Advice: How to Advance Your System Design Thinking
If you’re still learning system design—and I mean, truly learning for building real-world systems, not just for an interview—here is the direction I’d point you in:
1. Start with the Smallest Viable Thing
The simple setup is almost always the right starting point.
- Single service, single database, clear API boundaries.
- The first goal is not scalability; it’s making sure you can monitor it, debug it, and deploy it safely (i.e., you know how to roll back when things go sideways).
2. Learn How Systems Fail Before You Learn How They Scale
Scaling is sexy; failure is the harsh teacher.
- Read postmortems. Don't just read about how Netflix built their architecture; read about the time they went down.
- Study the outages of big companies (Google, AWS, Twitter). Why did the cascading failure happen?
- You will quickly understand why Principal Engineers are scared of unnecessary complexity—it's the source of 90% of your incidents.
3. Stop Collecting Tools, Start Collecting Trade-offs
Don't just add "Kafka" to your skillset. Learn when Kafka is useful and when it is overkill.
| Tool/Pattern | When it's useful | The Trade-off (The Pain) |
| Microservices | When teams and codebases must be decoupled for parallel work. | Distributed transactions, network latency, monitoring complexity. |
| Kafka/Queues | When high-volume events need durable, asynchronous processing. | Managing brokers, message delivery guarantees, at least once semantics. |
| Sharding | When a single database instance hits physical limits (disk space or IOPS). | Complex joins, operational overhead, rebalancing data, and managing distributed transactions. |
4. Always Design with Three Time Horizons in Mind
Your goal is not to design the fanciest system. Your goal is to design the smallest system that solves the problem and can grow when it actually needs to.
- Today: Can we ship something reliable quickly?
- 1 Year: Can we change this system without rewriting everything?
- 3 Years: If this becomes 10x bigger, can we evolve it (e.g., then add sharding) instead of replacing it?
In the end, engineering maturity is the ability to choose boring, simple solutions because you understand the operational cost of being too clever.