If you've spent any time on engineering Twitter or reading system design interview prep, you've probably internalized this idea: Complexity is progress. Complexity is progress. Microservices, Kafka, sharding, event sourcing... these are the words of the gods, the hallmarks of a real senior engineer, right? real Well, let me tell you a story based on the four phases of an engineer’s system design journey. The Four Stages of Architectural Grief (and Wisdom) This isn't about skill level; it's about scars. 1. The Junior SWE: The Single Service Dream "Let's just build one backend and one DB for everything." "Let's just build one backend and one DB for everything." The Junior engineer chooses simplicity because they just want the damn thing to work. One codebase, one database, one deploy command. The mental model is small, clear, and predictable. They are optimizing for shipping velocity and debugging ease. They don't have the scars yet, so they don't see the limits. shipping velocity debugging ease 2. The Mid-level SWE: The Pattern Enthusiast "We should move this into microservices with separate databases." "We should move this into microservices with separate databases." This is the engineer who just read Designing Data-Intensive Applications. They’ve learned about scalability, domain-driven design, queues, and the "Monolith is Bad" mantra. They are eager to apply every pattern they know. They are optimizing for resume bullet points and theoretical scalability. This is the most dangerous phase, where you create problems to use your newly acquired solutions. Designing Data-Intensive Applications resume bullet points theoretical scalability 3. The Senior SWE: The Anticipator of Doom "We need sharding, Kafka, CQRS, Redis, S3, and a global cache." "We need sharding, Kafka, CQRS, Redis, S3, and a global cache." The Senior engineer has seen the Monolith break. They are anticipating actual failures, potential spikes in traffic, and geographical distribution. They are optimizing for worst-case scenario reliability and preemptive scale. Their design is often correct for the scale they might reach, but not the scale they have today. This complexity debt starts to slow the team down. actual worst-case scenario reliability preemptive scale might have 4. The Principal SWE: The Complexity Survivor "Can this just be a single service with a read replica?" "Can this just be a single service with a read replica?" The Principal engineer has paid the price for every unnecessary piece of complexity. They have lived through the 3 AM on-call pages caused by a Kafka broker failing, the subtle race conditions between two separate microservice databases, and the bug where three different components were slow to deploy together. They are optimizing for on-call happiness and team velocity. They understand the cost of complexity. on-call happiness team velocity The Real Shift: From Tools to Constraints The fundamental difference between these four stages isn't a better knowledge of tools. It’s an evolving sense of constraints. sense of constraints. Junior: Constraint is getting it to work.Mid-Level: Constraint is using the right pattern.Senior: Constraint is handling future scale.Principal: Constraint is organizational cognitive load and operational pain. Junior: Constraint is getting it to work. Junior: getting it to work Mid-Level: Constraint is using the right pattern. Mid-Level: using the right pattern Senior: Constraint is handling future scale. Senior: handling future scale Principal: Constraint is organizational cognitive load and operational pain. Principal: organizational cognitive load operational pain The Principal knows that the most expensive part of any system is the people managing the complexity, not the servers running the code. people My Advice: How to Advance Your System Design Thinking If you’re still learning system design—and I mean, truly learning for building real-world systems, not just for an interview—here is the direction I’d point you in: truly 1. Start with the Smallest Viable Thing The simple setup is almost always the right starting point. The simple setup is almost always the right starting point. Single service, single database, clear API boundaries.The first goal is not scalability; it’s making sure you can monitor it, debug it, and deploy it safely (i.e., you know how to roll back when things go sideways). Single service, single database, clear API boundaries. The first goal is not scalability; it’s making sure you can monitor it, debug it, and deploy it safely (i.e., you know how to roll back when things go sideways). monitor it, debug it, and deploy it safely 2. Learn How Systems Fail Before You Learn How They Scale Scaling is sexy; failure is the harsh teacher. Read postmortems. Don't just read about how Netflix built their architecture; read about the time they went down.Study the outages of big companies (Google, AWS, Twitter). Why did the cascading failure happen?You will quickly understand why Principal Engineers are scared of unnecessary complexity—it's the source of 90% of your incidents. Read postmortems. Don't just read about how Netflix built their architecture; read about the time they went down. Read postmortems. Study the outages of big companies (Google, AWS, Twitter). Why did the cascading failure happen? Study the outages of big companies You will quickly understand why Principal Engineers are scared of unnecessary complexity—it's the source of 90% of your incidents. 3. Stop Collecting Tools, Start Collecting Trade-offs Don't just add "Kafka" to your skillset. Learn when Kafka is useful and when it is overkill. when Kafka is useful and when it is overkill. Tool/PatternWhen it's usefulThe Trade-off (The Pain)MicroservicesWhen teams and codebases must be decoupled for parallel work.Distributed transactions, network latency, monitoring complexity.Kafka/QueuesWhen high-volume events need durable, asynchronous processing.Managing brokers, message delivery guarantees, at least once semantics.ShardingWhen a single database instance hits physical limits (disk space or IOPS).Complex joins, operational overhead, rebalancing data, and managing distributed transactions. Tool/PatternWhen it's usefulThe Trade-off (The Pain)MicroservicesWhen teams and codebases must be decoupled for parallel work.Distributed transactions, network latency, monitoring complexity.Kafka/QueuesWhen high-volume events need durable, asynchronous processing.Managing brokers, message delivery guarantees, at least once semantics.ShardingWhen a single database instance hits physical limits (disk space or IOPS).Complex joins, operational overhead, rebalancing data, and managing distributed transactions. Tool/PatternWhen it's usefulThe Trade-off (The Pain)MicroservicesWhen teams and codebases must be decoupled for parallel work.Distributed transactions, network latency, monitoring complexity.Kafka/QueuesWhen high-volume events need durable, asynchronous processing.Managing brokers, message delivery guarantees, at least once semantics.ShardingWhen a single database instance hits physical limits (disk space or IOPS).Complex joins, operational overhead, rebalancing data, and managing distributed transactions. Tool/PatternWhen it's usefulThe Trade-off (The Pain) Tool/Pattern Tool/Pattern When it's useful When it's useful The Trade-off (The Pain) The Trade-off (The Pain) MicroservicesWhen teams and codebases must be decoupled for parallel work.Distributed transactions, network latency, monitoring complexity. Microservices When teams and codebases must be decoupled for parallel work. Distributed transactions, network latency, monitoring complexity. Kafka/QueuesWhen high-volume events need durable, asynchronous processing.Managing brokers, message delivery guarantees, at least once semantics. Kafka/Queues When high-volume events need durable, asynchronous processing. Managing brokers, message delivery guarantees, at least once semantics. ShardingWhen a single database instance hits physical limits (disk space or IOPS).Complex joins, operational overhead, rebalancing data, and managing distributed transactions. Sharding When a single database instance hits physical limits (disk space or IOPS). Complex joins, operational overhead, rebalancing data, and managing distributed transactions. 4. Always Design with Three Time Horizons in Mind Your goal is not to design the fanciest system. Your goal is to design the smallest system that solves the problem and can grow when it actually needs to. smallest system that solves the problem and can grow when it actually needs to. actually Today: Can we ship something reliable quickly?1 Year: Can we change this system without rewriting everything?3 Years: If this becomes 10x bigger, can we evolve it (e.g., then add sharding) instead of replacing it? Today: Can we ship something reliable quickly? Today: 1 Year: Can we change this system without rewriting everything? 1 Year: 3 Years: If this becomes 10x bigger, can we evolve it (e.g., then add sharding) instead of replacing it? 3 Years: 10x bigger then In the end, engineering maturity is the ability to choose boring, simple solutions because you understand the operational cost of being too clever.

Netflix

Mind

Load Balancer vs. API Gateway: Stop Confusing Your Backend Architecture

The System Design Paradox: Why Your Principal Engineer Asks for Less, Not More

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Django 6.0 Is Here: Background Tasks, Partials, and Why We Might Finally Delete Celery

How Machine Generated Virtual Assistants can 10x Your Productivity in 2022

My "Serverless" Dream Turned into an IP Ban Nightmare: Why I Moved to Desktop

Adapting to AI-Powered Workplaces and an Automated Future

The Noonification: Everything Missing in Cosmos DeFi (12/28/2023)

47 Stories To Learn About Kafka

Django 6.0 Is Here: Background Tasks, Partials, and Why We Might Finally Delete Celery

How Machine Generated Virtual Assistants can 10x Your Productivity in 2022

My "Serverless" Dream Turned into an IP Ban Nightmare: Why I Moved to Desktop

Adapting to AI-Powered Workplaces and an Automated Future

The Noonification: Everything Missing in Cosmos DeFi (12/28/2023)

47 Stories To Learn About Kafka

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps