“Measure twice, cut once.” It is an old cliché but the team at Segment (the customer data platform) learned it the hard way. Segment In 2018 their engineering team followed the industry hype and broke their core product into 140 microservices. It was an architectural marvel. It followed every “best practice” in the book. 140 microservices Three years later they published a famous post admitting it was a disaster. The complexity had exploded. Engineers were spending more time managing message queues and debugging distributed traces than shipping features. In 2020 they had to burn months of effort to move everything back to a Monolith. What went wrong? They jumped to the Solution (Microservices are scalable!) before rigorously validating the Problem (Does our team size justify this operational complexity?). Solution Problem The root cause is always the same. The rush to code beats the discipline to think. The rush to code beats the discipline to think. Senior engineers prevent this with a deceptively simple practice. They write things down before they write code. Not because they love documentation but because three days of planning prevents three years of pain. The “Google Doc” That Saves Months When an engineer says “let me write an RFC before we start” they are not adding bureaucracy. They are buying insurance. RFC stands for Request for Comments. It is a structured document that forces you to answer the hard questions before you open your IDE. RFC Request for Comments The magic isn’t the format. The magic is that it forces Explicit Thinking about Implicit Assumptions. Explicit Thinking Implicit Assumptions In the Segment case, an RFC would have surfaced the core question: “Do we actually have the staff to maintain 140 separate deployment pipelines?” “Do we actually have the staff to maintain 140 separate deployment pipelines?” Someone would have commented: “No. We will spend 50% of our week just keeping the lights on.” “No. We will spend 50% of our week just keeping the lights on.” That comment would have saved them millions of dollars. What Good Looks Like You don’t need a complex template. You just need to answer six questions in plain English. 1. What problem are we solving? Don’t use vague product-speak. Use specific pain. 1. What problem are we solving? Bad: “We need to re-architect the backend.”Good: “Our current monolithic build takes 45 minutes to deploy. This costs us 10 developer-hours per day in waiting time.” Bad: “We need to re-architect the backend.” Bad: Good: “Our current monolithic build takes 45 minutes to deploy. This costs us 10 developer-hours per day in waiting time.” Good: 2. What approaches did we consider? List at least three options. This proves you didn’t just fall in love with your first idea. 2. What approaches did we consider? Option A: Microservices (High scale, high operational cost).Option B: Modular Monolith (Medium scale, low cost).Option C: Serverless (High vendor lock-in). Option A: Microservices (High scale, high operational cost). Option A: Option B: Modular Monolith (Medium scale, low cost). Option B: Option C: Serverless (High vendor lock-in). Option C: 3. What are we proposing and why? Give a recommendation with clear reasoning. 3. What are we proposing and why? Verdict: “We recommend Option B. It solves the deployment speed issue without requiring us to hire a dedicated DevOps team.” Verdict: “We recommend Option B. It solves the deployment speed issue without requiring us to hire a dedicated DevOps team.” Verdict: 4. What are the tradeoffs? Every decision has a cost. If you pretend there are no downsides nobody will trust your plan. 4. What are the tradeoffs? Tradeoff: “A Modular Monolith is harder to split up later if we grow to 500 engineers. We accept this risk because we are currently only 20 engineers.” Tradeoff: “A Modular Monolith is harder to split up later if we grow to 500 engineers. We accept this risk because we are currently only 20 engineers.” Tradeoff: 5. What could go wrong? Paranoid thinking during planning prevents panic during execution. 5. What could go wrong? Risk: “What if the build time doesn’t actually improve?”Mitigation: “We will prototype the build pipeline for one module before migrating the rest.” Risk: “What if the build time doesn’t actually improve?” Risk: Mitigation: “We will prototype the build pipeline for one module before migrating the rest.” Mitigation: 6. What are the open questions? Admitting what you don’t know encourages others to help you. 6. What are the open questions? Question: “Do we need to update our CI/CD provider to support this?” Question: “Do we need to update our CI/CD provider to support this?” Question: The AI “Red Team” The hardest part of writing an RFC is seeing your own bad ideas. You are biased. You want your solution to work. This is where AI becomes a superpower. Not for writing the text but for challenging the logic. challenging the logic. Before I share a draft with my team I paste the “Tradeoffs” section into Claude or ChatGPT and I use this prompt: “Act as a cynical Principal Engineer. Read this architectural proposal. Tell me exactly how this system will fail at 10x scale and list three edge cases I have forgotten.” “Act as a cynical Principal Engineer. Read this architectural proposal. Tell me exactly how this system will fail at 10x scale and list three edge cases I have forgotten.” It simulates a harsh review instantly. It finds the holes in your logic before your boss does. Use AI to stress-test your thinking, not just to fix your grammar. The “Async” Review The best part of an RFC is that it kills meetings. You write the doc. You share it in Slack or Teams. You let people comment on their own time for 3 days. The Skeptic will ask: “This database schema looks complex. Have you considered a simple key-value store?”The Product Manager will clarify: “Actually we need to support offline mode. This architecture relies too much on the cloud.”The Junior will ask: “How does this handle user timezones?” The Skeptic will ask: “This database schema looks complex. Have you considered a simple key-value store?” The Skeptic The Product Manager will clarify: “Actually we need to support offline mode. This architecture relies too much on the cloud.” The Product Manager The Junior will ask: “How does this handle user timezones?” The Junior You fix the flaws in the document. Fixing a flaw in a Google Doc takes 5 minutes. Fixing that same flaw after you wrote the code takes 5 days. Disagreement during the RFC phase is productive. Disagreement during the Code Review phase is expensive. Disagreement during the RFC phase is productive. Disagreement during the Code Review phase is expensive. A Real Example: The “Comfort Zone” Trap A fintech team needed to store data for a new transaction service. They had a “Golden Hammer” which was a specific technology stack they loved and used for absolutely everything. When this new complex feature came up the team naturally reached for their favorite tool again. It was the comfortable choice. But the Lead Engineer wrote an RFC to challenge it. He pointed out that their “favorite tool” was actually terrible at handling the specific requirements of this new feature. He wrote: “If we use our standard tool we will have to write massive amounts of hacky code to make it work. It will be buggy and slow. I propose we use a specialized tool that is designed for this exact problem.” “If we use our standard tool we will have to write massive amounts of hacky code to make it work. It will be buggy and slow. I propose we use a specialized tool that is designed for this exact problem.” The team pushed back. “We don’t know that new tool! It will take too long to learn!” “We don’t know that new tool! It will take too long to learn!” But the RFC made the tradeoffs clear. The cost of learning the new tool was high but the cost of maintaining the hacky solution was higher. The team agreed. learning maintaining Six months later that decision saved them. The feature ran perfectly. If they had skipped the RFC and stayed in their comfort zone they would have been drowning in technical debt. When to Skip It Don’t be the person who writes an RFC for a button color change. Skip the RFC for: Skip the RFC for: Small, reversible decisions.Urgent bug fixes.Work that takes less than 3 days. Small, reversible decisions. Urgent bug fixes. Work that takes less than 3 days. Write the RFC for: Write the RFC for: Architectural changes.New tools or libraries.Decisions that are hard to reverse (like picking a database or a framework). Architectural changes. New tools or libraries. Decisions that are hard to reverse (like picking a database or a framework). Your Homework Look at the biggest ticket in your current sprint. Ask yourself: “If I am wrong about how I plan to build this, how expensive will it be to fix later?” “If I am wrong about how I plan to build this, how expensive will it be to fix later?” If the answer is “Very Expensive” then open a blank document. Write down the problem, three solutions, and your recommendation. Send it to your team. You might just save yourself three months of work.

The is an opinion piece based on the author’s POV and does not necessarily reflect the views of HackerNoon.

The Most Expensive Code is the Code Nobody Wants

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

AI-Assisted Coding with Tabnine

DOE v. Github: Code Stored on GitHub Ultimately Belong to Their Authors

DOE v. Github: Defendants' Conduct Raises Questions Related to DMCA Violations, Privacy, Data Breach

DOE v. Github: Defendants Never Sought Permission to Use Others' Code to Train Codex or Copilot

DOE v. Github: Lawsuit Falls Under California's Jurisdiction Because of GitHub & OpenAI's HQ

DOE v. Github: Plaintiffs Became Aware of the Use of Their Code After Copilot Was Launched

AI-Assisted Coding with Tabnine

DOE v. Github: Code Stored on GitHub Ultimately Belong to Their Authors

DOE v. Github: Defendants' Conduct Raises Questions Related to DMCA Violations, Privacy, Data Breach

DOE v. Github: Defendants Never Sought Permission to Use Others' Code to Train Codex or Copilot

DOE v. Github: Lawsuit Falls Under California's Jurisdiction Because of GitHub & OpenAI's HQ

DOE v. Github: Plaintiffs Became Aware of the Use of Their Code After Copilot Was Launched

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps