How to Dive into a New Domain and Ship a High-Load System Fast

I’d like to share my experience of how, after joining one team, I was transferred to another — into a completely new domain — and how I managed to dive in fast and build a system that became the main workhorse, handling 1000 RPS and running 24/7. 1000 RPS 24/7 I’ll share real examples, practical advice, and cases from my own work — the article will be useful both for experienced seniors and managers, and for those who are just starting out. real examples, practical advice How to Dive into a New Domain Fast First of all, you need to understand the new domain — don’t drown in documentation or glossaries. Don’t get stuck on terminology. The best approach is field learning. field learning If it’s a ride-hailing app — take a few rides through your own product. If it’s a marketplace — order items from different categories, go through the return process. No amount of documentation or training videos will replace real experience of using the product in real conditions. real experience My case: on the second day I went to the actual site to see how the system worked in real life. I didn’t understand anything — I was just pressing buttons randomly. But after leaving the site and coming back the next day, I already had a clear understanding of what was expected from me — and that field trip helped me many times later. My case: Preparation Stages Don’t rush to start coding, drawing diagrams, or building things right away. First, you need to go through several essential stages. Stage 1: Understanding business requirements and doing a quick surface analysis Stage 1: The goal is simple — you should be able to clearly explain what is expected from you in five sentences. explain what is expected from you in five sentences Remember: if you can’t explain the value — what you’re building and what problem it solves — in simple words to a non-technical person, it’s too early to start doing anything. Remember: Stage 2: Drafting the architecture and main components Stage 2: Below is an example of the architecture diagram — it shows, for instance, that there will be three services and how their responsibilities are divided. Stage 3: Deadlines and Commitments Stage 3: Here you need to realistically assess your capacity and set deadlines for your tickets. There must always be final dates — even if approximate. Clearly define what goes into the MVP and what can wait. Break the work into stages and set timelines for each. Break the work into stages Follow a simple principle: MVP → Increment 1 → Increment 2. MVP → Increment 1 → Increment 2. Tools You definitely need a task tracker — choose any, whether it’s Jira or something similar. Jira I also highly recommend using Miro — it’s an incredibly powerful tool that lets you quickly create both complex and simple diagrams, sketch out UI ideas, and collaborate in real time — literally out of the box in five minutes. Miro Another useful tool is an actual whiteboard and a marker. I wouldn’t call it practical today, since most teams work remotely, but back in the day it was priceless. I still remember those moments fondly — when real systems were born right there in meeting rooms, drawn by hand, with a sense of physical involvement. whiteboard and a marker Concept of the Future System Before writing the very first line of code, it’s crucial to define the principles you’re going to follow. You need to agree on the core business foundations upfront. core business foundations For example, if it’s a product ordering service — a marketplace — the principles might look like this: Trust is more important than conversion. The user must be confident that the order will arrive on time, and the seller — that they will receive the payment.Speed = revenue. Every second of delay during checkout is a minus to conversion.Predictability and consistency. If an item is marked “in stock,” it must really be available. Trust is more important than conversion. The user must be confident that the order will arrive on time, and the seller — that they will receive the payment. Trust is more important than conversion. Speed = revenue. Every second of delay during checkout is a minus to conversion. Speed = revenue. Predictability and consistency. If an item is marked “in stock,” it must really be available. Predictability and consistency. This allows you to rely on these pre-agreed principles and avoid unnecessary debates. For example, let’s say you’re building a feature called “Autumn Picks” — a curated selection of the most popular seasonal items. For example, “Autumn Picks” A discussion arises: it could be done faster (in one week), but you wouldn’t be 100% sure that all listed items are actually available. That would violate the first principle — trust over conversion — so you consciously take one extra week to refine stock validation and ensure the data is accurate. trust over conversion How to design the architecture when you don’t know all the nuances yet The main principle to follow is don’t overcomplicate. It’s dangerous to dive straight into details. don’t overcomplicate Always ask yourself: Can we launch (solve the business problem) without doing this? If the answer is yes, don’t do it now. Can we launch (solve the business problem) without doing this? yes A good practice is to create a separate backlog-accumulator task where you attach things that would be nice to have — but only after the target configuration is launched. backlog-accumulator only after What to think through first in the architecture: The core The core The core The very first step is to define the core entities your system will revolve around. Don’t be afraid to get them slightly wrong at the start — that’s normal. The deeper you dive, the clearer these entities will become. core entities Only after that should you add the smaller, supporting ones around the core. 2. Metrics Metrics You absolutely need metrics — to control the service and understand what’s going on. metrics The best combo is Prometheus + Grafana. Focus on business metrics so you can see the real state of the system. But don’t add too many or overcomplicate from day one. Prometheus + Grafana business metrics Start with 5–6 simple metrics (orders, cancellations, etc.). You can always add more once the system settles and it’s clear what really matters. 5–6 simple metrics Remember: metrics are not about analytics or funnels — they’re a real-time tool so you can glance at a chart and tell whether everything is operating normally or not. Remember: not real-time tool 3. Delivery Delivery When the system isn’t wired to prod yet, you can change code 1:1 — literally even copy files. But once real users are involved, a mistake can cost you users and money. It’s better to build controlled delivery upfront. The most common approach is a canary rollout: you run two releases — old and new — and a proxy routes traffic between them based on the canary percentage. controlled delivery canary rollout My recommended steps: 1% → 3% → … → 80% → 100%. Don’t rush the higher steps. Let the system “clear its throat” at each stage. 1% → 3% → … → 80% → 100% 4. Extensibility Extensibility Don’t try to plan for everything or build the perfect architecture — it’s pointless. Don’t try to plan for everything You can only truly see the system from the right perspective and build a solid foundation after running it in production with real users. after It’s completely normal to radically change your database structure several times during development, rewrite parts of the code, or adjust APIs.No genius can design a complex system in a new domain on a whiteboard and have it all work flawlessly without rework. What not to do not Below is what usually turns into wasted time and unjustified complexity in the early stages: Don’t build for the distant future or solve problems that haven’t happened yet Don’t build for the distant future or solve problems that haven’t happened yet I always follow the rule: solve problems as they come. The big risk is to start fixing what hasn’t even occurred. The right approach is to create separate tasks and leave comments in the code about potential issues so you can come back to them. My case: a third-party service returned 25+ different error types. It seemed logical to teach our system to handle each one and wrap them into user-friendly texts right away. Instead, we decided to show one unified message. In the end, that saved a lot of effort — only about 10 of those errors ever appeared in production. My case: ⛔ Premature optimization Premature optimization It’s always tempting to add caching right away, optimize queries, or make calls in parallel. But that almost always turns out to be a bad idea. Before launch, you simply don’t know which parts of the system will actually be under the most load or which products users will buy most often. It’s much smarter to wait, look at real traces and metrics, and then optimize consciously — based on facts, not guesses. real traces and metrics ⛔ Developing without a contract Developing without a contract It’s a bad idea to start building anything before you’ve agreed on the contract with neighboring teams or services. Even a simple alignment — where everyone just says “we’re good with this” — is usually enough to prevent serious mismatches and painful issues during end-to-end testing: rpc CreateOrder(CreateOrderRequest) returns (CreateOrderResponse); message CreateOrderRequest { string idempotency_key = 1; // unique key to ensure idempotent requests string user_id = 2; // identifier of the customer repeated OrderItem items = 3; // list of order items string delivery_address = 4; // delivery address } CreateOrderResponse ... rpc CreateOrder(CreateOrderRequest) returns (CreateOrderResponse); message CreateOrderRequest { string idempotency_key = 1; // unique key to ensure idempotent requests string user_id = 2; // identifier of the customer repeated OrderItem items = 3; // list of order items string delivery_address = 4; // delivery address } CreateOrderResponse ... Launch Finally, the big day comes — the system is written, tested, and ready to go live. The most important thing: prepare well, get some sleep, and assign roles. prepare well, get some sleep, and assign roles. Roles: Roles: Lead: the person responsible for a successful launch and empowered to make decisions that everyone else follows.Reporter: the one documenting bugs and issues that appear during the launch.Fixers: those who resolve problems and deploy fixes — includes both developers and QA engineers.Product/Operations: people who deeply understand the domain and know all the nuances of how things should behave. Lead: the person responsible for a successful launch and empowered to make decisions that everyone else follows. Lead: Reporter: the one documenting bugs and issues that appear during the launch. Reporter: Fixers: those who resolve problems and deploy fixes — includes both developers and QA engineers. Fixers: Product/Operations: people who deeply understand the domain and know all the nuances of how things should behave. Product/Operations: Also, plan in advance what to do if something goes wrong — and how to fix it fast. what to do if something goes wrong My case: we had very tight deadlines, so we didn’t have enough time for proper testing — we basically had to do it on the spot during launch. Honestly, there were a couple of moments of despair when it felt like we wouldn’t make it at all. My case: But in the end, all the major bugs were fixed, and the process finally went live — although instead of the planned three days, the launch took seven. How to Ask People to Report Problems How to Ask People to Report Problems You should pay special attention to creating clear instructions in advance — and formalize how you expect users or operators to report issues. clear instructions Below is an example of a well-structured bug report: well-structured bug report: A proper issue report should include: Scope: how many users or cases are affected.Steps to reproduce: what actions led to the problem.Evidence: a screenshot showing the error (a short video is even better).Description and contact: a short summary and a person to reach out to. Scope: how many users or cases are affected. Scope: Steps to reproduce: what actions led to the problem. Steps to reproduce: Evidence: a screenshot showing the error (a short video is even better). Evidence: Description and contact: a short summary and a person to reach out to. Description and contact: It’s also helpful to include trace_id, timestamp, and the user nickname — so you can quickly locate the error in logs. trace_id timestamp user nickname Here’s a good example of how to ask people to report issues: 🚨 BUG REPORT Title: {what’s not working} Scope: {how many users / cases affected} Steps: {what was done → what happened} Expected: {what should happen} Actual: {what actually happened} Evidence: {screenshot / trace_id / timestamp} Contact: {who to reach out to} 🚨 BUG REPORT Title: {what’s not working} Scope: {how many users / cases affected} Steps: {what was done → what happened} Expected: {what should happen} Actual: {what actually happened} Evidence: {screenshot / trace_id / timestamp} Contact: {who to reach out to} System Lifecycle Problems tend to follow a system for six months to a year after launch. Every system goes through its own stages of maturity: Infant (0–2 months) Infant (0–2 months) Infant At this stage, the system is the most fragile and demanding. Expect the highest number of incidents and bugs. 2. Adolescent (2 months–1 year) Adolescent The system becomes more predictable — both you and the support team start to understand its weak points and where things might break. 3. Mature (6 months–3 years) Mature Your direct involvement is barely needed. The system can run for weeks without attention. 4. Aging (3+ years) Aging By this point, the system likely no longer meets the business needs. Any attempt to adapt it to new realities becomes costly, and a replacement is probably already being prepared. How to Handle Problems (Incidents) Here are the key principles we followed during incidents — critical failures affecting the main flow: Screen sharing. Someone should always be sharing their screen so everyone stays synchronized.Stay calm. No panic, no chaos. Work fast but without rushing.One coordinator. There must be a single incident lead — the person authorized to give instructions that everyone follows.Say out loud what you’re doing. For example: “I’m restarting the service — Lev.” This prevents two people from performing the same action simultaneously. Screen sharing. Someone should always be sharing their screen so everyone stays synchronized. Screen sharing. Stay calm. No panic, no chaos. Work fast but without rushing. Stay calm. One coordinator. There must be a single incident lead — the person authorized to give instructions that everyone follows. One coordinator. Say out loud what you’re doing. For example: “I’m restarting the service — Lev.” This prevents two people from performing the same action simultaneously. Say out loud what you’re doing. “I’m restarting the service — Lev.” Don’t worry if incidents come one after another at the beginning — that’s normal. A new system is like a newborn: during the first days it demands constant attention, but soon both you and it learn to function together smoothly. After Launch You have to understand that writing the code and launching the system is only about 10% of the work. The real challenge begins a few days later — when a flood of bugs starts to appear. 10% of the work Soon after, you’ll probably begin scaling to new regions, markets, or facilities, which will bring even more urgent fixes and “must-happen-now” tasks. Approach everything with a cool head: prioritize bugs, group them by system components and flows. cool head Tip: write down instructions, categorize recurring problems, and share those guides with operations and support teams — it’ll save everyone a lot of time. Tip: Mistakes We Made The launch turned out to be tougher than expected — we seriously underestimated the business complexity of the system we had built. The main issue was that we tested it on ourselves. We did have local test runs to simulate real conditions, but once we handed the system to real users, they started interacting with it in unexpected ways — following scenarios we hadn’t planned for or even imagined. And that’s where most of the surprises came from. unexpected ways What We Did Right In the end, the launch was successful — the system fully covered the business need. A big part of that success came from a few simple but right decisions: We didn’t overcomplicate. Wrote the code quickly and focused on getting it to work.We prepared for production access. Added Swagger endpoints for fast hotfixes.We didn’t give up. We kept pushing until the system was up and running. We didn’t overcomplicate. Wrote the code quickly and focused on getting it to work. We didn’t overcomplicate. We prepared for production access. Added Swagger endpoints for fast hotfixes. We prepared for production access. We didn’t give up. We kept pushing until the system was up and running. We didn’t give up. The Further Life of the System If the system proves itself — stable, reliable, and effective — your stakeholders will most likely want to expand it and add new features. expand it This is where the plateau phase begins — it can last for several years. plateau phase At the start, you had to be like a turbine: dive into the domain fast, build the working core, and launch it quickly. But now, in the feature phase, the mindset must flip completely. The time of quick, on-the-fly deployments is over — every release must be carefully tested and verified. carefully tested and verified Technical Debt Every issue or imperfection you notice while working on new features should be logged with the tag “tech debt.” “tech debt.” It’s better to create one ticket too many and later close it as “not important” than to overlook an obvious weakness that might become a problem later. “not important” Follow this rule: 80% product features, 20% technical debt. This proportion is written in blood. If you don’t dedicate time to tech debt, you’ll drown quickly — at some point, you simply won’t be able to introduce anything new into the existing codebase. 80% product features, 20% technical debt. Sunset Every project, every service will die eventually. It’s an inevitable cycle (remember the System Lifecycle section). System Lifecycle Some projects will outlive their creators or even you; others will end sooner, replaced by something new. In my experience, most systems live for about three years — after that, they usually go into idle mode and can stay there for decades, like fading stars. three years That’s yet another reason not to build a spaceship from day one and not to over-engineer every little thing. Everything changes, ages, and eventually fades away — and that’s perfectly normal. not to build a spaceship from day one Conclusion The main things I’ve learned from this experience: Don’t overcomplicate. You’ll never account for everything anyway.Spend 80% of your attention on business needs and analysis, and 20% on architecture and code.Prepare carefully for the launch.From day one, build in metrics (collection and visibility) and controlled delivery to production. Don’t overcomplicate. You’ll never account for everything anyway. Spend 80% of your attention on business needs and analysis, and 20% on architecture and code. 80% 20% Prepare carefully for the launch. From day one, build in metrics (collection and visibility) and controlled delivery to production. controlled delivery Below are the metrics of the system as it ended up: I hope that my experience, the advice I’ve shared, and the real launch examples will be useful both for those who have already launched such systems — and for those who are about to face it for the first time. Wishing you success as you dive into new domains 🚀 Wishing you success as you dive into new domains Remember: it’s difficult at first, but soon you’ll be swimming like a fish in water — and eventually sharing your own experience with those standing at the edge, unsure of what comes next.