I am a software engineer who worked on many systems ranging from small and simple to large and complex. I have had the opportunity to design, build, maintain, and support those systems. In my years of experience, I have realized that quality begins in the design phase. I attempt to summarize that experience and give you some tips, even checklists you can use in your next design task. The content is intended to be general. So, you need to adapt some of the steps to your specific use case. Opinions expressed are solely my own and do not express the views or opinions of my employer. Step 0: The problem statement All successful engineering projects begin with a well-defined problem statement. Each project is different. Often, there is a gap between what you think is needed and what is needed. Sometimes, a Product/Project Manager(PM) can articulate those requirements into a nice document. Other times, you will have to do it yourself. In any case, here are some helpful things to do. Ask lots of questions.
The only stupid questions are the ones you don't ask
At least for the important bits, rephrase in your own words and ensure you got it right.
Do a feasibility analysis
Good PMs are unsurprisingly good at weeding out the wild goose chases. You should still conduct your due diligence to complement it.
You may need to do some technical analysis in some cases. Be sure to budget for that.
Check for any security red flags.
Check for privacy concerns.
Check for data availability.
Agree on trade-offs
Document trade-offs based on feasibility, security, and privacy aspects.
Discuss and agree on the proposed trade-offs. It is a good idea to document these. Doing so will aid clarity and also help your design process. Step 1: Collecting We have a problem, a good one. You know where you want to go but don't yet know how. It's like you know you want to get to the Grand Canyon. You know you can reserve flights and rooms online. But you haven't found the cheapest prices for the experience you would love to have. There are a lot of moving parts. You may have a lot of information on your hands or barely know anything at this point. That is normal. When I am in this situation, I do this Grab a pen and paper.
Draw some rough boxes for "things" that make the system tick. Don't worry about getting all the boxes right at this point.
Connect all the "things" with lines. Now, you have a rough flow.
Iterate on the flow until I have a rough flow that solves the problem. Step 2: Categorizing Now that you have all the fuzzy boxes you need. It is time to divide and conquer. You can divide the rough boxes into 3 categories. Clear There are tried and tested solutions to these or at least have a clear path forward.
These "basics" are available to you:
Specifications
Documentation
Support plan or support contacts Frosted These are the same as the clear boxes, except some basics are missing. Fuzzy These are the interesting ones (in a good or a bad way).
Most of the basics are missing but you have somewhat of a path forward For brevity's sake, I will only talk about the clear boxes. I will go into more detail on the next two categories in the coming articles. Step 3: Prioritizing “If everything’s a priority, then nothing’s a priority.” - Frank Sonnenberg You must have heard that numerous times. Think about the things that this product needs to do. Is it resource efficiency, low latency, high availability, or all of those things? Which of those comes first, which is second, and so on. This list of priorities is crucial to how you approach designing your product. Step 4: Discovery Now that you have the rough boxes and all the little squiggly connecting lines, it is time to dive deeper. In this step, you try or test different options for each rough box. You will need several things for each of those. The following is a list of the most common ones. Specifications.


User documentation.


Contacts: If a component is from another team, you will need at least 1 point of contact. This is useful if you have questions while exploring the component.


A test rig


Chances are, whoever built the components you are using already has a test rig.


Beg or borrow so you don't have to reinvent the wheel.


If there is none, build a simple one yourself. Just don't go all in. Do just enough to test the basic functionality you need.


A list of known dependencies


It is a great idea to check for vulnerabilities.


A list of known vulnerabilities


The presence of vulnerabilities may influence your decision to use or not to use a library.


Notes, lots of them


If you are designing a large system, chances are, there are several little details. So, keep well-organized notes you can refer to later. It is okay to collect the items from the list for one box at a time. The idea is to not get overwhelmed by the breadth of the system and work in small, easy-to-handle increments. Step 4: Assembly Now that you have the rough boxes and you've explored the options. By this time, you should have a pretty good idea of what everything does. Now is the time to grab a pen and paper again. This time, you look at your initial drawing and make a more detailed drawing. This drawing will give you a good picture of how everything works. Time for another checklist List the following scenarios: Happy path
Stuff is on fire
Transient failures
Draw a sequence diagram or a flow chart for each scenario
Write a test plan
Recheck the list of dependencies and vulnerabilities
Do you need to add or remove some of them?
Is a dependency too hard to handle or isn't worth the work anymore? You may have to redo discovery for that functionality and come back here.
Put the design together in a neat document. Key takeaways The design won't be perfect in the first iteration. There is no shame in going back to it and redoing some bits. Make sure you are weighing costs, risks, and benefits wisely. Lastly, estimate and put everything into whatever work tracker you use at your company. I am a software engineer who worked on many systems ranging from small and simple to large and complex. I have had the opportunity to design, build, maintain, and support those systems. In my years of experience, I have realized that quality begins in the design phase. I attempt to summarize that experience and give you some tips, even checklists you can use in your next design task. The content is intended to be general. So, you need to adapt some of the steps to your specific use case. Opinions expressed are solely my own and do not express the views or opinions of my employer. Opinions expressed are solely my own and do not express the views or opinions of my employer. Opinions expressed are solely my own and do not express the views or opinions of my employer. Step 0: The problem statement All successful engineering projects begin with a well-defined problem statement. Each project is different. Often, there is a gap between what you think is needed and what is needed. Sometimes, a Product/Project Manager(PM) can articulate those requirements into a nice document. Other times, you will have to do it yourself. In any case, here are some helpful things to do. Ask lots of questions. The only stupid questions are the ones you don't ask At least for the important bits, rephrase in your own words and ensure you got it right. Do a feasibility analysis Good PMs are unsurprisingly good at weeding out the wild goose chases. You should still conduct your due diligence to complement it. You may need to do some technical analysis in some cases. Be sure to budget for that. Check for any security red flags. Check for privacy concerns. Check for data availability. Agree on trade-offs Document trade-offs based on feasibility, security, and privacy aspects. Discuss and agree on the proposed trade-offs. It is a good idea to document these. Doing so will aid clarity and also help your design process. Ask lots of questions. The only stupid questions are the ones you don't ask At least for the important bits, rephrase in your own words and ensure you got it right. Do a feasibility analysis Good PMs are unsurprisingly good at weeding out the wild goose chases. You should still conduct your due diligence to complement it. You may need to do some technical analysis in some cases. Be sure to budget for that. Check for any security red flags. Check for privacy concerns. Check for data availability. Agree on trade-offs Document trade-offs based on feasibility, security, and privacy aspects. Discuss and agree on the proposed trade-offs. It is a good idea to document these. Doing so will aid clarity and also help your design process. Step 1: Collecting We have a problem, a good one. You know where you want to go but don't yet know how. It's like you know you want to get to the Grand Canyon. You know you can reserve flights and rooms online. But you haven't found the cheapest prices for the experience you would love to have. There are a lot of moving parts. You may have a lot of information on your hands or barely know anything at this point. That is normal. When I am in this situation, I do this Grab a pen and paper. Draw some rough boxes for "things" that make the system tick. Don't worry about getting all the boxes right at this point. Connect all the "things" with lines. Now, you have a rough flow. Iterate on the flow until I have a rough flow that solves the problem. Grab a pen and paper. Draw some rough boxes for "things" that make the system tick. Don't worry about getting all the boxes right at this point. Connect all the "things" with lines. Now, you have a rough flow. Iterate on the flow until I have a rough flow that solves the problem. Step 2: Categorizing Now that you have all the fuzzy boxes you need. It is time to divide and conquer. You can divide the rough boxes into 3 categories. Clear Clear There are tried and tested solutions to these or at least have a clear path forward. These "basics" are available to you: Specifications Documentation Support plan or support contacts There are tried and tested solutions to these or at least have a clear path forward. These "basics" are available to you: Specifications Documentation Support plan or support contacts Frosted Frosted These are the same as the clear boxes, except some basics are missing. Fuzzy Fuzzy These are the interesting ones (in a good or a bad way). Most of the basics are missing but you have somewhat of a path forward These are the interesting ones (in a good or a bad way). Most of the basics are missing but you have somewhat of a path forward For brevity's sake, I will only talk about the clear boxes. I will go into more detail on the next two categories in the coming articles. Step 3: Prioritizing “If everything’s a priority, then nothing’s a priority.” - Frank Sonnenberg “If everything’s a priority, then nothing’s a priority.” - Frank Sonnenberg Frank Sonnenberg You must have heard that numerous times. Think about the things that this product needs to do. Is it resource efficiency, low latency, high availability, or all of those things? Which of those comes first, which is second, and so on. This list of priorities is crucial to how you approach designing your product. Step 4: Discovery Now that you have the rough boxes and all the little squiggly connecting lines, it is time to dive deeper. In this step, you try or test different options for each rough box. You will need several things for each of those. The following is a list of the most common ones. Specifications. User documentation. Contacts: If a component is from another team, you will need at least 1 point of contact. This is useful if you have questions while exploring the component. A test rig Chances are, whoever built the components you are using already has a test rig. Beg or borrow so you don't have to reinvent the wheel. If there is none, build a simple one yourself. Just don't go all in. Do just enough to test the basic functionality you need. A list of known dependencies It is a great idea to check for vulnerabilities. A list of known vulnerabilities The presence of vulnerabilities may influence your decision to use or not to use a library. Notes, lots of them If you are designing a large system, chances are, there are several little details. So, keep well-organized notes you can refer to later. Specifications. Specifications. User documentation. User documentation. Contacts: If a component is from another team, you will need at least 1 point of contact. This is useful if you have questions while exploring the component. Contacts: If a component is from another team, you will need at least 1 point of contact. This is useful if you have questions while exploring the component. A test rig A test rig Chances are, whoever built the components you are using already has a test rig. Chances are, whoever built the components you are using already has a test rig. Beg or borrow so you don't have to reinvent the wheel. Beg or borrow so you don't have to reinvent the wheel. If there is none, build a simple one yourself. Just don't go all in. Do just enough to test the basic functionality you need. If there is none, build a simple one yourself. Just don't go all in. Do just enough to test the basic functionality you need. A list of known dependencies A list of known dependencies It is a great idea to check for vulnerabilities. It is a great idea to check for vulnerabilities. A list of known vulnerabilities A list of known vulnerabilities The presence of vulnerabilities may influence your decision to use or not to use a library. The presence of vulnerabilities may influence your decision to use or not to use a library. Notes, lots of them Notes, lots of them If you are designing a large system, chances are, there are several little details. So, keep well-organized notes you can refer to later. If you are designing a large system, chances are, there are several little details. So, keep well-organized notes you can refer to later. It is okay to collect the items from the list for one box at a time. The idea is to not get overwhelmed by the breadth of the system and work in small, easy-to-handle increments. Step 4: Assembly Now that you have the rough boxes and you've explored the options. By this time, you should have a pretty good idea of what everything does. Now is the time to grab a pen and paper again. This time, you look at your initial drawing and make a more detailed drawing. This drawing will give you a good picture of how everything works. Time for another checklist List the following scenarios: List the following scenarios: Happy path Stuff is on fire Transient failures Draw a sequence diagram or a flow chart for each scenario Write a test plan Recheck the list of dependencies and vulnerabilities Do you need to add or remove some of them? Is a dependency too hard to handle or isn't worth the work anymore? You may have to redo discovery for that functionality and come back here. Put the design together in a neat document. Happy path Stuff is on fire Transient failures Draw a sequence diagram or a flow chart for each scenario Write a test plan Recheck the list of dependencies and vulnerabilities Do you need to add or remove some of them? Is a dependency too hard to handle or isn't worth the work anymore? You may have to redo discovery for that functionality and come back here. Put the design together in a neat document. Key takeaways The design won't be perfect in the first iteration. There is no shame in going back to it and redoing some bits. Make sure you are weighing costs, risks, and benefits wisely. Lastly, estimate and put everything into whatever work tracker you use at your company.

Walkthroughs, tutorials, guides, and tips. This story will teach you how to do something new or how to do something better.

Read My Stories

Too Long; Didn't Read

Make resilience your competitive advantage

The Art of System Design: Transforming Rough Ideas into Robust Solutions

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

10x Programmers: The True Way to Increase Software Engineering Productivity

10 Ways to Optimize Your Database

10 Tips for Using Diagrams to Ace The System Design Interview

111 Stories To Learn About Architecture

13 Questions and Answers for Google Cloud Reference Architectures

10x Programmers: The True Way to Increase Software Engineering Productivity

10 Ways to Optimize Your Database

10 Tips for Using Diagrams to Ace The System Design Interview

111 Stories To Learn About Architecture

13 Questions and Answers for Google Cloud Reference Architectures

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps