30 Shared Principles for discussing Software Architectures

Imagine a fly-by architecture review. An architect walks in, looks over, glosses over, seeing though his binoculars. He provides comments that are often too generic or out of context. Comments are often met with deafening silence or winding arguments. They rarely help anybody if ever. Every programmer dreads it; every architect dreads it too.

It is said that a software architect should think like a gardener rather than a commander. The former shapes, curates, and removes weed while the latter defines and dictates. An architect should curate rather than dictate, shape rather than define, and incite discussion than label. But, how to make it work?

At WSO2, I have done architecture reviews for more than eight years. WSO2 has had an extensive portfolio of products, including WSO2 ESB, WSO2 API Manager, and WSO2 SP, that are well known. In the last eight years, we have debated, designed, improved and redesigned many products and features.

A crucial part of our design process is that architecture is not done by architects. We do not have a set of architects who dictate architecture blueprints, which are implement by others. In contrast, the design is done by the team who will write the code. Architects fix, complain, curate, and improve the design. We have an architecture team, but they are guides and gatekeepers, not the dictators.

In the following talk, Gregor Hohpe captures this idea beautifully.

https://www.youtube.com/watch?v=qVyt3qQ_7TA

In the short term, dictating an architecture is faster and may even be cheaper. It is true. However, in the long run, we build better teams by letting them think for themselves, letting them evolve the architecture, and sometimes letting them make their own mistakes. When we focus on the team, they get better with time. Execution gets easier as architecture is the team’s ideas in the first place.

Architecture reviews, however, have their pitfalls also. Paul (@pzfreo) used to call this drive by architecture where Architects walk in, listen, give comments, and move on. As an architect, you can easily inspect, complain, and take the design apart. However, if you are not careful, you might leave the team bewildered and not sure what is the right thing to do.

We address this problem by having a list of shared architecture principals. Those are principles that everyone agrees. Architects give feedback by saying this is not good due to principal X. Principles guides us and keeps our discussions rooted. They also avoid philosophical battles that go on forever. Finally, if the designer has never heard of the Principle, it is easy for him to learn.

Following are some of those principals. Some are well-known while some others we picked up the hard way.

Basic

Principle 1: KISS (Keep it simple) and “Everything should be made as simple as possible but not simpler.” The idea is to use the simplest solution that does the job.

Principle 2: YAGNI (You aren’t gonna need it) — do not build it until it is needed.

Principle 3: Crawl, walk, run. In other words, make it work, make it better, make it great. Do Iterative Development — Do agile, iterative development. For every feature, create milestones ( 2 weeks maximum) and Iterate.

Principle 4: The only way to build stable, high-quality products is via automation tests. Be creative with automated tests; everything can be automated!! Think about this when you design.

Principle 5: Always think Return for Investment ( ROI) and invest most attention on making the most impact.

Principle 6: Know your users and balance your efforts accordingly. For most products, there will be thousands of end users, 20 developers that extend the product, and 100 DevOp personals who set it up. For example, do not spend months building a UI forDevOp who is unlikely use it anyway ( they like scripts! instead). This is a special case of Principle 5.

Principle 7: Design and test Features as independently as possible. Think about this when you design. It will save lot of trouble in the long run as otherwise you can’t test the system until everything is built. Also with this principle, your releases will be smoother.

Principle 8: Lookout for Google Envy. We all like shiny designs. It is easy to bring features and solutions into your architecture that you will never need.

Choosing Features

Principle 9: It is impossible to think through how users will use our product fully. So embrace MVP ( Minimal Viable Product). The idea is to figure out few use cases, only do features that support those, ship the product, and shape the future product based on feedback and experience.

Principle 10: Do fewer Features as possible, when in doubt leave it out. Many of features are never used, maybe you should leave an extension point instead.

Principle 11: Wait for someone to ask for it (e.g., features that are not deal breakers, wait until it is needed)

Principle 12: Have the courage to fight the customer if features he asks for messes up the big picture. Find out the bigger picture, try to find another way to handle the problem. Remember Fords’ quote “If I had asked people what they wanted, they would have said faster horses.” Also, remember you are the expert. You are supposed to lead. It is the leader’s job to do what is right, not what is popular. Users will thank you later.

Server Design and Concurrency

Principle 13: Know how a server works from Hardware, operating system up to your programming language. Optimizing the Number of IO calls is the first guiding lights towards the best architecture.

Principle 14: Know Amdhal’s law about synchronization. Mutable data shared between threads slows your program. Use concurrent data structures if you can, and use synchronization only if you must. Try to hold the locks as little time as possible. If you plan to block while holding a lock, make sure you know what you are doing. If it can break, it will.

Principle 15: If your design is a none-blocking, event-driven architecture based, never block threads or do IO from those threads. If you do, the system will be as slow as a mule.

Distributed Systems

Principle 16: Stateless systems are scalable and straightforward. Know and use Shared Nothing Architecture whenever possible.

Principle 17: Exactly once message delivery irrespective of failures is hard unless you control code in both client and server. Try to design your systems to need less (e.g. use Principle 18). Knows that most systems that promise exactly-once-delivery cuts a corner somewhere.

Principle 18: Implement operations as idempotent operations whenever possible. Then it is easy to recover, and you can live with at least once delivery.

Principle 19: Know the CAP Theorem. Scaling transactions is hard. Use compensation when possible. RDBMS based transactions do not scale.

Principle 20: Distributed Consensus does not scale, nor do group communication, nor cluster-wide reliable messaging. Maximum node limit for either is about eight nodes in a good day.

Principle 21: You can never hide latency and failures in a distributed system. (see Fallacies of Distributed Computing Explained).

User Experience

Principle 22: Know your users and know their goals: is he a novice, expert, or casual user? How much does he know computer science? Geeks love extension points, developers like samples and scripts, normal people like UIs.

Principle 23: Best products do not need a manual. Its use is self-evident.

Principle 24: When you cannot decide between two choices, do not pass on the problem by making it a configuration option. You are making the life hard for users and solution Architects. They know even less about how the system works than you, how can they decide? The best option is finding a choice that works every time; the next best is automatically making the choice, the third best is adding a configuration parameter and setting a reasonable default.

Principle 25: Always have sensible defaults for configurations.

Principle 26: Poorly designed configurations can create a lot of confusion. Always document few example values for the configuration.

Principle 27: Ask configuration values in terms of things the user can answer off his head without making calculations to set the value (e.g., do not ask for the number of max cache entries, instead ask for maximum memory that should be used for cache)

Principle 28: Throw an error if you see an unknown configuration. Never ignore it silently. Silent configuration errors are source of many lost hours while debugging.

Hard Problems

Principle 29: Dreaming up new languages is easy, but getting them right is very hard. Try not to do it unless team can spend at least ten person-years on it. If you are still not sure, read Five Questions about Language Design.

Principle 30: Composable drag and drop UIs are hard, do not start one unless the team ready to invest ten person-years into it.

Finally, let me talk about something that I have changed my mind about over time. In an ideal world, a platform must be composed of orthogonal components — components where each handles one aspect (e.g., security, messaging, registry, mediation, analytics). A system built with such features would be optimal.

Unfortunately, it is hard to get to that state. It is even hard to stay there. It could be a mistake to enforce this rigidly, especially at the initial state of new features where simple features can cascade into big changes because we try to make everything orthogonal. Sometimes we find that the feature we added was not useful after all and then all the additional work is spent for nothing. Finally, if this lead to negotiations between multiple teams, the feature might never get done.

With hindsight, now I am willing to live with duplication when trying to remove it lead to significant complexity. The cure can be worse than the disease.

Conclusion

As the architect, one should think like a Gardner, who shape, curate, and remove weed rather than define and build. You should curate rather than dictate, shape rather than define, and incite discussion than label.Although it might be cheaper and easier in the short term to dictate the architecture, in the long term, guiding and letting the team finding their way pays dividends.

If you are not careful, it is easier to do fly by architecture, where the designer only told his architecture is wrong, but not why it is wrong. One way to avoid this is to have a set of principles that are generally accepted, which become the anchor for discussion as well as learning path for budding architects.

We discussed some principles that helped me handle some of the challenges. Following is a summary.

If you can think of any others, I will be thrilled to hear about them via comments.

Hope this was useful. If you enjoyed this post you might also like Mastering the 4 Balancing Acts in Microservices Architecture and Concurrency ideologies of Java, C#, C, C++, Go, and Rust.