In today's rapidly advancing world, it is imperative to embrace the latest technologies in order to craft systems that are scalable, reliable, maintainable, and portable. However, this approach often leads to reactive designs that are not adaptable to evolving needs. System design should be driven by requirements, not by technology.  Achieving a successful system design calls for a profound understanding of both architecture and business requirements, ensuring that the resulting solution aligns seamlessly with the intended goals. For instance, when designing a system to handle 10 million daily requests with 20 ms latency, it's tempting to opt for , database sharding, and highly distributed cloud infrastructure. While these choices may work, they often overlook the importance of employing architectural design principles. Microservices architecture A prime example of this system is "Stack Overflow," which is built on a monolithic architecture. Its success stems from a profound technical understanding and the utilization of software architecture principles. The key point here is that fundamental design choices determine the system's trajectory. Architects gravitate towards the latest technologies or attempt to implement a multitude of design principles to achieve scalability. Like patterns:  DDD, EDA, Microservices, serverless, CQRS, layered architecture, and design principles: , Principle Of Least Astonishment, DRY, Explicit dependency, etc SOLID While architects aspire to implement these principles, in reality, architects are often constrained by time, resources, and urgency, and the evolving business and technology landscape may not always provide an opportunity to rebuild and implement the best architecture choices. This is why it is crucial to design systems based on fundamental design principles. Separation of Concerns This article highlights the importance of one of the core architectural principles that establish the very foundation for achieving success and facilitate the implementation of other architectural principles. The principle states that a system should be built on top of manageable components, with each component addressing a specific concern while minimizing overlap. A successful SoC implementation requires clearly defined component responsibilities, clearly bounded contexts, well-defined interfaces, loose coupling, and high cohesion. Enablers for SoC include SOLID Principles, Message Queues, Event Bridges, state management constraints, API gateway, and Streaming platforms. SoC is THE most important architectural principle as it allows your architecture to evolve over time to support both technology and business vision. Some major advantages of this architecture include: Promotes Modularity Reduce blast radius Enable Faster time to market Enhances  Maintainability, Scalability, and durability Improves Data Quality Enable evolving architecture and modernization like Cloud Migration, Containerization, Event streaming, etc. This principle is applicable to any system, whether it's legacy or modern: Monolith, Microservices, SOA, Enterprise IT, SaaS applications, traditional web applications, backend engines, or analytics platforms. Consider SoC as the foundational design pattern for any system. A system lacking SoC will have poor extensibility and hinder future architectural evolution How Twitter Monetization offering embodies this principle The architecture of is built upon several components, each with clearly defined capabilities and boundaries. It serves as a good reference for large-scale systems built using the principle of Separation of Concerns, coupled with high-performance infrastructure to support high throughput, availability, and low latency, as demonstrated by their "Manhattan" infrastructure. Twitter Some major components and Services include: Timeline Mixer Ad Mixer Onboarding Service People discovery Prediction service Recommendation and Analytics Engine Search Service Indexing and MapReduce And some technology choice that enables scalability Storage - Hadoop, Manhattan, Vertica, flockDB Cache  - Redis and Memcache Events Processing - Kafka, TimeSeries Aggregator (TSAR) For deep dive into Twitter’s infrastructure please refer to: The infrastructure behind Twitter's scale Processing-billions-of-events Manhattan So how does the principle of Separation of Concerns benefit Twitter? Application modernization: Retire Timeline Mixer with a more advanced Home Mixer service. Deliver value sooner: Twitter subscription service for content creator Let’s understand how Twitter might have delivered this service faster using the SoC architecture principle. High-level Requirements The requirement has the following broad features Capability: Offer Monetization for content creators Features: Content creators : Enable monetization, receive payments, send exclusive tweets, view exclusive tweets on user timelines Subscriber: Subscribe to a creator, Make payment, View exclusive posts on home timeline Payment gateway: Collect payment methods and process payment : The architecture below demonstrates how these components are orchestrated to build the desired capability. High-Level architecture Build & Development Based on existing Twitter architecture, following major services might have been modified Timeline Mixer Author timeline - view exclusive tweets in a separate section User home timeline - view exclusive tweets in a separate tab Publisher Service Persist and cache exclusive tweets Graph service Fetch subscription information for building follower graph Security and access control Control who can view/retweet a post Major components to build: Payment gateway: For this phase, Twitter used Stripe but maybe in the future Twitter will build its own Payment gateway service Subscription service: Enable Monetization Persist subscription information Integrations to existing services and data store How did this architecture benefit Twitter? Time to Value Build components independently without impacting existing stack Separate teams can build/test each components for faster build New component can be build on pilot infrastructure and scaled independently based on adoption Reduce blast radius By separating concerns into distinct modules or components, failures or errors within one module are contained and do not affect other modules. Isolation limits the blast radius and prevents widespread system failures. Enable evolvability The architecture allows for evolution over time to accommodate changing requirements. For example, the payment gateway can be replaced with a custom-built gateway without impacting the overall stack. Future services can be integrated or built to further enhance capabilities. this article emphasizes the significance of building software systems based on fundamental design principles, with Separation of Concerns being the most critical principle. The example of Twitter showcases how applying this principle contributes to successful system design, enabling scalability, maintainability, and adaptability to evolving business and technological landscapes. In conclusion, What's not covered in the article? Twitter Specific technology stack: Manhattan, Gizmoduck, Apache storm, Lucene, Redis, graphql, etc. Detail on various Twitter services mentioned above Architecture styles: Domain-driven design, Microservices pattern, Event-driven architecture, CQRS, etc Please feel free to reach out if you have any questions!

This story will praise and/or roast a product, company, service, game, or anything else people like to review on the Internet.

Walkthroughs, tutorials, guides, and tips. This story will teach you how to do something new or how to do something better.

The Core Principle of Separation of Concerns & Twitter's Value-Driven Approach

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

A Guide to Building an AWS Serverless API and CICD Pipeline

9 Basic (and Crucial) Tips for Microservices Developers

A Comprehensive Guide to Approaching Package Organization in Go

An Introduction to Domain-Driven Design - Part 2

The 4 Layers of Single Page Applications You Need to Know

Behavior-Driven Development From Scratch [101]

A Guide to Building an AWS Serverless API and CICD Pipeline

9 Basic (and Crucial) Tips for Microservices Developers

A Comprehensive Guide to Approaching Package Organization in Go

An Introduction to Domain-Driven Design - Part 2

The 4 Layers of Single Page Applications You Need to Know

Behavior-Driven Development From Scratch [101]

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps