This is a continuation of a series of articles in which I briefly cover the main points of a specific topic in system architecture design. The previous article can be read and the full guide you can find on my . here github Message queues are a form of asynchronous service-to-service communication. They are important in enhancing a , reliability, and maintainability. system's scalability The list of key features: : Allows different parts of a system to communicate without needing to respond immediately, leading to more efficient use of resources. Asynchronous Communication : Enables services to operate independently, reducing the system's complexity and enhancing maintainability and scalability. Decoupling of Services : Distributes messages evenly across different services or workers, helping to manage workload and improve system performance. Load Balancing : Some message queues can ensure that messages are processed in the order they are sent, which is crucial for specific applications. Order Preservation : Facilitates easy scaling of applications by adding more consumers or resources to handle increased message flow. Scalability : Controls the rate at which messages are processed, which is important for managing resources and preventing system overloads. Rate Limiting and Throttling : Message queues often include a fan-out mechanism, which allows a single message to be delivered to multiple consumers or services simultaneously. Fan-out Capability : Offers the ability to store messages on disk or in memory until they are successfully processed, ensuring data is not lost in case of system failures. Data Persistence Allows messages to be routed or filtered based on specific criteria or content, enabling more targeted and efficient processing. Message Filtering and Routing: Components In the context of message queues, the concepts of producers, consumers, and messages form the core of how these systems operate. is an application or service responsible for creating and sending messages to the message queue. It does not need to be aware of who will process the message or when it will be processed. Producer is an application or service that retrieves and processes messages from the queue. It acts on the data sent by producers. Consumer are the data packets sent from producers to consumers. They can vary in size and format, ranging from simple text strings to complex data structures like JSON or XML. Messages is a middleware tool that facilitates communication between different applications or services by receiving messages from a sender and routing them to the appropriate receiver. It typically provides features like message queuing, routing, transformation, and delivery assurance. Message Broker Messaging Models Globally, there are two types of messaging: Point-to-Point and Publish-Subscribe. Point-to-Point Messages sent by a producer are placed in a queue and are consumed by a single consumer. It ensures that each message is processed only once by one receiver. Publish-Subscribe Messages are published to a specific topic rather than a queue. Multiple consumers can subscribe to a topic and receive messages broadcast to that topic. However, some messaging protocols and the brokers that support them use an additional Exchange component for routing. In that case, messages are published to an exchange in the broker first. The Exchange, acting as the routing agent, forwards these messages to the appropriate queue using its routing rules. The following exchange operating modes are distinguished: Direct exchange The message is routed to the queues whose binding key matches the message's routing key. Topic exchange The topic exchange will perform a wildcard match between the routing key and the routing pattern specified in the binding to publish messages to the queue. Fanout exchange A message sent to a fan-out exchange is copied and forwarded to all the queues bound to the exchange. Header exchange Header exchanges will use the message header attributes for routing. Dead letter Dead letter queues collect messages that couldn’t be processed successfully for various reasons like processing errors, message expiration, or delivery issues. Protocols Message brokers are responsible for delivering messages from producers to consumers. They use specific protocols that define the rules and formats for messaging. The most popular protocols in this domain are: AMQP (Advanced Message Queuing Protocol) A binary protocol designed for message-oriented middleware with robustness, security, and interoperability. Ideal for complex and reliable enterprise messaging systems. Enterprise applications, financial systems, and business processes Use Cases: point-to-point, publish-subscribe Messaging Model: , SASL, PLAIN Security: TLS/SSL Uses exchange and queue-based addressing with routing capabilities Addressing: Broker-based Architecture: MQTT (Message Queuing Telemetry Transport) A lightweight, publish-subscribe network protocol optimized for high-latency or unreliable networks, ideal for IoT scenarios. IoT devices, home automation, mobile messaging applications Use Cases: publish-subscribe Messaging Model: TLS/SSL, SASL, PLAIN Security: It uses topic-based addressing where messages are published to topics Addressing: Broker-based Architecture: JMS (Java Message Service) A Java-based messaging standard offers interfaces for point-to-point and publish-subscribe messaging patterns in Java applications. Enterprise Java applications, integration of multiple Java-based systems Use Cases: point-to-point, publish-subscribe Messaging Model: Relies on the underlying Java EE security model Security: Uses JNDI for locating queues and topics Addressing: It is often implemented on top of enterprise service buses or application servers. Architecture: ) STOMP (Simple Text Oriented Messaging Protocol A simple, text-based protocol that is easy to implement, suitable for scenarios where advanced messaging features are not a priority. Rapid development environments and simple messaging applications Use Cases: point-to-point, publish-subscribe Messaging Model: PLAIN; relies on the underlying transport protocol for encryption Security: Frame-based with headers for destination, content type, etc. Addressing: Broker-based Architecture: Kafka Protocol Associated with Apache Kafka, a distributed streaming platform capable of handling high-throughput data streams. Real-time analytics, data pipelines, stream processing applications Use Cases: publish-subscribe Messaging Model: SSL/TLS, SASL Security: Topic-based with partitioning for scalability. Addressing: Distributed system architecture with brokers and coordination. Architecture: ZMTP (ZeroMQ Message Transport Protocol) The underlying protocol for ZeroMQ is a high-performance asynchronous messaging library for building scalable, distributed applications. High-throughput, low-latency applications, microservices architecture. Use Cases: request-reply, publish-subscribe, pipeline, exclusive pair, etc. Messaging Model: PLAIN, CurveZMQ, and ZAP Security: Flexible addressing using sockets Addressing: Library-based, enabling a brokerless design or various brokered configurations Architecture: Brokers ActiveMQ RabbitMQ Kafka ZeroMQ Written in Java Erlang Scala C++ Cross-platform yes yes yes yes Opensource yes yes yes yes Multiple languages yes yes yes yes Protocols AMQP, AUTO, MQTT, OpenWire, REST,  RSS and Atom, Stomp, WSIF,  WS Notification, XMPP, WebSocket AMQP, STOMP, MQTT, HTTP Binary over TCP TCP, UDP, inproc, PGM, IPC, TIPC, NORM, SOCKS5 QoS at-least-once at-most-once at-least-once at-most-once at-least-once at-most-once exactly-once at-least-once at-most-once Message patterns Queue, Pub-Sub Queue, Pub-Sub, RPC Pub-Sub Request-Reply, Pub-Sub, Push-Pull, Dealer and Router,  Pair, Exclusive Pair, etc Persistence Disk, DB Mem, Disk Disk - ActiveMQ Apache is an open-source, multi-protocol, Java-based message broker designed by Apache. It's known for its robustness and flexibility, supporting various messaging protocols and clients, making it a versatile choice for integrating disparate systems. ActiveMQ Architecture Features: : ActiveMQ supports a wide range of messaging protocols, including AMQP, MQTT, OpenWire, STOMP, and , making it highly adaptable to different client requirements. Multi-Protocol Support JMS (Java Message Service) : As a JMS provider, ActiveMQ complies with the JMS API, which allows loose coupling, asynchronous communication, and reliability for Java applications. JMS Provider : ActiveMQ uses a broker architecture, where a central broker handles message routing, delivery, and queuing. Broker-Based Architecture : Offers options for message persistence, including database storage (for durability) and file-system storage, supporting both high-performance and high-durability scenarios. Pluggable Persistence and Storage : Supports clustering and load balancing, enabling high availability and scalability. Clustering and Load Balancing : Provides different options for message acknowledgments, enhancing message reliability. Client-Side Acknowledgements Scenarios for Use: : Ideal for integrating different systems within an enterprise, mainly where Java-based or multiple protocols are used. Enterprise Integration : Useful in scenarios where decoupling system components is essential, like in microservices architecture. Asynchronous Communication : Facilitates message communication in distributed systems, ensuring data consistency and reliability. Distributed Computing : Can be used in IoT setups, especially where MQTT is preferred. IoT Communication Pros: : One of the key strengths of ActiveMQ is its support for multiple protocols, offering flexibility in various environments. Versatility in Protocol Support : Provides reliable message delivery and durable storage. Reliability and Durability Clustering and High Availability: Supports clustering for load balancing and high availability. Comprehensive support for the JMS API makes it a strong candidate for Java-based systems. JMS Support: Cons: : While robust, ActiveMQ may not match the performance of some newer message brokers, especially in scenarios with extremely high throughput requirements. Performance : Can be difficult to configure and manage, especially in clustered setups. Complex Configuration : Might require significant resources, particularly under heavy load, for optimal performance. Resource Usage : While it offers management tools, they might be less comprehensive and user-friendly than those of some newer brokers. Management and Monitoring RabbitMQ is an open-source message broker software known as a message-oriented middleware. It's written in Erlang and is built on the Open Telecom Platform framework for clustering and failover. RabbitMQ is widely used for handling asynchronous processing, enabling communication between distributed systems through various messaging protocols, primarily AMQP (Advanced Message Queuing Protocol). RabbitMQ Architecture Features: : While RabbitMQ is primarily known for AMQP, it also supports MQTT, STOMP, and other protocols through plugins. Support for Multiple Messaging Protocols : It follows the standard producer-consumer pattern, where producers send messages and consumers receive them, with RabbitMQ acting as the broker. Producer-Consumer Model : Messages in RabbitMQ are published to exchanges, which are then routed to bound queues based on routing keys and patterns. Exchange-Queue Binding : Supports durable (persistent on disk) and transient (in-memory) messages. Durable and Transient Messaging : RabbitMQ can be clustered for high availability and scalability, distributing queues among multiple nodes. Clustering and High Availability Offers several exchange types (like direct, topic, fanout, and headers) for diverse routing logic. Flexible Routing: : Supports pluggable authentication modules, including LDAP. Pluggable Authentication and Authorization Scenarios for Use: : Ideal for decoupling heavy processing tasks in web applications, ensuring responsive user interfaces. Asynchronous Processing : Used in a microservices architecture for communicating between services. Inter-Service Communication : Well-suited for handling background tasks like sending emails or processing images. Task Queues : Facilitates message communication in distributed systems, maintaining consistency and reliability. Distributed Systems Pros: : RabbitMQ is known for its reliability and ability to ensure message delivery. Reliability : Its routing capabilities are more advanced than those of many message brokers. Flexible Routing Capabilities : Supports scalable clustering, which is crucial for large-scale applications. Scalability and High Availability : The ability to support multiple messaging protocols increases adaptability. Wide Protocol Support : Comes with a user-friendly management interface, which simplifies monitoring and managing message flows. Management Interface Cons: : Understanding RabbitMQ's routing and setup can be complex for beginners. Learning Curve : It can be memory-intensive, especially under heavy load, requiring proper monitoring and tuning. Memory Usage : Being built on Erlang, it introduces an additional technology stack that teams might need to familiarize themselves with. Erlang Dependency : While generally performant, performance tuning might be necessary under extremely high loads or in complex routing scenarios. Performance Under High Load Kafka is an open-source stream-processing software platform developed by LinkedIn and later donated to the Apache Software Foundation. It's designed to handle high volumes of data and enable real-time data processing. Kafka is a distributed, partitioned, and replicated commit log service. Apache Kafka Architecture Features: : Kafka operates on a producer-consumer model. Producers publish messages to Kafka topics, and consumers subscribe to those topics to read the messages. Producer-Consumer Model : Data in Kafka is categorized into topics. Each topic can be split into partitions, allowing for parallel data processing. Partitions also enable Kafka to scale horizontally. Topics and Partitions : Kafka runs as a cluster on one or more servers, and the Kafka cluster stores streams of records in categories called topics. Distributed System : Kafka replicates data across multiple nodes (brokers) to ensure fault tolerance. If a node fails, data can be retrieved from other nodes. Replication : Kafka uses ZooKeeper for cluster management and coordination, ensuring consistency across the cluster. Zookeeper Coordination : Kafka stores all data as a sequence of records (or a commit log), providing durable message storage. Commit Log Storage Scenarios for Use: : Ideal for real-time analytics and monitoring systems where quick data processing is crucial. Real-Time Data Processing : Suitable for recording the sequence of events in applications. Event Sourcing : Effective for collecting and processing logs from multiple services. Log Aggregation : Can be used for complex stream processing tasks like aggregating data streams or real-time filtering. Stream Processing : Often used with big data tools for data processing and analysis. Integration with Big Data Technologies Pros: : Can handle high volumes of data and many simultaneous transactions. High Throughput : Easily scalable both horizontally and vertically. Scalability : Provides durable storage of messages. Durability and Reliability : High fault tolerance due to data replication. Fault Tolerance : Can be used for a wide range of use cases, from messaging systems to activity tracking and log aggregation. Flexibility Cons: : Setup and management can be complex, especially for large clusters. Complexity : Can be resource-intensive, requiring a good amount of memory and CPU. Resource Intensive Relies on ZooKeeper for coordination, adding an extra component to manage. Dependency on ZooKeeper: : While fast, it may not be suitable for use cases requiring extremely low latency. Latency ZeroMQ (ØMQ, 0MQ, or ZMQ) is a high-performance asynchronous messaging library for distributed or concurrent applications. It's not a message broker but a library that abstracts socket communication into a message-oriented middleware, making it easier to implement complex communication patterns in a scalable way. Developed in C++, ZeroMQ can be used in various programming languages through bindings. ZeroMQ Architecture Features: : ZeroMQ uses sockets that abstract away the complexity of low-level network programming. These sockets can be used in patterns like publish-subscribe, request-reply, and fan-out. Socket-Based Communication : Unlike traditional message brokers, ZeroMQ is brokerless, allowing direct communication between endpoints without requiring a central message broker. Brokerless Design : Provides a way to manage multiple threads with socket-based communication, facilitating scalable I/O bound operations. Scalable Multithreading : Supports non-blocking, asynchronous I/O operations, which is critical for building responsive, high-performance applications. Asynchronous I/O : Offers bindings for multiple programming languages, making it accessible from different technology stacks. Language Agnostic Scenarios for Use: : Ideal for inter-service communication in a microservices architecture. Microservices : Used in parallel processing systems where performance is critical. High-Performance Computing : Suitable for scenarios requiring complex, distributed messaging patterns without the overhead of a broker. Distributed Systems : Effective in systems needing low-latency, real-time data exchange. Real-time Communication Pros: : ZeroMQ is designed for high throughput and low latency, making it suitable for performance-critical applications. High Performance : Supports various messaging patterns, providing flexibility for different communication scenarios. Flexibility in Messaging Patterns : The brokerless architecture simplifies deployment and reduces system complexity. Reduced Complexity : Facilitates easy scaling of applications with its efficient handling of multiple connections. Scalability : Less resource-intensive compared to traditional messaging brokers. Lightweight Cons: : Lacks built-in message durability or persistence support, which must be handled externally. No Built-in Durability or Message Persistence : Developers need to manage connections, retries, and error handling, which can add complexity to application logic. Requires Explicit Management of Connections : Understanding and effectively using ZeroMQ's patterns can require a steep learning curve. Learning Curve : While this can be an advantage, it also means needing more centralized management, monitoring, and control over the messaging system. Lack of a Central Broker

Too Long; Didn't Read

Making security fun one episode at a time

The System Design Cheat Sheet: Message Queues - ActiveMQ, RabbitMQ, Kafka, ZeroMQ

Too Long; Didn't Read

Aleksandr Gavrilenko

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

The System Design Cheat Sheet: Message Queues - ActiveMQ, RabbitMQ, Kafka, ZeroMQ

Too Long; Didn't Read

Aleksandr Gavrilenko

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

RELATED STORIES