First In First Out (FIFO) in Message Brokers Message brokers are a crucial part of many asynchronous distributed systems. Just like blood vessels transport blood throughout the human body, message brokers transmit streams of countless messages between applications. Today, we're going to take a quick look (as always) at a very useful yet often forgotten feature of our beloved friends — message brokers — the First In First Out mechanic. The Problem Imagine we have a producer — a service that emits messages. Every message belongs to a certain group. There are a few consumers — services that process these messages. We want to secure the order of the messages related to the same group while allowing messages for different groups to be processed in a non-deterministic way. At first glance, there seem to be no issues: the message broker fans out the messages to the consumers in the same order it received them from the producer, so we can expect them to be processed sequentially, right? Unfortunately, no. For instance, if consumer A is much slower than the others, while it processes message 1, consumer B cansage 2, then receive from the broker and process message 3. As a result, the messages will be processed in the following order: 2, 3, 1 instead of 1, 2, 3. ☠️ Approach 1: Keep Only One Consumer Reduce the number of consumers to one, turn off parallel message processing to ensure instance doesn't process multiple messages at the same time, and enjoy the result (no, you won't enjoy it). I've seen it done this way a few times. Don't do this, please. Just don't, okay? ✅ Approach 2: Use The Power Of Message Broker Many messaging systems support the FIFO pattern, making it fairly easy to maintain the order of messages. Azure Service Bus In Azure Service Bus, it's called Sessions: a mechanism to ensure that messages with the same value of the session ID header will be processed by consumers in the same order as they were received by the Message Bus from publishers. (picture from https://learn.microsoft.com/en-us/azure/service-bus-messaging/message-sessions) Amazon SQS AWS SQS supports FIFO queues, which ensure the order of messages. These queues maintain the order in which messages are sent and received and also ensure that a message is delivered only once. Similar to Azure Service Bus, FIFO queues from AWS support grouping messages (messages in a group will be processed in order while different groups are processed in parallel). Apache Kafka Kafka is designed with partitioning at its core, which inherently supports message ordering within each partition. By designating a key for messages, Kafka ensures that all messages with the same key go to the same partition, thus preserving the order. Conclusion Many popular message brokers provide this powerful and often exceptionally useful feature — message ordering. It's good to know about it so that when you need to deal with message ordering or synchronization of parallel events, you have this tool in your toolbox. Thanks for reading, and see you next time ❤️ First In First Out (FIFO) in Message Brokers Message brokers are a crucial part of many asynchronous distributed systems. Just like blood vessels transport blood throughout the human body, message brokers transmit streams of countless messages between applications. Today, we're going to take a quick look (as always) at a very useful yet often forgotten feature of our beloved friends — message brokers — the First In First Out mechanic. The Problem Imagine we have a producer — a service that emits messages. Every message belongs to a certain group. There are a few consumers — services that process these messages. We want to secure the order of the messages related to the same group while allowing messages for different groups to be processed in a non-deterministic way. At first glance, there seem to be no issues: the message broker fans out the messages to the consumers in the same order it received them from the producer, so we can expect them to be processed sequentially, right? Unfortunately, no. For instance, if consumer A is much slower than the others, while it processes message 1, consumer B cansage 2, then receive from the broker and process message 3. As a result, the messages will be processed in the following order: 2, 3, 1 instead of 1, 2, 3. ☠️ Approach 1: Keep Only One Consumer Reduce the number of consumers to one, turn off parallel message processing to ensure instance doesn't process multiple messages at the same time, and enjoy the result (no, you won't enjoy it). I've seen it done this way a few times. Don't do this, please. Just don't, okay? ✅ Approach 2: Use The Power Of Message Broker Many messaging systems support the FIFO pattern, making it fairly easy to maintain the order of messages. Azure Service Bus In Azure Service Bus, it's called Sessions : a mechanism to ensure that messages with the same value of the session ID header will be processed by consumers in the same order as they were received by the Message Bus from publishers. Sessions (picture from https://learn.microsoft.com/en-us/azure/service-bus-messaging/message-sessions ) https://learn.microsoft.com/en-us/azure/service-bus-messaging/message-sessions Amazon SQS AWS SQS supports FIFO queues , which ensure the order of messages. These queues maintain the order in which messages are sent and received and also ensure that a message is delivered only once. Similar to Azure Service Bus, FIFO queues from AWS support grouping messages (messages in a group will be processed in order while different groups are processed in parallel). FIFO queues Apache Kafka Kafka is designed with partitioning at its core, which inherently supports message ordering within each partition. By designating a key for messages, Kafka ensures that all messages with the same key go to the same partition, thus preserving the order. Conclusion Many popular message brokers provide this powerful and often exceptionally useful feature — message ordering. It's good to know about it so that when you need to deal with message ordering or synchronization of parallel events, you have this tool in your toolbox. Thanks for reading, and see you next time ❤️

This story contains AI-generated text. The author has used AI either for research, to generate outlines, or write the text itself. 

This story will praise and/or roast a product, company, service, game, or anything else people like to review on the Internet.

How FIFO Mechanisms in Message Brokers Ensure Message Order

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

90% of Developers Were Outperformed by GPT-4

The Noonification: If You Cant Beat Em, Buy Em! (10/18/2023)

The Noonification: The Conversational AI Arms Race Has Begun (2/9/2023)

3 Top Resources To Learn About Apache Kafka

47 Stories To Learn About Kafka

5 Things Every Apache Kafka Dev Needs To Know: A Performance and Architectural Deep Dive

90% of Developers Were Outperformed by GPT-4

The Noonification: If You Cant Beat Em, Buy Em! (10/18/2023)

The Noonification: The Conversational AI Arms Race Has Begun (2/9/2023)

3 Top Resources To Learn About Apache Kafka

47 Stories To Learn About Kafka

5 Things Every Apache Kafka Dev Needs To Know: A Performance and Architectural Deep Dive

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps