Enterprise messaging and event streaming are two technologies you can use to implement asynchronous, loosely coupled, and highly scalable applications. Today, many people use them interchangeably, not knowing that are differences exist. When you look at their inner architectures to understand why they exist and what problems they are trying to solve, you’ll realise that these two are complementary technologies rather than competing technologies. In this article, we are going to compare and contrast these technologies across several dimensions. Towards the end, you can see a summary of use cases that each technology will be ideal. Example — A thermostat application For a better understanding, consider the following thermostat application. A temperature sensor publishes readings to a messaging system every 10 seconds. A thermostat application consumes these readings from the messaging system and decides whether to turn on/off a heater. A typical message published to the messaging system looks like the following. It has the sensor ID, room where the sensor is located, timestamp, and the reading value in Celcius. Dimension 1 — Message processing style The messaging approach One way is to look at each message and make decisions independently. For example, if the reading is less than 20, the thermostat turns on the heater. It repeats the same logic the next time a message comes. Even though this is unnecessary, that is how the messaging works. In messaging, each message is discrete. A consumer can understand each message in isolation, and they can directly act upon them The streaming approach We can also aggregate readings over a time window to calculate the average value of reading. Then make a decision based on that. For example, the thermostat can calculate the average reading over the last minute and decides whether to turn on the heater. The average seems a realistic measure here. Streaming is a good fit here because you are not making decisions by looking at discrete messages. You try to discover patterns by looking at an incoming stream of messages (an event is a proper term). In streaming, an individual message doesn’t make much sense. You need to aggregate and perform certain operations on them to gain more insights Messages can be aggregated over a time window or based on the count. Then perform aggregate operations(sum, average, mean, count), filtering, and transformations to uncover patterns. Dimension 2 — Message consumption style Message consumption is a destructive operation in messaging systems. Messages are stored in the messaging system until they are consumed. Then get deleted to make way for new messages. In contrast, consumers never delete messages in streaming systems. The messaging system removes messages independently from the consumers, irrespective of their consumption status. Dimension 3 — Access to message history Consumption style leads to another difference in both technologies. In messaging systems, messages are deleted after they are consumed. Therefore, newly joined consumers can not see the older messages. Some pub/sub systems can keep messages for an extended period. But that is not the point here. In streaming systems, new and old consumers have access to a complete history of messages. Newly joined consumers can with the older messages as they move forwards and backward to read messages. catch up That will add benefits from many perspectives. For example, you can debug a consumer by replaying the messages from the beginning that it had consumed. Apart from that, when a new version of a consumer deploys, it can quickly synchronise to the latest state by consuming the stream from the beginning. In our example, a new consumer can be introduced to analyse past readings, and visualise the temperature variance. When it deploys, it will consume messages from day one to produce a dashboard. Dimension 4 — Fine-grained subscription to messages Messaging systems provide a fine-grained subscription to messages with subscription filters. With hierarchical topics, consumers can subscribe to messages that they are only interested in. The messaging system performs event filtering there. Conversely, streaming consumers will receive all the events in a partition. The consumer’s logic has to decide which events to process and which events to discard. In our example, the thermostat in the living room can tune-in to receive temperature readings that only relate to the living room. It is made possible by subscribing to the topic to filter out any unrelated messages. temperature/rooms/living Dimension 5 — Message delivery guarantees Message delivery guarantee is another dimension to compare these two technologies. Both have been designed and built to serve different levels of delivery guarantees. Different levels of delivery guarantees are listed below. At least once delivery Messages are not lost in the event of a failure, but any message may be duplicated. Applications need to be designed to ignore or handle duplicate messages. Exactly once delivery Messages are not lost in the event of a failure and each message can be processed individually to prevent possible duplication. Applications reliably know that each message will be processed only once, almost independently of application logic. Transactionally coordinated delivery An application can coordinate the processing of a message with a specific update to an external resource, such as a database. Applications can update multiple resources without fear of them ending up out of step. Based on each other’s design criteria, we can come to following conclusion. systems are good at handling and delivery use cases while is good at handling scenarios. Messaging exactly once transactionally coordinated streaming at least once delivery Conclusion Messaging and streaming are complementary technologies rather than competing technologies. They serve different purposes in the messaging industry. Deciding which technology to pick should be based on the use case. I hope you can use the above dimensions as guidelines. The following figure provides a summary of the discussion we have had so far. When you are looking at building a messaging solution, you can consider a general-purpose message broker like or . and are good candidates when developing a distributed, durable, and highly-scalable event streaming solution. Apache ActiveMQ RabbitMQ Apache Kafka Amazon Kinesis A visual version of this comparison can be found below. Previously published at https://medium.com/event-driven-utopia/comparing-enterprise-messaging-and-event-streaming-e714f7b5fc40

Amazon

Apache

Making Sense of Unbounded Data & Real-Time Processing Systems

Choosing Between Enterprise Messaging and Event Streaming For Your Architecture

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

API Development in the Time of COVID-19

5 Common Amazon Kinesis Issues

Amazon Kinesis: The AWS Data Streaming Solution

AWS Decoupling: The Big Comparison

AWS Lambda with Kinesis Trigger: 6 Pitfalls and How to Fix Them

Using the Binary Search Algorithm With Git and AWS Kinesis

API Development in the Time of COVID-19

5 Common Amazon Kinesis Issues

Amazon Kinesis: The AWS Data Streaming Solution

AWS Decoupling: The Big Comparison

AWS Lambda with Kinesis Trigger: 6 Pitfalls and How to Fix Them

Using the Binary Search Algorithm With Git and AWS Kinesis

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps