Enterprise messaging and event streaming are two technologies you can use to implement asynchronous, loosely coupled, and highly scalable applications. Today, many people use them interchangeably, not knowing that are differences exist.
When you look at their inner architectures to understand why they exist and what problems they are trying to solve, you’ll realise that these two are complementary technologies rather than competing technologies.
In this article, we are going to compare and contrast these technologies across several dimensions. Towards the end, you can see a summary of use cases that each technology will be ideal.
Example — A thermostat application
For a better understanding, consider the following thermostat application.
A temperature sensor publishes readings to a messaging system every 10 seconds. A thermostat application consumes these readings from the messaging system and decides whether to turn on/off a heater.
A typical message published to the messaging system looks like the following. It has the sensor ID, room where the sensor is located, timestamp, and the reading value in Celcius.
Dimension 1 — Message processing style
The messaging approach
One way is to look at each message and make decisions independently.
For example, if the reading is less than 20, the thermostat turns on the heater. It repeats the same logic the next time a message comes. Even though this is unnecessary, that is how the messaging works.
In messaging, each message is discrete. A consumer can understand each message in isolation, and they can directly act upon them
The streaming approach
We can also aggregate readings over a time window to calculate the average value of reading. Then make a decision based on that.
For example, the thermostat can calculate the average reading over the last minute and decides whether to turn on the heater. The average seems a realistic measure here.
Streaming is a good fit here because you are not making decisions by looking at discrete messages. You try to discover patterns by looking at an incoming stream of messages (an event is a proper term).
In streaming, an individual message doesn’t make much sense. You need to aggregate and perform certain operations on them to gain more insights
Messages can be aggregated over a time window or based on the count. Then perform aggregate operations(sum, average, mean, count), filtering, and transformations to uncover patterns.
Dimension 2 — Message consumption style
Message consumption is a destructive operation in messaging systems. Messages are stored in the messaging system until they are consumed. Then get deleted to make way for new messages.
In contrast, consumers never delete messages in streaming systems. The messaging system removes messages independently from the consumers, irrespective of their consumption status.
Dimension 3 — Access to message history
Consumption style leads to another difference in both technologies.
In messaging systems, messages are deleted after they are consumed. Therefore, newly joined consumers can not see the older messages. Some pub/sub systems can keep messages for an extended period. But that is not the point here.
In streaming systems, new and old consumers have access to a complete history of messages. Newly joined consumers can catch up with the older messages as they move forwards and backward to read messages.
That will add benefits from many perspectives. For example, you can debug a consumer by replaying the messages from the beginning that it had consumed. Apart from that, when a new version of a consumer deploys, it can quickly synchronise to the latest state by consuming the stream from the beginning.
In our example, a new consumer can be introduced to analyse past readings, and visualise the temperature variance. When it deploys, it will consume messages from day one to produce a dashboard.
Dimension 4 — Fine-grained subscription to messages
Messaging systems provide a fine-grained subscription to messages with subscription filters. With hierarchical topics, consumers can subscribe to messages that they are only interested in. The messaging system performs event filtering there.
Conversely, streaming consumers will receive all the events in a partition.
The consumer’s logic has to decide which events to process and which events to discard.
In our example, the thermostat in the living room can tune-in to receive temperature readings that only relate to the living room. It is made possible by subscribing to the topic temperature/rooms/living to filter out any unrelated messages.
Dimension 5 — Message delivery guarantees
Message delivery guarantee is another dimension to compare these two technologies. Both have been designed and built to serve different levels of delivery guarantees.
Different levels of delivery guarantees are listed below.
At least once delivery
Messages are not lost in the event of a failure, but any message may be duplicated. Applications need to be designed to ignore or handle duplicate messages.
Exactly once delivery
Messages are not lost in the event of a failure and each message can be processed individually to prevent possible duplication. Applications reliably know that each message will be processed only once, almost independently of application logic.
Transactionally coordinated delivery
An application can coordinate the processing of a message with a specific update to an external resource, such as a database. Applications can update multiple resources without fear of them ending up out of step.
Based on each other’s design criteria, we can come to following conclusion.
Messaging systems are good at handling exactly once and transactionally coordinated delivery use cases while streaming is good at handling at least once delivery scenarios.
Conclusion
Messaging and streaming are complementary technologies rather than competing technologies. They serve different purposes in the messaging industry. Deciding which technology to pick should be based on the use case. I hope you can use the above dimensions as guidelines.
The following figure provides a summary of the discussion we have had so far. When you are looking at building a messaging solution, you can consider a general-purpose message broker like Apache ActiveMQ or RabbitMQ. Apache Kafka and Amazon Kinesis are good candidates when developing a distributed, durable, and highly-scalable event streaming solution.
A visual version of this comparison can be found below.