Messaging System — Hands On!

So, I just learned about microservices. Here what I got.

One of the main concept of microservices is separating each service. To separate the services, we can apply Domain Driven Development concept, it means we need to separate the service by its domain. After separating the service by its domain, know we should make a communication mechanism between each service. So now I want to implement it.

Previously I had created a monolith system. The main purpose of the system is user can upload feed and if there are some hashtags found on the feed’s caption, they should be stored to a database with the feed’s ID too. Based on the microservices concept, I should separate services on my system. The services that I can create is services to handle the feed and service to handle the hashtag. So, here problem statement

User can create new feed through Feed Service, the feed contain location and caption. The Hashtag API should handle hashtag from the feed’s caption. Each service may written using different language.

Because I want to keep it simple, I decided to build the Feed Service using Lumen and the Hashtag Service using Golang. So the system will look like this

First Design

Until this step, everything looks okay. When a user creates a new feed, the Feed Service will handle it. Here is how the JSON data look like when user create a new feed

{"location": "Sitoluama","caption" : "Today is a beautiful day #sunset #beach #love"}

After the new feed stored, feed_id and feed_caption will send to the Hashtag Service to get all hashtag from feed’s caption and store them to Hashtag DB. The Feed Service will send the feed data as JSON to Hashtag Service by REST API call. The payload will be like this

{"feed_id": 23,"feed_caption": "Today is a beautiful day #sunset #beach #love"}

After Hashtag Service received the request, it will filter all hashtag from the caption. The extract hashtag function is simple, just iterate each character in the caption and store them to an array. Each hashtag from the array will be paired with the feed_id and store them to the Hashtag DB. So, here the extracted hashtags

[{"feed_id" : 23,"name" : "#sunset"},{"feed_id" : 23,"name" : "#beach"},{"feed_id" : 23,"name" : "#love"},]

Until this step, everything just fine and technically it works. But now I added the case.

One advantage of using microservices architecture is service reusability. One service can be used by multiple services. As we scale our system, now our users can create comment and event in our application. So we need to create Comment Service and Event Service. So, we can reuse the Hashtag Service to handle hashtag from Comment Service and Event Service

From the problem statement above, we can redesign our system. Here it will look like.

Second Design

Technically it still works. But how if there is a lot of upcoming requests from Feed Service, Comment Service, and Event Service to the Hashtag Service? How the single Hashtag Service handle all the requests? Remember, as can as possible we don’t want to miss even a single hashtag. So we need to make sure every hashtag should be processed.

The naive solution is to pair Hashtag Service to each other services. If there are three different services use the Hashtag Service, we need to create the Hashtag Service to three instances.

Naive Solution

Technically it will work. But this design is breaking the reusable concept. From the design, we know that we not reuse the Hashtag Service but creating new. So what other solution that can help us?

Messaging System, what are you?

Here I don’t explain what messaging system is, but you can find it here, here, and here. Some of the popular messaging systems are Apache Kafka, Google Pub/Sub, and RabbitMQ.

After reading what is the messaging system, maybe you also read about the publish-subscribe pattern. In a simple definition, a publish-subscribe pattern is when a system publishes messages into a topic and others system subscribe the to the topic will receive the message. After understanding what is a messaging system, now I will implement it on my design.

Lets Rock, My Cute Rabbit!

So here my new system design. Here I tried to use RabbitMQ.

RabbitMQ Applied

Sow how the communication will be? Let me explain. After a feed or a comment or an event stored into the database, the service that handles it will send a payload that contained its ID and field that contained hashtag.

Here the payload from Feed Service

{"feed_id": 23,"feed_caption": "Today is a beautiful day #sunset #beach #love"}

Here the payload from Comment Service

{"commnet_id": 1,"comment_body": "Hi Jude, don't make it bad #keepfighting"}

Here the payload from Event Service

{"event_id": 97,"event_description": "We are always welcome for every body that want to learn togerher with us. So don't miss the tutorail session. #tutorail #bootcamp #rabbitmq #microservices"}

But here, the payload did not send directly to the Hashtag Service. The payload from each service will be published to a topic in the RabbitMQ Server. The Hashtag Service will receive the published message from the RabbitMQ Server — of course, the Hashtag Service should subscribe to the same topic.

Actually, that was enough. But we need to modify our payload, so the Hashtag Service know what service publish the message. We also need to modify the payload to make all the payload have the same structure — as the contract to the Hashtag Service. Here are new payload example.

{"message_type": "feed""message_id": 23,"message": "Today is a beautiful day #sunset #beach #love"}

What others advantages?

Message Durability

Messages that published to RabbitMQ server will stay there until there is subscriber take it. Even if there is a message that not fully processed, the message will not directly delete from the queue. The RabbitMQ Server will delete it after receiving message acknowledgment — we need to configure it before. After the subscriber available, the RabbitQM Server will send messages from its queue to the subscriber.

But how if the RabbitMQ Server down? We will lose the data too. Read here for more detail information.

Marking messages as persistent doesn't fully guarantee that a message won't be lost. Although it tells RabbitMQ to save the message to disk, there is still a short time window when RabbitMQ has accepted a message and hasn't saved it yet. Also, RabbitMQ doesn't do fsync(2) for every message -- it may be just saved to cache and not really written to the disk. The persistence guarantees aren't strong, but it's more than enough for our simple task queue. If you need a stronger guarantee then you can use publisher confirms.

- RabbitMQ-Tutorial Two

Easy to scale

Implementing message broker, we can scale up our system design easily. As explained above, we added two others service to our system and reuse the existing service. If later we add Place Service, which the place has a description that contained hashtags, we just publish the payload to the RabbitQM server.

Flexible Tech Stack

Another advantage is we can use different programming languages and different database as I did before. We can choose the stack we want to use flexible, so we can get the advantages of all the tech stack. For example, we can consider using NoSQL for speed performance or SQL for data integrity, Node JS for dynamic typing or Java for static typing.

I think that was all that I can share for you. I’m myself still need to explore more about messaging system and microservices. If there any critic or feedback, feel free to response this article or reach me at [email protected]. Oh ya, you also can find the full code of Hashtag Service here and the Feed Service here.

Cappy Hoding! 💻