Creating Our Own Custom Messaging Solution for IoT Devices

Effective data transmission is essential for smart networks. There is a wide range of communication protocols that can provide smooth interaction between connected devices. The alternative solution designed by our engineers can compete with existing technologies such as HTTP and MQTT. Here, we will tell you about Integra’s messaging library, structure, functionality, and competitive strengths. 

The Internet of Things (IoT) relies on the interoperation of network nodes, including the collection and exchange of data. To implement a messaging system in your product, you can use a suitable out-of-the-box technology. However, ready-made solutions are not universal, so they cannot fit every project’s needs.

In one of the IoT systems developed by the Integra team, we needed to connect the client devices with the database and peripheral modules. Our developers were looking for a highly flexible solution that could meet all the project requirements. 

Finally, we created a custom technology that improved the efficiency of the client-server interaction. Our team integrated several components, uniting them into Integra Sources Messaging Library, or ISML. In this article, we’ll elaborate on the design and capabilities of our tailor-made solution. 

Why We Decided to Develop Our Own Messaging Solution

In one of our recent IoT projects, we built an API on the server’s side of a device to interact with the peripherals. During the project development, we had to face some common challenges. Initially, we tried to meet them using standard networking technologies. 

Thus, we used HTTP to interact with the web dashboard of the IoT system. To establish the communication between the server and the client device, which is a charging station, we tried to use the MQTT protocol. This lightweight networking standard is widely used for client-server communication in low-bandwidth systems. 

However, it didn’t appear suitable for our project, as it couldn’t guarantee efficient data transfer and a stable connection with the charging devices (some of them worked on 2G networks that provided poor throughput and irregular communication).  

So we needed more reliable technology to keep a steady connection with the chargers. That’s why we decided to develop our own messaging solution, which optimized communication between the network components. 

ISML has much in common with similar IoT messaging protocols. It is also a lightweight solution intended for smart networks that require fast and efficient data exchange. It uses a communication model similar to the messaging mechanism of MQTT and has easy implementation like the POrtable COmponents C++ Libraries (POCO).

However, we can point out several advantages of our custom IoT messaging product that can optimize the interaction of components within your system.

Documentation. The Integra engineers prepared clear and concise documentation that helps you design, deploy, and support your ISML-based project.

Flexibility. It is a highly customizable cross-platform solution that you can use for messaging in various environments.

Minimal overhead. ISML has a very light architecture. It runs over TCP, and its overhead is lower compared to MQTT and HTTP.  

Scalability. This is a highly scalable custom solution that allows you to build networks with an unlimited number of nodes. This can only be restricted by your server’s capacity.

Security. The library has a malware protection system that filters incoming messages.

The simplicity of implementation. This is a developer-friendly library that you can easily customize and integrate into your current C++ code. 

The simplicity of deployment. You can deploy ISML with only a local or cloud server available at hand. We provide a full set of components, tools, and instructions.

Our messaging library has wide use and can fit a number of projects connected with data exchange and serialization, session management, and inter-networking. ISML is one of the solutions that the Integra engineering team creates within our system and application software development services. We build network applications and protocols that provide smooth communication between the components of IoT and embedded systems.

What is ISML?

ISML is a set of components designed to integrate message-oriented middleware (MOM). MOM is the software that supports communication within a distributed environment. The message-oriented infrastructure of the ISML library facilitates the implementation of a dispatcher, router, and other components related to data delivery. 

You can use the library to develop applications with client-server and peer-to-peer (P2P) architectures. ISML provides interfaces to create, send, receive, and store communication data. It is designed for the efficient transmission of binary files. There are several classes used to generate different types of messages and deliver them.

ISML works with sessions, so it doesn’t support connectionless protocols, such as UDP and IP. At the present moment, it is compatible with only one network protocol, which is TCP.

However, we singled out the transport layer as a separate class. It helps the client device interact with the peripherals, for example, via RS-485. So, in the future, you can build a data exchange system using any bus and transport protocol - either connectionless or connection-oriented.

ISML uses a publish/subscribe pattern which differs slightly from the model applied in similar messaging technologies. For example, MQTT employs a broker that serves as a mediator between the network nodes. In our communication model, clients can interact with the server directly.  

There is no need for topics since the server has a list of clients subscribed to certain sessions. To simplify the data transfer, you can also create a message channel for each of the sessions.

The architecture of the ISML library enables you to add any encryption and authentication systems and cryptographic protocols. By customizing it for your particular project, you can achieve the required security level.

ISML is a highly flexible and customizable solution that enables users to implement their own logic, leveraging the components of the library. 

ISML Components 

ISML comprises interrelated components or classes used to implement the functionality of a data exchange system. The basic classes include Session, Transport, SessionManager, and Message.

ISML class diagram

Session 

A communication session introduced as the Session class is one of the key components of the library. It helps identify the messaging client in the system. This class represents a high-level interface for sending and receiving messages. 

You can also use Session to store various data necessary for the management of the latest session. The data includes session lifetime, latest activity time, the number of transmitted messages, etc. 

The user can expand and customize the list of the stored parameters with the help of a built-in mechanism for the Properties class - a heterogeneous container for data storage.  

Session also provides a high-level interface for sending requests and replies. It differs from the standard sending and receiving of messages in its synchronized nature. It allows the requestor to send a message and switch to a waiting mode without changing the execution context. 

This mechanism can help send confirmations or notifications. It assists in implementing the Remote Procedure Call (RPC) and requests using the messaging system (for example, a request to the network node granting access to the database).

Transport

In an ordinary use case, sessions appear in the system during the setup of the connection with a remote client (on the server’s side) and upon server connectivity (on the client’s side). In both cases, the socket connection gets established regardless of the applied protocol - whether it is TCP or WebSocket.

ISML offers the Transport component that encapsulates all the necessary resources to manage the connection and is a part of the logic responsible for sending and receiving messages. 

Transport equips the library user with a high-level interface that allows for messaging without referencing all low-level details specific to the used protocol. For example, we use the Asio library that provides an input/output context and a set of objects to work with TCP. The Transport class implements the messaging mechanism on the basis of this protocol hiding the TCP/IP stack routine from the user.

Along with that, Transport is responsible for storing the messages sent and received. It releases users from building their own mechanisms that implement this functionality. The messages are stored in a queue that can operate either in a single-thread system or under concurrency. 

Thus Transport makes it possible to work with a queue in a multithreaded environment feeling secure about data or API races. The queue interface eliminates the risk of the races using the optional types. It also excludes all possible methods that could lead to such situations, for example, calculations of the queue size.

Creating a session, you should pass a class instance that implements message Transport. The session uses it to send and receive messages. 

SessionManager

ISML contains the SessionManager component that manages sessions and provides the interface to create them. For the component’s usability, you can register a callback in the network sub-system on one of the possible events - connection accepted or connected. 

An Acceptor or ConnectionListener is a low-level component of the library. It is responsible for receiving the connection events and notifying SessionManager of the necessity to create a session. 

Thus, in case of a successful connection, an instance of the message Transport will be passed to SessionManager. The latter will create a new session and connect it with the specified Transport.   

Creating a session

Message and Related Classes

The Message class introduced by ISML is a data unit of a messaging system. A Message is a set of fields used to store valuable data that builds up the message body. It also includes helper fields used to identify a message, determine its type and owner.  

The MessageFactory class is a special component that creates messages out of the raw data. Building messages outside MessageFactory is unacceptable and will lead to system errors. To create a message, you should know its type and session that acts as a context.

The type of message defines its structure. There can be any number of message types registered in the system. What is important is that they should be unique. A message type itself is an ordinary numerical identifier. Its underlying type is determined by MessageType.

As soon as a message is created, it is then put into the message channel or queue. A Dispatcher is the library component that registers handlers for different message types. Thus, handler 1 is invoked to process  MessageType 1, etc. 

To create a reply, we should provide MessageFactory with a specific MessageType and a Session. 

Message flow 

One numerical identifier is obviously not enough to describe the message structure. ISML comprises the Field class, which is a basic type of message field, and FieldDescriptor. The latter provides an interface to receive additional information about the field, for example, its name, and assists in building the desired field type.

Both components belong to abstract classes, and they can’t serve the purposes mentioned above directly. To store data, you can use the descendant classes - DataField and DataFieldDescriptor, respectively. This method is called type erasure. It is used to store any data, including built-in types of the C++ language and user-defined types (UDTs). 

FiledDescriptors get united into a message descriptor that is passed to MessageFactory as a prototype. This prototype will be the basis for an instant message of a specific type.

Schematically, we can show it as follows:

So, each message has a descriptor that contains information about the message type (also called a numeric identifier) and its fields. These fields are not stored in the message descriptor itself, but field descriptors are used - they create an actual field of the desired type with the desired name.

Thus, the message descriptor is the prototype of the message; when using information about the message type and fields, messages can be created from its descriptor an unlimited number of times. 

In other words, MessageDescriptor contains descriptors of all the fields that the message of this type will include. Thus, 

Message is a heterogeneous container, and it can embrace the fields that store values of any data. 

Conclusion

The widespread use of the Internet of Things is closely connected with the growth of energy-efficient communication technologies. They make it possible to create networks of smart devices that effectively interact with each other. 

Such standards as MQTT, HTTP, and POCO are broadly used in IoT development. However, they are not multipurpose, and you can’t implement them in any scenario. Some projects require much more flexible and easy-to-use solutions. 

Integra Sources Messaging Library is a custom technology that provides strong and stable communication within a network of connected devices. It can make your project development easy, timely, and cost-friendly. If you want to learn more about our IoT messaging solution or customize it according to your needs, feel free to get in touch with our engineering team.   

Copyright © 2021 Integra Sources. All rights reserved.

This library is an internal development of Integra Sources Ltd. 

This library is published with usage restrictions. 

If you are interested in using this library in your project, please contact us, and we will permit you to use it on your personal terms.

License.

ISML on GiTHub

Also Published On: https://www.integrasources.com/blog/custom-messaging-solution-iot-devices/