A detailed guide to the world of MQTT

MQTT (formerly known as MQ Telemetry Transport) was in 1991 by Andy Stanford-Clark (IBM) and Arlen Nipper (Eurotech) in order to connect oil pipelines over unreliable, satellite networks. It is a machine-to-machine ( )/”Internet of Things” connectivity protocol. created M2M . It is mostly useful for establishing connections with remote locations where a small code footprint is required and/or network bandwidth is at a premium. The protocol is data-centric, unlike for example HTTP, which is document centric. It was designed as an extremely lightweight publish/subscribe messaging transport where clients instead of focusing on who the receiver of the message is, they focus on categorizing the messages in topics of interest. Messages are sent to where interested clients subscribe to. It is based on the Pub/Sub messaging pattern an intermediate, a message broker The message structure is compact and the overhead of the binary header is just 2 bytes. , which sometimes makes the usage of the protocol not very efficient in cases of . Connecting and disconnecting to the broker has a resource cost which, in certain cases, can make the usage of the protocol as efficient as other protocol alternatives. Connection utilization is important in MQTT. Every client has to be connected to the Message Broker to interact with the system one message duration connections At the moment the most widely used version of the protocol is MQTT v3.1.1 which is implemented by many . came out on November 7, 2014. brokers Version 3.1.1 On April 3, 2019, became an official standard bringing a lot of new features. Some of them are: MQTT v5.0 OASIS Shared SubscriptionsSession ExpirationMessage Expiration For the details of this version, you can refer to this . link MQTT and Message Queues Although some versions of the protocol can implement components that have similar behavior to queues, this is not one of the protocols described capabilities. This is usually a confusing subject. Many sources label MQTT as Message Queueing Telemetry Transport which is not correct. “MQ” refers to the MQ Series, an IBM product developed to support MQ telemetry transport. When the protocol was created, it was named after the IBM product. The protocol is not directly related to message queues . In message queues, messages are until a consumer gets the message. In MQTT, messages that arrive in a topic without any subscribers are not kept in queues (unless a persistent session is enabled where all QOS1 & QOS2 messages, that are not yet confirmed or missed, are stored until consumption). stored in a queue Messages in queues can be consumed by one client. In MQTT a message can have multiple subscribers. Message Queuing and Pub/Sub Messaging are two completely different messaging patterns. The biggest difference with queues is that they need to be explicitly created and they also must be named. This is not the case with MQTT, where topics are flexible and created on the fly under a tree-like structure. MQTT can provide a type of message queues under certain circumstances. This can happen when a persistent session is present between a client and the broker. If a client that has a persistent session is disconnected from the broker and at some point reconnects, the following information is available immediately: All the subscriptions of a clientAll QOS1 & QOS2 flow messages that have not been confirmedAll new QOS1 & QOS2 messages that the client missedAll QOS2 messages received from the client and are not yet completely acknowledged More information regarding persistent sessions can be found . here Protocol Characteristics Simplicity By design, the MQTT protocol focuses on simplicity and mainly exposes 5 ways of interaction. — — — — Client Connect Client waits for a connection to be established with the message broker. Disconnect Broker waits for the MQTT client to finish any work, which needs to be done for the TCP/IP session to disconnect. Subscribe Client Requests to subscribe to one or more topics. Unsubscribe Requests to unsubscribe from one or more topics. Publish — Client or Broker, publish a message to a topic. Multiple Service Levels (QoS Quality of Service) Message delivery can be achieved in different levels by the protocol which can be very convenient depending on the use case. QoS can be different for subscribing and publishing. Below we can see how the different levels work. : Entities can either refer to any client or the broker. three NOTE QOS0 The message will be published or received (fire and forget). at most once QOS1 The message will be published or received meaning that the sender stored the message and will probably repeat the procedure if he does not receive an acknowledgment. at least once, QOS2 The message will be published or received and will include an extra verification step for the communicated message. exactly one time the publisher, the broker, and the subscriber. What is really worth making crystal clear, is that the involved entities in each communication scenario are three, (QoS). Each client communicates only with the intermediate, the broker. . Publisher and subscriber are not designed to use the same type of service Clients are not destined to know the existence of one another Let’s analyze an example. Say we have a that connects to the message broker and in a topic every 10 seconds in a “ ” type of service. An for example on the other side in an “ ” type of service, and displays the received temperature on the screen. smart thermometer publishes its temperature fire and forget app has subscribed to the temperature topic, at least once In this example, the smart thermometer has a very small network bandwidth footprint since it is publishing in a manner. In most cases, temperatures do not tend to drastically change in the time frame of 10 seconds so QOS0 seems to be the most appropriate type to use. fire and forget The app, on the other hand, should display differences in temperature, so it should receive any messages . any at least once This summarizes our design and brings us to the conclusion that involved entities, always depending on the case, should not use the same type of service. One To ∞ Message Distribution & Topic Structure . Messages could be identified as interests or topics of discussion that anyone interested, could receive knowledge of. Each client that publishes a message on the message broker, by design, has no awareness that anyone other than the broker has received the message What I find particularly funny is that when a client subscribes to a topic, it has no knowledge, whether or not, the particular topic actually exists or has (LWT) or will have any data in it. This reminds me of the , but instead of the cat, we are referring to the data that exist or will exist in a topic. Schrödinger’s cat thought experiment The only way to find out about them is to subscribe to a topic. Topics that exist or will exist can have multiple or no subscribers at all. . It is a tree-like structure that is split into levels. Topics defined by clients are UTF-8 strings that are processed by the broker to filter messages. Each topic consists of one or more topic levels. Topic levels are separated by forward-slashes “/”. MQTT topic structure is often mentioned as the topic tree When subscribing to a topic, there are specific characters that can be used as wildcards. MQTT supports two types of wildcards. The single-level wildcard “ ” and the multi-level wildcard “ ”. + # Let’s take the following topic tree as an example. If we wanted to subscribe to messages regarding the humidity of the kitchen, all we would have to do is subscribe to: home/kitchen/generic/humidity What if we wanted to subscribe to the temperature of all the available rooms? We would use the single level wildcard character. home/+/generic/temperature Last but not least, where would we subscribe to, if we wanted to listen to all messages regarding the kitchen and everything that is in there? This time we would use the multi-level wildcard character. home/kitchen/# There are some best practices regarding topic structure that we will discuss later on. Compact Packet One of MQTT’s main strengths is the small overhead it adds to communications. Its packet headers are compact and have a binary structure. They consist of 3 parts at most. , present in all MQTT Control Packets , present in some MQTT Control Packets , present in some MQTT Control Packets Fixed header Variable header Payload The is always present in each MQTT packet. It is two bytes in size. The detailed structure of the fixed header is given below. Fixed header The is not present in all MQTT control packets. Its structure is different for different MQTT requests. Variable header is the actual data which is going to be sent. It’s not present in all the MQTT control packets. The payload of the message is limited to bytes. Payload 268,435,456 The length of the actual topic string is at most bytes. This is a limit imposed by the MQTT spec. It is also worth noting that the topic is encoded in UTF-8, so there might be less than characters available. 65536 65536 The MQTT spec details can be found . here Connection loss awareness The broker is always aware of all the connections that clients establish or terminate. There are also certain features that utilize this event. (LWT), where every connected client describes his will, that upon disconnection a message will be published on a specific topic. that we referred to earlier Last Will and Testament Persistent Sessions Simple but not limiting authentication By design, MQTT offers a very . simple authentication mechanism The Packet contains and fields. CONNECT Username Password They may provide their own authentication mechanism, use an external authentication system such as LDAP or OAuth tokens, or leverage operating system authentication mechanisms. Implementations passing authentication data in clear text, obfuscating such data elements or requiring no authentication data should be aware, that this can give rise to Man-in-the-Middle and replay attacks. Implementations can choose how to make use of the content of these fields. [RFC4511] [RFC6749] TLS can provide encryption of data sent over the network. There are valid TLS cipher suites that include a NULL encryption algorithm that does not encrypt data. To ensure privacy, Clients and Servers should avoid these cipher suites. [RFC5246] An application might independently encrypt the contents of its Application Messages. This could provide privacy of the Application Message both over the network and at rest. This would not provide privacy for other properties of the Application Message such as Topic Name. Client and Server implementations can provide encrypted storage for data at rest such as Application Messages that are stored as part of a Session. The use of VPNs to connect Clients and Servers can provide privacy of data across the section of the network covered by a VPN. Every broker that exists in the market has its own modules to handle authentication but generally they can be extended further to support authentication mechanisms such as those mentioned above. These implementations are usually custom, although some brokers on the market have some modules already implemented. Best Practices Although there are no restrictions on how Apps using the protocol are implemented, there are some practices that are widely adopted. Topic structure in terms of interest. The topic structure should resemble a tree structure and should make possible the utilization of wildcards for subscriptions. Information regarding a specific topic should provide the ability to be retrieved by multiple entities. Topics are meant to categorize information into groups, To achieve that, since it introduces another unused topic layer in the tree with a zero character that is not used. This can cause unnecessary confusion. consistency between topic layers should be maintained.Do not use a leading forward slash , UTF-8 has many different white space types and can cause trouble in debugging. Characters like these should be always avoided. Never use spaces in a topic , the client might not be able to process the load and might also put some unnecessary strain on the broker. If you need, for some reason let’s say store all messages on a database, you should implement some kind of plugin on the broker to achieve this. Never subscribe to “#” Topic name conventions should be maintained across all topics in order for your system to be able to adapt to new topic addition in the future without breaking the tree structure. Specific topics and Extensibility should always be on the designer's mind since your topic structure should be able to extend. You should be able to distinguish each entity’s information or group it by a single level wildcard. Use a unique identifier on topics that refer to specific entities. Proper Service Levels As we already mentioned, Always take note that QOS2 service level is expensive and should be used only when it is absolutely necessary. QOS0 should be used only on non-sensitive data as message loss can occur. QOS1 should be used when receiving duplicate data will not cause any trouble. service levels can be different from client to broker & broker to client. Connection Utilization In MQTT, . Always avoid connections that will be utilized for a single information transaction. Maintain connection or send data to batches if possible. Single transaction connections almost ruin all the pros of the fast and compound nature of the protocol. establishing a connection is not a cheap task Security . Your connected entities will most likely be on a remote location meaning that your broker should be visible to the world. If your entities can execute commands or take orders from messages, then it’s not only a matter of security but also a matter of safety since sophisticated attackers can perform man-in-the-middle attacks at least. Security should never be taken lightly and TLS over TCP should always be used along with all the best practices that come along This is a matter of great importance, not only on MQTT related implementations but, generally on IoT technologies, we all can remember what the attacks taught us in 2016. Mirai-bot Brokers on the market There are several brokers out there that implement the MQTT protocol. Here are some of them that are my personal favorites. — Is particularly interesting due to it’s scaling capabilities and provided MQTT features. VerneMQ — (Mosquitto on Kubernetes) — having a broker “natively” on kubernetes. KubeEdge — I actually started out experimenting some years ago using Mosca. Mosca and Mosquitto are most likely the best brokers to learn, test and debug. Mosca-Mosquitto Conclusions It has many features such as the multiple service levels, persistent sessions, last will & testament and many more. It implements a unique tree topic structure that enables flexible management of information. Last but not least, there are several brokers and clients out there can cope with the scaling demands of the present. MQTT is a PUB/SUB communication protocol that enables light, fast and effective communication between entities.