Our Ambitious Quest to Democratize Distributed Systems — Act 1: Protocols

In our last post we asked the question of whether early stage start-ups should adopt microservices architecture. Today we`re going a little bit further in highlighting the great potential distributed systems can bring to start-up companies and propose a possible easy route of adopting a new breed of distributed systems using some of tools we are currently developing at Nanosai.com.

Why Implement Distributed Systems?

A couple years ago the benefits of distributed computing seemed to be only centered around the use case of academic research. A common use case often cited is the ability of scientists tackling hard scientific problems in fields such as genomics via cross disciplinary/institutional collaboration using grid computing to easily perform huge data processing/analysis tasks that otherwise would take months to accomplish.

Fast forward today we live in a 24/7 data driven world where it is estimated that on average we generate around 2.5 quintillion bytes of data per day, a lot of that data being generate by users of popular services from household names such as Google, Facebook, Amazon, Netflix etc. A recent report by Cisco also predicts the following:

● By 2020, the gigabyte (GB) equivalent of all movies ever made will cross the global Internet every 2 minutes.

● Globally, IP traffic will reach 511 terabits per second (Tbps) in 2020, the equivalent of 142 million people streaming Internet high-definition (HD) video simultaneously, all day, every day.

● Global IP traffic in 2020 will be equivalent to 504 billion DVDs per year, 42 billion DVDs per month, or 58 million DVDs per hour.

Building smarter distributed systems to handle such a high volume of data represent a huge opportunity for both startups and consumers. For startups in financial services for example, it puts them in a strong position to challenge the big incumbent financial institutions that mostly rely on centralised legacy technologies that were not designed for the current explosion of connectivity and big data. For consumers it opens up more choice and better quality of services at more competitive pricing than with the big incumbents.

Why Protocols Matter?

Good communication is key for distributed systems to work smoothly as nodes need to be able to communicate properly with each other internally and also often externally depending on whether the system needs to access third parties or be exposed to third parties.

A fundamental challenge that developers face today when building modern distributed systems is that there are currently no viable standard protocols that make it easier for them to implement distributed systems the same way as what HTTP did for the old web. A common approach regarding how nodes communicate to each other when dealing with modern distributed systems such as microservices is to use either synchronous protocols (e.g. HTTP/REST) or asynchronous protocols (e.g. AMQP). We say HTTP is a synchronous protocol in the sense that it works on request/response pattern i.e. the client issues a request and waits for a response (often very long wait!). On the other hand opposite to HTTP approach, message passing (e.g. AMQP) approach is asynchronous in the sense that as a client you don`t need to wait for a response and in particular responses to messages can be sent back in a different order than they were sent.

To tackle the protocol challenges that developers face today, at Nanosai we are developing a new breed of open network protocol called IAP aimed at bringing wider applicability, performance, simplicity and intelligence to modern distributed systems as part of a smart eco-system. We designed IAP from ground up with following intentions;

• To be a viable replacement/supplement to HTTP, FTP, SNMP, ODBC, RPC etc.

• To support more message flows and thus more use cases than HTTP

• To be simpler and more general purpose than HTTP

• To be more compact than HTTP

• To be a lot smarter than HTTP

What Is IAP Protocol?

In a nutshell IAP (see IAP Specification) is an async free flow message oriented protocol. By free flow we mean that messages can flow freely in both directions between communicating nodes. IAP does not limit itself to the request-response communication pattern of HTTP, or other RPC protocols. Being a free flow message oriented protocol means for example that any of the following scenarios are valid:

• A sends N messages to B without B responding.
• A sends N messages to B. B responds every M messages.
• A sends 1 message to B. B responds with 1 message.
• A sends 1 message to B. Be sends N messages to A.
• B sends N messages to A without A responding.

An interesting feature of IAP is that its messages are encoded using a binary encoding called ION. ION resembles a binary version of JSON but is more compact on the wire, easier and faster to read and write and it comes with a few more data types than JSON. Essentially IO is more versatile, usable for things like data files, log files and network messages. We`ll cover ION with more details in future posts.

Another interesting feature of IAP is that it allows developers to define different semantic protocols on top of the basic message format. The basic message format contains a set of fields that are pretty much always used in network communication. Semantic protocols can hence extend this message format and add protocol specific fields.

Ultimately our goal is to make IAP a viable alternative to HTTP and capable of supporting a wider variety of message exchange patterns. We believe that in order to create a more versatile internet communication we really need a more versatile protocol than HTTP. In fact our co — founder and CTO Jakob Jenkov also wrote a post `Why HTTP2 and WebSockets Are Not Enough` making the case for why we need to move away from HTTP.

Finally, over the coming weeks and months we`ll be sharing our journey with you as follow up posts i.e. Act 2, Act 3… Act N. In meantime please feel free to visit github.com/nanosai to checkout our code and stay up to date with our progress.

Posted by Bambordé Baldé, Co - Founder at Zaiku.