Looking to 2020 and beyond, the proportion of data produced and consumed in realtime is growing exponentially. that by 2025 1/3 of all data produced globally will be realtime. IDC predict The engineers and organizations that make up the realtime ecosystem, included, have yet to agree on how we describe the APIs we’re creating and consuming that are powering this growth. The problem is that there are various ways to describe APIs that provide realtime functionality. Ably API evangelist , and many others, have been and talking about these new types of APIs for a while. Event-driven seems to be the most common descriptor. Gartner has adopted the term, stating that . Kin Lane writing by 2020 50% of all managed APIs will be event-driven But still today there is no consistency or consensus between what terms like realtime API, event-driven API, or streaming API refer to. Often they’re used interchangeably. As an engineering team and API provider working on a global pub/sub messaging platform, we work with these ideas everyday. Talking to our users we see the language they use to describe both what we do and what they achieve using our APIs. And we’ve watched discussions around event-driven architecture, webhooks, and streaming data proliferate. Over the years we’ve thought extensively about the best terminology to use and arrived at what we call the Realtime API Family. The Realtime API Family Defining realtime Streaming API Pub/Sub APIs Push APIs Event-Driven APIs Defining realtime When it comes to realtime in the context of user interaction, it can be defined as a function or interaction perceived as immediate. As a concept in this context then, realtime is easy enough to understand. Ably defines it as anything that happens in under 100ms as this is typically the threshold for perceiving something as happening immediately. Any API designed such that data flowing from producers to consumers happens in the shortest amount of time possible can therefore be described as realtime. Ably has settled on realtime API as an umbrella for event-driven, streaming, pub/sub, push, and other APIs. We’ve seen many more than four ways of describing this, but these are the most popular. It’s why we call it the Realtime API Family. Streaming APIs A stream is a means of transporting data. Streaming is a consumer pattern that describes how consumers receive events through a stream. A streaming API will commonly address issues of data integrity with: - ensure messages are delivered in the order they were published - upon disconnection, resume from the disconnected point within a set period of time - a simple series of `ACK` or `NACK` responses, each addressing a contiguous sequence of messages Message ordering Stream continuity / resume Contiguous serial numbers Kafka offers a following this model for internal systems. streaming API Pub/Sub APIs Pub/Sub (Publisher/Subscriber) is an infamous messaging design pattern. When it debuted in 1987 it solved loose decoupling of servers. The earliest distributed systems used to send events between servers using Pub/Sub, but all within reliable networks. This pattern operates by publishing messages on a topic (or ) within a message bus, and subscribers can listen for events based on those topics. channel Ably's Pub/Sub Channels API acts as a serverless message bus Pub/Sub is over 30 years old now. The design makes no attempt to define semantics around ordering or continuity (loss of connection). While it’s still a pattern used widely today, these are issues that must be and in our world of unreliable connections. considered addressed Ably provides a following this model but with guarantees around ordering, continuity, idempotency, and more. pub/sub API Push APIs Ably's unified push notifications API Push is another umbrella term for every API that is real time. But it's just a producer messaging pattern. It means that data is pushed upstream over a connection, versus a pull mechanism used by the request/response pattern. An example is helpful. A is an example of a push API. Or a might send a single score update that traverses multiple systems via push APIs, resulting in millions of messages to global fans within a few hundred milliseconds. push notifications API sporting event Push can also refer to triggering a new request as with a . Or it can mean ‘push subscription’ - i.e. a producer needs to reach out to a consumer. WebSub is an example of this. But that’s a whole other blog post. Webhook Event-driven APIs Event-driven is an architectural design pattern that defines how a system processes data. It simply states a system should be responding/ to events as they happen. reacting Streaming, Pub/Sub, and Push are all messaging patterns that can delivered through an event-driven architecture. As such they can all fall under the umbrella of event-driven APIs. Most of us know what an event-driven API is but an example never hurts. Unlike traditional request/response APIs where data is requested, event-driven APIs push data from a producer to a consumer. They can be quite simple or very complex. An event-driven retail experience In the above example a retailer needs an event-driven architecture to respond to things as they happen. An event in an event-driven world triggers a chain of events which must be processed downstream, extending through the entire data supply chain. All components in this supply chain are reactive, rapidly responding to events and performing onwards processing. The time-bound, reactive nature of this data supply chain causes increased engineering and infrastructure complexity. And when compared to REST APIs following the request/response model, the complexity is inverted and put onto API producers rather than API consumers. The inversed complexity of REST and event-driven APIs However, that’s a conversation beyond the scope of this article. But it’s something we’ve spoken about before at 2019: . API World Taming the rising complexity of event-driven APIs And so we arrive at Realtime APIs Streaming, Pub/Sub, and Push are all patterns that can be delivered through event-driven architecture. But the outcomes of streaming, push, pub/sub, and event-driven are all realtime functionality. They’re all means of getting data from producer to subscriber in the shortest possible time. So we can call them realtime APIs. But you can’t have a realtime API unless it’s delivered as an event-driven, pub/sub, streaming, or push API. Hence how we’ve settled on the umbrella term of realtime API to encompass them all. Navigating complexity and fragmentation As you can see, it’s easy to use these terms interchangeably despite them all being different. The realtime ecosystem is still maturing and changes all the time. Eventually we believe developers and organizations will naturally come to a shared and standardized terminology for APIs designed to deliver real time functionality. Until then, the Realtime API Family helps us to reduce complexity when we explain our and to the developer community, our users, and our potential customers. cloud infrastructure APIs About Ably Ably provides cloud infrastructure and APIs to help developers simplify complex realtime engineering. Organizations build with Ably because we make it easy to power and scale realtime features in apps , or distribute data streams to third-party developers as realtime APIs. We support multiple open protocols and a growing number of third-party integrations . Take a look at what our customers say about us and our realtime APIs. . We're also hiring for roles across our commercial and engineering teams
![featured image - The Realtime API Family [A Deep Dive]](https://hackernoon.imgix.net/drafts/rjzh3nsi.png?auto=format&fit=max&w=1920)
![Delta Compression: Diff Algorithms And Delta File Formats [Practical Guide]](https://firebasestorage.googleapis.com/v0/b/hackernoon-app.appspot.com/o/images%2FYPYeQGgK3uVuaAoY97gRJBaXbLU2-cjb3evi.jpeg?alt=media&token=283b0cbd-f48d-4fc1-8566-be435375499b&auto=format&fit=max&w=3840)
