The Realtime API Family [A Deep Dive]

Looking to 2020 and beyond, the proportion of data produced and consumed in realtime is growing exponentially. IDC predict that by 2025 1/3 of all data produced globally will be realtime.

The engineers and organizations that make up the realtime ecosystem, Ably included, have yet to agree on how we describe the APIs we’re creating and consuming that are powering this growth. The problem is that there are various ways to describe APIs that provide realtime functionality.

API evangelist Kin Lane, and many others, have been writing and talking about these new types of APIs for a while. Event-driven seems to be the most common descriptor. Gartner has adopted the term, stating that by 2020 50% of all managed APIs will be event-driven.

But still today there is no consistency or consensus between what terms like realtime API, event-driven API, or streaming API refer to. Often they’re used interchangeably.

As an engineering team and API provider working on a global pub/sub messaging platform, we work with these ideas everyday. Talking to our users we see the language they use to describe both what we do and what they achieve using our APIs.

And we’ve watched discussions around event-driven architecture, webhooks, and streaming data proliferate.

Over the years we’ve thought extensively about the best terminology to use and arrived at what we call the Realtime API Family.

The Realtime API Family

1. Defining realtime
2. Streaming API
3. Pub/Sub APIs
4. Push APIs
5. Event-Driven APIs

Defining realtime

When it comes to realtime in the context of user interaction, it can be defined as a function or interaction perceived as immediate. As a concept in this context then, realtime is easy enough to understand. Ably defines it as anything that happens in under 100ms as this is typically the threshold for perceiving something as happening immediately.

Any API designed such that data flowing from producers to consumers happens in the shortest amount of time possible can therefore be described as realtime.

Ably has settled on realtime API as an umbrella for event-driven, streaming, pub/sub, push, and other APIs. We’ve seen many more than four ways of describing this, but these are the most popular. It’s why we call it the Realtime API Family.

Streaming APIs

A stream is a means of transporting data. Streaming is a consumer pattern that describes how consumers receive events through a stream.

A streaming API will commonly address issues of data integrity with:

Message ordering - ensure messages are delivered in the order they were publishedStream continuity / resume - upon disconnection, resume from the disconnected point within a set period of timeContiguous serial numbers - a simple series of `ACK` or `NACK` responses, each addressing a contiguous sequence of messages

Kafka offers a streaming API following this model for internal systems.

Pub/Sub APIs

Pub/Sub (Publisher/Subscriber) is an infamous messaging design pattern. When it debuted in 1987 it solved loose decoupling of servers. The earliest distributed systems used to send events between servers using Pub/Sub, but all within reliable networks. This pattern operates by publishing messages on a topic (or channel) within a message bus, and subscribers can listen for events based on those topics.

Ably's Pub/Sub Channels API acts as a serverless message bus

Pub/Sub is over 30 years old now. The design makes no attempt to define semantics around ordering or continuity (loss of connection). While it’s still a pattern used widely today, these are issues that must be considered and addressed in our world of unreliable connections.

Ably provides a pub/sub API following this model but with guarantees around ordering, continuity, idempotency, and more.

Push APIs

Ably's unified push notifications API

Push is another umbrella term for every API that is real time. But it's just a producer messaging pattern. It means that data is pushed upstream over a connection, versus a pull mechanism used by the request/response pattern.

An example is helpful. A push notifications API is an example of a push API. Or a sporting event might send a single score update that traverses multiple systems via push APIs, resulting in millions of messages to global fans within a few hundred milliseconds.

Push can also refer to triggering a new request as with a Webhook. Or it can mean ‘push subscription’ - i.e. a producer needs to reach out to a consumer. WebSub is an example of this. But that’s a whole other blog post.

Event-driven APIs

Event-driven is an architectural design pattern that defines how a system processes data. It simply states a system should be responding/reacting to events as they happen.

Streaming, Pub/Sub, and Push are all messaging patterns that can delivered through an event-driven architecture. As such they can all fall under the umbrella of event-driven APIs.

Most of us know what an event-driven API is but an example never hurts. Unlike traditional request/response APIs where data is requested, event-driven APIs push data from a producer to a consumer. They can be quite simple or very complex.

An event-driven retail experience

In the above example a retailer needs an event-driven architecture to respond to things as they happen.

An event in an event-driven world triggers a chain of events which must be processed downstream, extending through the entire data supply chain. All components in this supply chain are reactive, rapidly responding to events and performing onwards processing.

The time-bound, reactive nature of this data supply chain causes increased engineering and infrastructure complexity. And when compared to REST APIs following the request/response model, the complexity is inverted and put onto API producers rather than API consumers.

The inversed complexity of REST and event-driven APIs

However, that’s a conversation beyond the scope of this article. But it’s something we’ve spoken about before at API World 2019: Taming the rising complexity of event-driven APIs.

And so we arrive at Realtime APIs

Streaming, Pub/Sub, and Push are all patterns that can be delivered through event-driven architecture. But the outcomes of streaming, push, pub/sub, and event-driven are all realtime functionality.

They’re all means of getting data from producer to subscriber in the shortest possible time. So we can call them realtime APIs.

But you can’t have a realtime API unless it’s delivered as an event-driven, pub/sub, streaming, or push API. Hence how we’ve settled on the umbrella term of realtime API to encompass them all.

Navigating complexity and fragmentation

As you can see, it’s easy to use these terms interchangeably despite them all being different. The realtime ecosystem is still maturing and changes all the time.

Eventually we believe developers and organizations will naturally come to a shared and standardized terminology for APIs designed to deliver real time functionality.

Until then, the Realtime API Family helps us to reduce complexity when we explain our cloud infrastructure and APIs to the developer community, our users, and our potential customers.

About Ably

Ably provides cloud infrastructure and APIs to help developers  simplify complex realtime engineering. Organizations build with Ably  because we make it easy to power and scale realtime features in apps, or distribute  data streams to third-party developers as realtime APIs. We support multiple open protocols and a growing number of third-party integrations. Take a look at what our customers say about us and our realtime APIs.

We're also hiring for roles across our commercial and engineering teams.