Adding Realtime Data Streaming To Your App

Discover the technologies driving the realtime data ecosystem

Realtime data currently enables some of our most beloved apps: Slack, Google Maps, Uber, Lyft… the list goes on. It has become essential for our time-sensitive, reactive lifestyles, where even a few seconds of delay causes us substantial grief. Realtime has become a cornerstone of big data and IoT, where the faster the data is received, the faster it can be processed and made actionable. The difference between your app succeeding or failing will come down to its reliability, functionally, and ultimately its speed.

“[Realtime] is about the ability to make better decisions and take meaningful actions at the right time. It’s about detecting fraud while someone is swiping a credit card, or triggering an offer while a shopper is standing on a checkout line, or placing an ad on a website while someone is reading a specific article. It’s about combining and analyzing data so you can take the right action, at the right time, and at the right place.” ― Mike Barlow, Real-Time Big Data Analytics: Emerging Architecture

For the past few years, Silicon Valley has capitalized on the rising demand for realtime data by developing both realtime APIs and services. These solutions allow you to easily stream data to multiple clients (devices like phones, laptops, other servers), and are typically more out-of-the-box infrastructure services. Realtime APIs can drop into existing applications and works great with serverless or microservice backends. On the other hand, realtime app services as much more client focused, like delivering realtime updates to an app.

This list will outline a few realtime technologies and hopefully give you, the app developer or aspiring entrepreneur, more insight into the technological landscape. Keep in mind, these aren’t listed in any particular order because they all depend on your particular use cases and stack.

Apache Kafka

One of the most popular distributed streaming platforms, Kafka is used for building realtime data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, fast, and runs in production in thousands of companies. It has two main use cases:

• Building realtime streaming data pipelines that reliably get data between systems or applications
• Building realtime streaming applications that transform or react to the streams of data

Pubnub

Pubnub is one of the industry leaders in realtime services. They frame themselves as a global Data Stream Network (DSN) and realtime infrastructure-as-a-service (IaaS) company, which focuses on streaming updates to web, mobile, and IoT applications. PubNub focuses on data streaming using a publish/subscribe model for devices, which harnesses communication patterns already in the browser. One of their main strengths is the breadth of their SDK and integration libraries, enabling realtime for almost any stack.

Fanout/Pushpin

Funded by Seed Sumo, Fanout.io is a drop-in realtime API that lets you push data to connected devices instantly. Because it serves as a proxy and can work with any backend language, you do not have to substantially reconfigure your current API inner-workings. Fanout is different than most realtime providers because it supports API development, whereas the others do not. Hence, this product is ideal for developers that want to do more advanced application control, like use Fanout with their API management tool or to manage microservices. Pushpin is the open source version of Fanout. Since Pushpin is a proxy server, the backend can be written in any language, which makes integration very easy. Moreover, it works nicely with any language or framework, including any existing API management services that you use.

Pusher

Similar to Pubnub, Pusher focuses on data streaming for devices that uses a publish/subscribe model. They provide over 30 libraries to help with integration — including developer-friendly features for debugging and access controls. One of the main differences between Pubnub and Pusher is that Pusher uses websockets, whereas Pubnub does not. You should definitely ask your developers which approach would be better for your app.

Realtime Messaging by Realtime.co

Realtime Messaging is a cloud based message broker that enables developers to build cross-platform apps that require realtime communication between devices. It offers quick-start guides for JavaScript, Android, and iOS — and SDKs for all major platforms. One of the main benefits is that is handles your auto-scaling for you, so you can grow from 1 device to millions.

Ably

An alternative to PubNub, Ably’s platform ensures interoperability between clients using a native realtime protocol, and clients using other realtime or queuing protocols. Ably uses a publish and subscribe service to organize the message traffic within applications into named channels. Clients (laptops, phones, servers) attach to channels to subscribe to messages, and every message published to a unique channel is broadcast by Ably to all subscribers.

Socket

Socket.io is an awesome open source project. It is a JavaScript library for realtime web applications. It has two parts: a client-side library that runs in the browser, and a server-side library for Node.js. Systemically, it is similar to Pusher insofar as it relies on websockets (with polling as a fallback). Socket’s getting started guide will show you how to create lots of amazing applications in fewer than 200 lines of code.

Firebase

One of the fastest growing realtime DB’s, the Firebase Realtime Database is a cloud-hosted database where data is stored as JSON and synchronized in realtime to every connected client. All clients share one Realtime Database instance and automatically receive updates with the newest data. Firebase a flexible, expression-based rules language (Firebase Realtime Database Security Rules) to define how your data should be structured and when data can be read from or written to.

Streamdata.io

Streamdata.io a SaaS API proxy tool that converts standard API requests into a streaming API. In other words, it provides a proxy as a service for any HTTP API by polling and acting as a streaming API. More specifically, it is a realtime cache proxy that allows you to poll JSON REST APIs and push updates to clients — keeping a history of change modifications that occur between pollings. This way, Streamdata.io is able to give you the list of modifications which happened since the last data request.

Layer

Layer provides developers with APIs, SDKs, and UI toolkits to integrate realtime messaging into your products. They aim to provide developers with the proper tools that matches the rapid development and user experience of leading consumer messengers such as iMessage. Layer is 100% managed infrastructure so developers can easily integrate messaging into their applications. In other words, they handle servers, scaling, performance, and encryption as well. They are HIPPA-complaint, designed for asynchronous, person-to-person messaging or business-to consumer uses cases as well. Here are the many services they offer:

Amazon Kinesis

Amazon Kinesis makes it easy to collect, process, and analyze realtime, streaming data so you can get timely insights and react quickly to new information. It allows you to process and analyze data as it arrives and respond in realtime instead of having to wait until all your data is collected before the processing can begin. One of the main benefits is that it is a fully managed platform, so you run streaming applications without requiring any infrastructure management.

Looking to get started with building realtime APIs or apps? Check out Getting Started with Building Realtime API Infrastructure