Exposing Kafka messages via a public HTTP streaming API is the new hotness when it comes to adding realtime messaging capabilities to your system. At its core, it is an open source distributed messaging system that uses a publish-subscribe system for building realtime data pipelines. But, more broadly speaking, it is a distributed and horizontally scaleable commit log. Apache Kafka In a Kafka cluster, you will have topics, producers, consumers, and brokers: Topics — A categorization for a group of messages Producers — Push messages into a Kafka topic Consumers — Pulls messages off of a Kafka topic Kafka Broker — A Kafka node Kafka Cluster— A collection of Kafka brokers Take a deep dive into Kafka here . Overall, Kafka provides fast, highly scalable and redundant messaging through a publish-subscribe model. A pub-sub model is a messaging pattern where publishers categorize published messages into topics without knowledge of which subscribers would receive those messages (if any). Likewise, subscribers express interest in one or more topics and only receive messages that are of interest, without knowing anything about the publishers ( ). source Kafka Strengths As a messaging system, Kafka has some transformative strengths that have catalyzed its rising popularity — Can handle realtime messaging throughput with high currency Realtime Data Pipeline — Ability to support high-velocity and high-volume data (1000’s per second) High-throughput — Due to its distributed nature, it is relatively resistant to node failure within a cluster Fault-tolerant — Milliseconds to handle thousands of messages Low Latency — Kafka’s distributed nature allows you to add additional nodes without downtime, facilitating partitioning and replication Scalability Kafka Limits Due to its intrinsic architecture, Kafka is not optimized to provide API consumers with friendly access to realtime data. As such, many orgs are hesitant to expose their Kafka endpoints publicly. In other words, it is difficult to expose Kafka across a public API boundary if you want to use traditional protocols (like websockets or HTTP). To overcome this limit, we can integrate Pushpin into our Kafka ecosystem to handle more traditional protocols and expose our public API in a more accessible and standardized way. Pushpin + Kafka Server-sent events (SSE) is a technology where a browser receives automatic updates from a server via HTTP connection (standardized in HTML5 standards). Kafka doesn’t natively support this protocol, so we need to add an additional service to make this happen. primary value prop is that it is an solution that enables realtime push — a requisite of evented APIs ( ). At its core, it is a reverse proxy server that makes it easy to implement WebSocket, HTTP streaming, and HTTP long-polling services. Structurally, Pushpin communicates with backend web applications using regular, short-lived HTTP requests. Pushpin’s open source GitHub Repo Integrating Pushpin and Kafka provides you with some notable : benefits — Provides a more logical resource-oriented API to consumers that fits in with an existing REST API. In other words, you can expose data over standardized, more-secure protocols. Resource-Oriented API — Reuses existing authentication tokens and data formats. Authentication — Harnesses your existing API management system or load balancers. API Management — If the number of your web consumers grows substantially, then it may be more economical and performant to scale out your web tier, rather than your Kafka cluster. Web Tier Scaleability In this next example, we will expose Kafka message via HTTP streaming API. Building Kafka Server-Sent Events This reads messages from a Kafka service and exposes the data over a streaming API using Server-Sent Events (SSE) protocol over HTTP. It is written using Python & Django, and relies on Pushpin for managing the streaming connections. example project How it Works In this demo, we drop a Pushpin instance on top of our Kafka broker. Pushpin acts as a Kafka consumer, subscribes to all topics, and re-publishes received messages to connected clients. Clients listen to events via Pushpin. More granularly, we use to set up an SSE endpoint, while handles the messaging input and output. views.py relay.py First, we need to setup virtualenv and install dependencies: virtualenv --python=python3 venv. venv/bin/activatepip install -r requirements.txt 2. Create a suitable **.env** with Kafka and Pushpin settings: KAFKA_CONSUMER_CONFIG={"bootstrap.servers":"localhost:9092","group.id":"mygroup"}GRIP_URL=http://localhost:5561 3. Run the Django server: python manage.py runserver 4. Run Pushpin: pushpin --route="* localhost:8000" 5. Run the **relay** command: python manage.py relay The command sets up a Kafka consumer according to , subscribes to all topics, and re-publishes received messages to Pushpin, wrapped in SSE format. relay KAFKA_CONSUMER_CONFIG Clients can listen to events by making a request (through Pushpin) to : /events/{topic}/ curl -i http://localhost:7999/events/test/ The output stream might look like this: HTTP/1.1 200 OKContent-Type: text/event-streamTransfer-Encoding: chunkedConnection: Transfer-Encoding event: messagedata: hello event: messagedata: world Repo on GitHub
