

Open-Source Web UI for Apache Kafka
As a messaging platform, Kafka needs no introduction.ย Since its inception, it has virtually rewritten the book on event streaming and has catalyzed the adoption of the now household design patterns โ microservices, event-sourcing, and CQRS.
Being such a godsend, itย almostย gets away with its notorious lack of tooling. Youโd be hard-pressed to find a developer who hasnโt at one time looked at the built-in CLI tools, cupped their face and uttered:ย โIs this it? Are you kidding me?โ
With the popularity of Kafka, itโs no surprise that several commercial vendors have jumped on the opportunity to monetize Kafkaโs apparent lack of tooling by offering their own.ย Kafka Tool,ย Landoop,ย andย KaDeckย are some examples, but theyโre all for personal use only unless youโre willing to pay. (And itโs not to say that you shouldnโt,ย au contraire, but thatโs rather beside the point.)
Any non-trivial use in a commercial setting would be a violation of their licensing terms. Itโs one thing using them at home for tutorials or personal projects; when youโre using a commercial tool without the appropriate license, you are putting your employer at risk of litigation and playing Russian Roulette with your career.
But what about open-source?
When it comes to Kafka topic viewers and web UIs, the go-toย open-sourceย tool isย Kafdrop. With 800K Docker pulls at the time of writing, and growing at a rate of 10K pulls/day, there arenโt many Kafka tools that have enjoyed this level of adoption. And thereโs a reason behind that: Kafdrop does an amazing job of filling the apparent gaps in the observability tooling of Kafka, solving problems that the community has been pointing out for too long.
Kafdrop is an Apache 2.0 licensed project, like Apache Kafka itself. So itโs wonโt cost you a penny. If you havenโt used it yet, you probably ought to. So letโs take a deeper look.
The Kafdrop web UI project is hosted on GitHub at obsidiandynamics/kafdrop.
Thereโs a couple of options at your disposal. You could show a little bravery byย cloning the repositoryย and building from source. Itโs a Java (JDK 11) Spring Boot project, and you can build it with a single Maven command, providing you have the JDK installed.
If you want to go down this path, the repoโsย
README.md
ย file will guide you through the steps. For now, let's take the easy way โ Docker. (I sure would.)Docker images areย hosted on DockerHub. Images are tagged with theย Kafdrop release number. Theย
latest
ย tag points to the latest stable release.To launch the container in the foreground, run the following command:
docker run -it --rm -p 9000:9000 \
-e KAFKA_BROKERCONNECT=<host:port,host:port> \
obsidiandynamics/kafdrop
Theย
KAFKA_BROKERCONNECT
ย environment variable must be set to the bootstrap list of brokers.Thatโs it. We should be up and running. Once it starts, you can launch the Kafka web UI by navigating toย localhost:9000.
Note:ย The above example assumes an authenticated connection over a plaintext TCP socket. If your cluster is configured to use authentication and/or transport-level encryption, consult theย README.mdย for connection options. It's actually surprisingly easy to configure Kafdrop for a SASL/SSL locked-down cluster.
Donโt have a Kafka broker running? No worries. Just use the followingย
docker-compose.yaml
ย file to bring up aย Kafka + Kafdropย stack:version: "2"
services:
kafdrop:
image: obsidiandynamics/kafdrop
restart: "no"
ports:
- "9000:9000"
environment:
KAFKA_BROKERCONNECT: "kafka:29092"
depends_on:
- "kafka"
kafka:
image: obsidiandynamics/kafka
restart: "no"
ports:
- "2181:2181"
- "9092:9092"
environment:
KAFKA_LISTENERS: "INTERNAL://:29092,EXTERNAL://:9092"
KAFKA_ADVERTISED_LISTENERS: "INTERNAL://kafka:29092,EXTERNAL://localhost:9092"
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT"
KAFKA_INTER_BROKER_LISTENER_NAME: "INTERNAL"
Now launch the stack withย docker-compose up. Once started, browse toย localhost:9000.
Theย Cluster Overviewย screen is the landing page of the web UI.
You get to see the overall layout of the clusterโโโthe individual brokers that make it up, their addresses and some key broker statsโโโwhether they are a controller and the number of partitions each broker owns. The latter is quite importantโโโas your cluster size and the number of topics (and therefore partitions) grows, you generally want to see an approximately level distribution of partitions across the cluster.
Next is the Topics List, which in most cases is what youโre really here for. Any reasonably-sized microservices-based ecosystem might have hundreds, if not thousands of topics. As youโd expect, the list is searchable. The stats displayed alongside each topic are fairly ho-hum. The one worth noting is the under-replicated column.
Essentially, itโs telling us the number of partition replicas that have fallen behind the primary. Zero is a good figure. Anything else is indicative of either a broker or a network issue that requires immediate attention.
Note: Kafdrop is a discovery exploration tool; it is not a real-time monitoring tool. You should instrument your brokers and raise alerts when things go awry.
Click on a topic on the list to get to the Topic Overview screen.
The screen is subdivided into four sections.
On the top-left, there is a summary of the topic statsโโโa handy view, not dissimilar to what you would have seen in the cluster overview.
On the top-right, you can view the custom configuration. In the example above, the topic runs a stock-standard config, so thereโs nothing to see. Had the configuration been overridden, youโd see a set of custom values like in the example below.
The bottom-left section enumerates over the partitions. The partition indexes are linksโโโclicking through will reveal the first 100 messages in the topic.
There are several interesting parameters displayed in this section:
The consumers' section on the bottom-right lists the consumer group names as well as their aggregate lag (the sum of all individual partition lags).
Clicking on the consumer group on the Topic Overview gets you into the Consumer View. This screen provides a comprehensive breakdown of a single consumer group.
The view is sectioned by topic. For each topic, a separate table lists the underlying partitions. Against each partition, we see the committed offset, which we can compare against the first and last offsets to see how our consumer is tracking. Conveniently, Kafdrop displays the computed lag for each partition, which is aggregated at the footer of each topic table.
Note: Some amount of lag is unavoidable. For every message published, there will invariably be a quantum of time between the point of publishing and the point of consumption. In Kafka, this period is usually in the order of tens or hundreds of milliseconds, depending on both the producer and consumer client options, network configuration, broker I/O capabilities, the size of the pagecache and a myriad of other factors.
What you need to look out for is growing lagโโโsuggesting that the consumer is either unable to keep up or has stalled altogether. In the latter case, youโll also notice that the lag isnโt being depleted even when the producer is idling. This is when youโll need to ALT-TAB away from Kafdrop into your favourite debugger.
The Message View screen is the coveted topic viewer that has in all likelihood brought you here. You can get to the message view in one of two ways:
Itโs exactly what youโd expectโโโa chronologically-ordered list of messages (or records, in Kafka parlance) for a chosen partition.
Each entry conveniently displays the offset, the record key (if one is set), the timestamp of publication, and any headers that may have been appended by the producer.
Thereโs another little trick up Kafdropโs sleeve. If the message happens to be a valid JSON document, the topic viewer can nicely format it. Click on the green arrow on the left of the message to expand it.
The more you use Kafka, the more you come to discover and appreciate its true potentialโโโnot just as a versatile event streaming platform, but as a general-purpose messaging middleware that lets you assemble complex business systems from asynchronous, loosely coupled services.
Youโll invariably experience the frustrations that are to be expected from a technology that has only recently entered the mainstream, by comparison with the more mature MQ brokers of yore. Whatโs reassuring is that the Open Source community hasnโt stood still, producing an evolving ecosystem, complete with the documentation and tooling necessary for us to get on with our jobs. The least we can do in return is raise a pull request once in a while or maybe answer a StackOverflow question or three.
Was this article useful to you? Iโd love to hear your feedback, so donโt hold back! If you are interested in Kafka or event streaming, or just have any questions, follow me on Twitter.
Create your free account to unlock your custom reading experience.