A deep dive into a system that serves as the heart of many companies’ architecture\n----------------------------------------------------------------------------------\n\n!(https://hackernoon.com/hn-images/1*d3KMaUnrLbWPI5Pxtgfc_w.png)\n\nSome of the monoliths who use Apache Kafka\n\n### Introduction\n\nKafka is a word that gets heard a lot nowadays… A lot of leading digital companies seem to use it as well. But what is it actually?\n\nKafka was originally developed at LinkedIn in 2011 and has improved a lot since then. Nowadays it is a whole platform, allowing you to redundantly store absurd amounts of data, have a message bus with huge throughput (millions/sec) and use real-time stream processing on the data that goes through it all at once.\n\nThis is all well and great, but stripped down to its core, Kafka is a distributed, horizontally-scalable, fault-tolerant, commit log.\n\nThose were some fancy words, let’s go at them one by one and see what they mean. Afterwards, we will dive deep into how it works.\n\n#### Distributed\n\nA distributed system is one which is split into multiple running machines, all of which work together in a cluster to appear as one single node to the end user. Kafka is distributed in the sense that it stores, receives and sends messages on different nodes (called brokers).\n\n> I have a [Thorough Introduction](https://hackernoon.com/a-thorough-introduction-to-distributed-systems-3b91562c9b3c) on this as well\n\nThe benefits to this approach are high scalability and fault-tolerance.\n\n#### Horizontally-scalable\n\nLet’s define the term vertical scalability first. Say, for instance, you have a traditional database server which is starting to get overloaded. The way to get this solved is to simply increase the resources (CPU, RAM, SSD) on the server. This is called **vertical scaling** — where you add more resources to the machine. There are two big disadvantages to scaling upwards:\n\n1. There are limits defined by the hardware. You cannot scale upwards indefinitely.\n2. It usually requires downtime, something which big corporations cannot afford.\n\n**Horizontal scalability** is solving the same problem by throwing more machines at it. Adding a new machine does not require downtime nor are there any limits to the amount of machines you can have in your cluster. The catch is that not all systems support horizontal scalability, as they are not designed to work in a cluster and those that are are usually more complex to work with.\n\n!(https://hackernoon.com/hn-images/1*ulnsAJdpvet9EW4bKMK-Rw.png)\n\nHorizontal scaling becomes **much cheaper** after a certain threshold\n\n#### Fault-tolerant\n\nSomething that emerges in non-distributed systems is that they have a single point of failure (SPoF). If your single database server fails (as machines do) for whatever reason, you’re screwed.\n\nDistributed systems are designed in such a way to accommodate failures in a configurable way. In a 5-node Kafka cluster, you can have it continue working even if 2 of the nodes are down. It is worth noting that fault-tolerance is at a direct tradeoff with performance, as in the more fault-tolerant your system is, the less performant it is.\n\n#### Commit Log\n\nA commit log (also referred to as write-ahead log, transaction log) is a persistent ordered data structure which only supports appends. You cannot modify nor delete records from it. It is read from left to right and guarantees item ordering.\n\n!(https://hackernoon.com/hn-images/1*A6a1_dtaOTg-xFT3SoviUg.png)\n\nSample illustration of a commit log, taken from [here](https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying)\n\n_\\- Are you telling me that Kafka is such a simple data structure?_\n\nIn many ways, yes. This structure is at the heart of Kafka and is invaluable, as it provides ordering, which in turn provides deterministic processing. Both of which are non-trivial problems in distributed systems.\n\nKafka actually stores all of its messages to disk (more on that later) and having them ordered in the structure lets it take advantage of sequential disk reads.\n\n* Reads and writes are a constant time O(1) _(knowing the record ID)_, which compared to other structure’s O(log N) operations on disk is a huge advantage, as each disk seek is expensive.\n* Reads and writes do not affect another. Writing would not lock reading and vice-versa (as opposed to balanced trees)\n\nThese two points have huge performance benefits, since the data size is completely decoupled from performance. Kafka has the same performance whether you have 100KB or 100TB of data on your server.\n\n### How does it work?\n\nApplications (**producers**) send messages (**records**) to a Kafka node (**broker**) and said messages are processed by other applications called **consumers**. Said messages get stored in a **topic** and consumers subscribe to the topic to receive new messages.\n\n!(https://hackernoon.com/hn-images/1*brLQW_K4WrLhFKCTdmsr-w.png)\n\nAs topics can get quite big, they get split into **partitions** of a smaller size for better performance and scalability. (ex: _say you were storing user login requests, you could split them by the first character of the user’s username) \n_Kafka guarantees that all messages inside a partition are ordered in the sequence they came in. The way you distinct a specific message is through its **offset**, which you could look at as a normal array index, a sequence number which is incremented for each new message in a partition.\n\n!(https://hackernoon.com/hn-images/1*GoRlq7O8qMNui6tvnq30cg.png)\n\nKafka follows the principle of a dumb broker and smart consumer. This means that Kafka does not keep track of what records are read by the consumer and delete them but rather stores them a set amount of time (e.g one day) or until some size threshold is met. Consumers themselves poll Kafka for new messages and say what records they want to read. This allows them to increment/decrement the offset they’re at as they wish, thus being able to replay and reprocess events.\n\nIt is worth noting that consumers are actually consumer groups which have one or more consumer processes inside. In order to avoid two processes reading the same message twice, each partition is tied to only one consumer process per group.\n\n!(https://hackernoon.com/hn-images/1*BgaUBaHhE-8-vE1Omk76zg.png)\n\nRepresentation of the data flow\n\n### Persistence to Disk\n\nAs I mentioned earlier, Kafka actually stores all of its records to disk and does not keep anything in RAM. You might be wondering how this is in the slightest way a sane choice. There are numerous optimizations behind this that make it feasible:\n\n1. Kafka has a protocol which groups messages together. This allows network requests to group messages together and reduce network overhead, the server in turn persist chunk of messages in one go and consumer fetch large linear chunks at once\n2. Linear reads/writes on a disk are fast. The concept that modern disks are slow is because of numerous disk seeks, something that is not an issue in big linear operations.\n3. Said linear operations are heavily optimized by the OS, via **read-ahead** (prefetch large block multiples) and **write-behind** (group small logical writes into big physical writes) techniques.\n4. Modern OSes cache the disk in free RAM. This is called **pagecache**.\n5. Since Kafka stores messages in a standardized binary format unmodified throughout the whole flow _(producer->broker->consumer)_, it can make use of the **zero-copy** optimization. That is when the OS copies data from the pagecache directly to a socket, effectively bypassing the Kafka broker application entirely.\n\nAll of these optimizations allow Kafka to deliver messages at near network speed.\n\n### Data Distribution & Replication\n\nLet’s talk about how Kafka achieves fault-tolerance and how it distributes data between nodes.\n\n#### Data Replication\n\nPartition data is replicated across multiple brokers in order to preserve the data in case one broker dies.\n\nAt all times, one broker _“owns”_ a partition and is the node through which applications write/read from the partition. This is called a **partition leader**. It replicates the data it receives to **_N_** other brokers, called **followers**. They store the data as well and are ready to be elected as leader in case the leader node dies.\n\nThis helps you configure the guarantee that any successfully published message will not be lost. Having the option to change the replication factor lets you trade performance for stronger durability guarantees, depending on the criticality of the data.\n\n!(https://hackernoon.com/hn-images/1*08Cs4AHszdnzceAEhKhPLg.png)\n\n4 Kafka brokers with a replication factor of 3\n\nIn this way, if one leader ever fails, a follower can take his place.\n\nYou may be asking, though:\n\n_\\- How does a producer/consumer know who the leader of a partition is?_\n\nFor a producer/consumer to write/read from a partition, they need to know its leader, right? This information needs to be available from somewhere. \nKafka stores such metadata in a service called **Zookeeper**.\n\n#### **What is Zookeeper?**\n\nZookeeper is a distributed key-value store. It is highly-optimized for reads but writes are slower. It is most commonly used to store metadata and handle the mechanics of clustering (heartbeats, distributing updates/configurations, etc).\n\nIt allows clients of the service (the Kafka brokers) to subscribe and have changes sent to them once they happen. This is how brokers know when to switch partition leaders. Zookeeper is also extremely fault-tolerant and it ought to be, as Kafka heavily depends on it.\n\nIt is used for storing all sort of metadata, to mention some:\n\n* Consumer group‘s offset per partition (although modern clients store offsets in a separate Kafka topic)\n* ACL (Access Control Lists) — used for limiting access/authorization\n* Producer & Consumer Quotas —maximum message/sec boundaries\n* Partition Leaders and their health\n\n#### _How does a producer/consumer know who the leader of a partition is?_\n\nProducer and Consumers used to directly connect and talk to Zookeeper to get this (and other) information. Kafka has been moving away from this coupling and since versions 0.8 and 0.9 respectively, clients fetch metadata information from Kafka brokers directly, who themselves talk to Zookeeper.\n\n!(https://hackernoon.com/hn-images/1*tlgLYE45Q2wd5woB4uxR3g.png)\n\nMetadata Flow\n\n### Streaming\n\nIn Kafka, a stream processor is anything that takes continual streams of data from input topics, performs some processing on this input and produces a stream of data to output topics (or external services, databases, the trash bin, wherever really…)\n\nIt is possible to do simple processing directly with the producer/consumer APIs, however for more complex transformations like joining streams together, Kafka provides a integrated [Streams API](https://www.google.bg/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0ahUKEwiG1fu-oYDYAhVCCewKHeZMDfEQFggsMAA&url=https%3A%2F%2Fkafka.apache.org%2Fdocumentation%2Fstreams%2F&usg=AOvVaw30e_Zle1rMLJugJuCTx9tx) library.\n\nThis API is intended to be used within your own codebase, it is not running on a broker. It works similar to the consumer API and helps you scale out the stream processing work over multiple applications (similar to consumer groups).\n\n#### Stateless Processing\n\nA stateless processing of a stream is deterministic processing that does not depend on anything external. You know that for any given data you will always produce the same output independent of anything else. An example for that would be simple data transformation — appending something to a string `"Hello"` -> `"Hello, World!"`.\n\n!(https://hackernoon.com/hn-images/1*RfJabWrHJT3ti-diKbodnA.png)\n\n#### Stream-Table Duality\n\nIt is important to recognize that streams and tables are essentially the same. A stream can be interpreted as a table and a table can be interpreted as a stream.\n\n#### Stream as a Table\n\nA stream can be interpreted as a series of updates for data, in which the aggregate is the final result of the table. This technique is called [**_Event Sourcing_**](https://martinfowler.com/eaaDev/EventSourcing.html)**_._**\n\nIf you look at how synchronous database replication is achieved, you’ll see that it is through the so-called **streaming replication**, where each change in a table is sent to a replica server. Another example of event sourcing is Blockchain ledgers — a ledger is a series of changes as well.\n\nA Kafka stream can be interpreted in the same way —events which when accumulated form the final state. Such stream aggregations get saved in a local [RocksDB](https://github.com/facebook/rocksdb/wiki/rocksdb-basics) (by default) and are called a **KTable.**\n\n!(https://hackernoon.com/hn-images/1*r6oNBRd6Fo98gQUypQHwZQ.png)\n\nEach record increments the aggregated count\n\n#### Table as a Stream\n\nA table can be looked at as a snapshot of the latest value for each key in a stream. In the same way stream records can produce a table, table updates can produce a changelog stream.\n\n!(https://hackernoon.com/hn-images/1*yfucVQ9TFSQIeGuvaQC6eA.png)\n\nEach update produces a snapshot record in the stream\n\n#### Stateful Processing\n\nSome simple operations like `map()` or `filter()` are stateless and do not require you to keep any data regarding the processing. However, in real life, most operations you’ll do will be stateful (e.g `count()`) and as such will require you to store the currently accumulated state.\n\nThe problem with maintaining state on stream processors is that the stream processors can fail! Where would you need to keep this state in order to be fault-tolerant?\n\nA naive approach is to simply store all state in a remote database and join over the network to that store. The problem with this is that there is no locality of data and lots of network round-trips, both of which will significantly slow down your application. A more subtle but important problem is that your stream processing job’s uptime would be tightly coupled to the remote database and the job will not be self-contained _(a change in the database from another team might break your processing)_.\n\nSo what’s a better approach? \nRecall the duality of tables and streams. This allows us to convert streams into tables that are co-located with our processing. It also provides us with a mechanism for handling fault tolerance — by storing the streams in a Kafka broker.\n\nA stream processor can keep its state in a local table (e.g RocksDB), which will be updated from an input stream (after perhaps some arbitrary transformation). When the process fails, it can restore its data by replaying the stream.\n\nYou could even have a remote database be the producer of the stream, effectively broadcasting a changelog with which you rebuild the table locally.\n\n!(https://hackernoon.com/hn-images/1*u-RmxBwTlxw7ZFeiN72BtA.png)\n\nStateful processing, joining a KStream with a KTable\n\n#### KSQL\n\nNormally, you’d be forced to write your stream processing in a JVM language, as that is where the only official Kafka Streams API client is.\n\n!(https://hackernoon.com/hn-images/1*jI6I0FwDBozsc1ovM7C1hg.png)\n\nSample KSQL setup\n\n[Released in April 2018](https://www.confluent.io/blog/confluent-platform-4-1-with-production-ready-ksql-now-available/), [**KSQL**](https://www.confluent.io/blog/ksql-open-source-streaming-sql-for-apache-kafka/) is a feature which allows you to write your simple streaming jobs in a familiar SQL-like language.\n\nYou set up a KSQL server and interactively query it through a [CLI](https://en.wikipedia.org/wiki/Command-line_interface) to manage the processing. It works with the same abstractions (KStream & KTable), guarantees the same benefits of the Streams API (scalability, fault-tolerance) and greatly simplifies work with streams.\n\nThis might not sound as a lot but in practice is way more useful for testing out stuff and even allows people outside of development (e.g product owners) to play around with stream processing. [I encourage you to take a look at the quick-start video and see how simple it is](https://www.youtube.com/watch?v=A45uRzJiv7I&t=2m13s).\n\n#### Streaming alternatives\n\nKafka streams is a perfect mix of power and simplicity. It arguably has the best capabilities for stream jobs on the market and it integrates with Kafka way easier than other stream processing alternatives ([**Storm**](https://storm.apache.org/), [**Samza**](https://samza.apache.org/), [**Spark**](https://spark.apache.org/)**,** [**Wallaroo**](https://github.com/WallarooLabs/wallaroo)).\n\nThe problem with most other stream processing frameworks is that they are complex to work with and deploy. A batch processing framework like Spark needs to:\n\n* Control a large number of jobs over a pool of machines and efficiently distribute them across the cluster.\n* To achieve this it has to dynamically package up your code and physically deploy it to the nodes that will execute it. (along with configuration, libraries, etc.)\n\nUnfortunately tackling these problems makes the frameworks pretty invasive. They want to control many aspects of how code is deployed, configured, monitored, and packaged.\n\nKafka Streams let you roll out your own deployment strategy when you need it, be it [Kubernetes](https://kubernetes.io/), [Mesos](https://mesos.apache.org/), [Nomad](https://www.nomadproject.io/), [Docker Swarm](https://github.com/docker/swarm) or others.\n\nThe underlying motivation of Kafka Streams is to enable all your applications to do stream processing without the operational complexity of running and maintaining yet another cluster. The only potential downside is that it is tightly coupled with Kafka, but in the modern world where most if not all real-time processing is powered by Kafka that may not be a big disadvantage.\n\n### When would you use Kafka?\n\nAs we already covered, Kafka allows you to have a huge amount of messages go through a centralized medium and store them without worrying about things like performance or data loss.\n\nThis means it is perfect for use as the heart of your system’s architecture, acting as a centralized medium that connects different applications. Kafka can be the center piece of an event-driven architecture and allows you to truly decouple applications from one another.\n\n!(https://hackernoon.com/hn-images/1*fAF4B-rH1kdn5c90CpHKhg.png)\n\nKafka allows you to easily decouple communication between different (micro)services. With the Streams API, it is now easier than ever to write business logic which enriches Kafka topic data for service consumption. The possibilities are huge and I urge you to explore how companies are using Kafka.\n\n### Why has it seen so much use?\n\nHigh performance, availability and scalability alone are not strong enough reasons for a company to adopt a new technology. There are other systems which boast similar properties, but none have become so widely used. Why is that?\n\nThe reason Kafka has grown in popularity (and continues to do so) is one key thing — businesses nowadays benefit greatly from event-driven architecture. This is because the world has changed — an enormous (and ever-growing) amount of data is being produced and consumed by many different services (Internet of Things, Machine Learning, Mobile, Microservices).\n\nA single real-time event broadcasting platform with durable storage is the cleanest way to achieve such an architecture. Imagine what kind of a mess it would be if streaming data to/from each service used a different technology specifically catered to it.\n\nThis, paired with the fact that Kafka provides the appropriate characteristics for such a generalized system _(durable storage, event broadcast, table and stream primitives, abstraction via KSQL, open-source, actively developed)_ make it an obvious choice for companies.\n\n### Summary\n\nApache Kafka is a distributed streaming platform capable of handling trillions of events a day. Kafka provides low-latency, high-throughput, fault-tolerant publish and subscribe pipelines and is able to process streams of events.\n\nWe went over its basic semantics (producer, broker, consumer, topic), learned about some of its optimizations (pagecache), learned how it’s fault-tolerant by replicating data and were introduced to its ever-growing powerful streaming abilities.\n\nKafka has seen large adoption at thousands of companies worldwide, including a third of the Fortune 500. With the active development of Kafka and the recently released first [major version 1.0](https://www.confluent.io/blog/apache-kafka-goes-1-0/) _(1st November, 2017),_ there are predictions that this Streaming Platform is going to be as big and central of a data platform as relational databases are.\n\nI hope that this introduction helped familiarize you with Apache Kafka and its potential.\n\n### Further Reading Resources & Things I did not mention\n\nThe rabbit hole goes deeper than this article was able to cover. Here are some features I did not get the chance to mention but are nevertheless important to know:\n\n[Controller Broker, in-sync replicas](https://hackernoon.com/apache-kafkas-distributed-system-firefighter-the-controller-broker-1afca1eae302) — The way in which Kafka keeps the cluster healthy and ensures adequate consistency and durability.\n\n[Connector API](https://www.confluent.io/blog/announcing-kafka-connect-building-large-scale-low-latency-data-pipelines/) — API helping you connect various services to Kafka as a source or sink (PostgreSQL, Redis, ElasticSearch)\n\n[Log Compaction](http://cloudurable.com/blog/kafka-architecture-log-compaction/index.html) — An optimization which reduces log size. Extremely useful in changelog streams\n\n[Exactly-once Message Semantics](https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/) — Guarantee that messages are received exactly once. This is a big deal, as it is difficult to achieve.\n\n#### Resources\n\n[Apache Kafka’s Distributed Systems Firefighter — the Controller Broker](https://hackernoon.com/apache-kafkas-distributed-system-firefighter-the-controller-broker-1afca1eae302) — Another blog post of mine where I dive into how coordination between the broker works and much more.\n\n[Confluent Blog](https://www.confluent.io/blog/) — a wealth of information regarding Apache Kafka\n\n[Kafka Documentation](https://kafka.apache.org/documentation/) — Great, extensive, high-quality documentation\n\n[Kafka Summit 2017 videos](https://www.confluent.io/kafka-summit-sf17/resource/)\n\nThank you for taking the time to read this.\n\nIf you think this information was helpful, please consider giving it a hefty amount of claps to increase its visibility and help new people find it!\n\n_~Stanislav Kozlovski_\n\n#### Update\n\nThis article opened the door for me to join [Confluent](http://confluent.io). I am immensely grateful for the opportunity they have given me — I currently work on Kafka itself, which is beyond awesome! Confluent is a big data company founded by the creators of Apache Kafka themselves! We currently work on the whole Kafka ecosystem, including a managed Kafka-as-a-service cloud offering.\n\nWe are hiring for a lot of positions (especially SRE/Software Engineers) in Europe and the USA! If you are interested in working on Kafka itself, looking for new opportunities or just plain curious — make sure to message me on [Twitter](https://twitter.com/StanKozlovski) and I will share all the great perks that come from working in a bay area company.