What makes Kafka so Fast? A Deep Dive into Kafka Storage Internals. At most tech companies, we are constantly exposed to a variety of challenging problems, within the real-time streaming realm, and hence we make use of a distributed system like Kafka very heavily. I feel highly encouraged to have a solid understanding of the especially when we plan to use them in production, so to be able to define the use-case well with such tools, and be able to debug issues, or make decisions like: internals of such systems, when to use what! My aim with this series of blogs is to be able to comprehensively explain the fundamental details of how Kafka was designed. There are various documents and other really good resources that are available out there, but the purpose of this series is to be , and than the rest, so that the beginners who want to learn about such Distributed Systems have all the resources and prerequisite fundamental knowledge under one roof, which sometimes, in my opinion, is just referenced in original docs. Hence, with this series, I wish to , and make it more beginner-friendly to work with! more elaborative with visual representations more comprehensive avoid most of the resource branching Fundamental Pre-requisites Before diving deep into the Kafka Storage architecture, let’s discuss some important aspects of disk access patterns. Question: Are disks really slow? : It highly depends on . For around a decade, it has been observed that the throughput i.e. of disks have been increasing, and these operations are being highly optimised by Operating Systems when comparing them to . Answer how we access data the linear reads and writes disk seek In the case of the linear reads and writes, the access pattern becomes very . For example, for the specific case of reading, the OS can use certain techniques like that can prefetch a larger amount of data at once, hence increasing the performance on reads. predictable read-ahead Some research has found that in some cases can be faster than [1] disk access sequential memory access! random Question: What is a PageCache? How is it different from In-Process Caching? PageCache is a cache created of a page from a storage device like a disk. It is managed by the Operating System, implemented in , and is kept in the unused portions of the main memory i.e in the RAM. Answer: kernels This helps in quickly accessing the contents of the page, as it is now stored in memory. PageCache is transparent to all the applications being run and is . Hence, even if an application crashes, the PageCache still remains and can be used again. The operating system might also mark this as free or available memory. Whenever the system requires more memory the PageCache can be discarded or flushed to disk (if modified) and the memory can be made available to the Operating System. not bound to any application process There’s also a different kind of cache called the , which is primarily controlled by the application process. Unlike PageCache, whenever the application crashes or restarts, the in-process cache is also discarded and needs to be while booting up the application, hence might take more time to make the cache warm. in-process cache recreated How Kafka does it? Kafka heavily relies on the file system and uses a lot of storage and caching-related aspects discussed above. The developers of Kafka find that using a PageCache over an in-process cache would be much more efficient when amortised over time and under high load. It doesn’t wait for the data to actually persist to disk, leaving it to the OS to take care for, and just sends out once it writes to the , thus making it more performant. ack PageCache So, for a system that heavily relies on writing and reading data from a disk, using an in-process cache would cause of data as this data would also be written in a PageCache, hence storing everything twice. Also, since the access to data inside Kafka is sequential and not random, it’s easier for the OS to fetch and load pages from disk to PageCache using techniques. duplication read-ahead Kafka is mostly written in Java, and only with languages based on JVM. Question: What are some things to keep in mind for applications that are based on JVM? Whenever we plan to store something in memory through objects in JVM, the memory for storing them becomes or even more. Which could be mostly due to some extra bytes of housekeeping information, some bytes of the actual fields and reference to other objects, and some more extra space because JVM and/or the processor architecture will typically require object data to be padded to some specific number of bytes. Answer: almost twice Also, as the in-heap data grows, java’s can become garbage collection increasingly slow. To summarise, the following points helped Kafka Developers to make the decision for using a PageCache than an In-Process cache: Automatic access to all the freely available memory the size of a JVM based in-process cache Double Cache stays warm even if the Kafka process crashes The process of reading and writing from the cache to disk is to the operating system, hence simplifying the implementation to move data to and from the Cache and Disk delegated If your disk usage favours linear reads then read-ahead is effectively this cache with useful data on each disk read. pre-populating Hence, this makes the design very simple by just reading and writing directly to PageCache, and not needing to flush everything every time to disk, which is efficiently handled by the operating system. So, how Reliable and Durable is the Data Stored inside Kafka? Question: The astute reader might argue about the reliability and durability of data that is written to a PageCache which belongs to the kernel space of the main memory, as compared to writing to a persistent storage device like a disk. If the node on which Kafka runs crashes (assuming a single-node Kafka cluster), there is a big possibility of . As flushing to disk is handled by operating system, the un-flushed PageCache data would be lost on node crash as it resides on memory. To make Kafka more fault-tolerant it is generally advised to have enough replicas of topics stored in Kafka, running on different nodes, with which can make it very less probable to lose data. Answer: data loss Rack-Awareness With that, it is important to configure a reasonable and values. replication factor in-sync replica Unlikely though, data loss can still occur if all the nodes on which the replicas run, go down, simultaneously. Kafka developers encourage using Kafka with enough replication to avoid such a scenario, hence making data loss less likely. What Next? We saw how Kafka makes the file system look really cool; while keeping the caches warm! This would be all for the Storage and Persistence section in the Kafka Design. Please feel free to comment below and ask me anything related to what we discussed above. Further, in the next part, we would dive deeper into the and and how it is efficiently done in Kafka! data transfer networking aspects References The Pathologies of Big Data

This story contains new, firsthand information uncovered by the writer.

Kafka Storage Design - Making File Systems Cool Again!

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

10 Ways to Reduce Data Loss and Potential Downtime Of Your Database

49 Stories To Learn About Data Storage

5 Ways to Store Market Data: CSV, SQLite, Postgres, Mongo, Arctic

57 Stories To Learn About Cloud Storage

9 Data Trends You’ll See in 2023

A Beginner's Guide to Understanding SQL Window Functions and Their Capabilities

10 Ways to Reduce Data Loss and Potential Downtime Of Your Database

49 Stories To Learn About Data Storage

5 Ways to Store Market Data: CSV, SQLite, Postgres, Mongo, Arctic

57 Stories To Learn About Cloud Storage

9 Data Trends You’ll See in 2023

A Beginner's Guide to Understanding SQL Window Functions and Their Capabilities

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps