Shocker: Big Data derives its name not just from the size. The datasets of Big Data are larger, more complex information bits fetched from new data sources. These massive volumes of data can be used to address business problems more intelligently. However, traditional data processing algorithms fall flat to handle this magnitude. Deterministic data structures like HashSet do the trick with smaller amounts of data. But when we have to deal with something like streaming applications, those structures cannot process everything in one pass and assist in incremental updates. That is why we need more space-efficient and fast algorithms. Thus, probabilistic data structures are a great fit for modern Big Data applications. With that said, let’s have a look at probabilistic data structures and algorithms as well as their common use. Deterministic Vs Probabilistic Data Structure Deterministic data structures are common for a techie. Thus, we often bump into Array, List, HashTable, HashSet, etc. The latter is suitable for a wide variety of operations including insert, find, and delete (provided you have specific key values). As a result of such operations, we get deterministic or accurate results. However, probabilistic data structures work according to their name. Probabilistic data structures cannot give you a definite answer, instead, they give you a reasonable approximation of the answer and a way to approximate that estimate. How do they work? These data structures are a great fit for a large data set. The most prominent examples of operations may include identifying some unique or frequent items. To complete the operation, probabilistic data structures use hash functions to randomize items. Because they ignore collisions, they keep the size constant. Yet, this is also the reason why they cannot give you exact values. The higher the number of hash functions is, the more accurate determination you get. The main use cases of probabilistic data structures include: Huge datasets. Statistical analysis. Mining tera-bytes of data sets, and others. Examples of probabilistic data structures are as follows: Membership query (Bloom filter, Bloom count filter, private filter, cuckoo filter). Power (linear counting, probabilistic counting, LogLog, HyperLogLogLog, HyperLogLog++). Frequency (Counting sketch, Counting-minimal sketch). Similarity (LSH, MinHash, SimHash), and others. Let’s have a look at the most widely used data structures within this realm. Bloom Filter The Bloom filter is an implementation of a probability set, invented by Burton Bloom in 1970. This approximate member data query structure allows you to compactly store elements and check if a given element belongs to the set. In this case, you can get a false positive (the element is not in the set, but the data structure says it is), but not a false negative. Bloom's filter can use any memory size, predefined by the user. And the larger it is, the lower the probability of a false positive. The operation of adding new elements to the set is supported. However, you can’t delete the existing ones. Bloom filter allows you to perform three kinds of operations: add an item to the set check whether the element belongs to the set check whether the element does not belong to the set When the structure flags the element as Found/Present, there is a small chance that it’s lying. But if we’re talking about the Not Found/Not Present category, the Bloom filter boasts 100% accuracy plus space-saving perks. Its hype can be attributed to the fact that Bloom filters have this powerful combo of simplicity and multi-purpose nature. In layman’s terms, they support operations similar to the hash tables but use less space. Apache Cassandra, for example, benefits from these structures to process massive amounts of information. This storage system taps into Bloom filters to find out whether an SSTable has data for a specific partition. HyperLogLog HyperLogLog is a beautiful, yet simple algorithm for handling cardinality estimation. It excels when dealing with sets of data with a huge number of values. Let’s say, you have a massive dataset of elements with duplicate entries. The latter is taken from a set of cardinality n and you are required to find n, which is the number of unique components in the set. This can be helpful when identifying the amount of Google searches performed by end-users in a day. If you try to squeeze all the data into the memory, you’ll need storage proportionate to the number of Google searches per day. Thereby, the HyperLogLog data structure turns the data into a hash of random numbers that represent the data's cardinality, allowing it to solve the problem with as little as 1.5kB of RAM.g HyperLogLog operates by predicting an approximate count of distinct elements through a function known as APPROX_DISTINCT. Operations: Insert: add an element to the data structure Merge: generate a structure which is a combination of two structures Cardinality: calculate the total number of elements inserted Count-min Sketch The counting-minimal sketch is another efficient algorithm for counting streaming data. This data structure boils down to keeping track of the count of things. Therefore, by performing this algorithm, you can find out how many times an element is met in the set. You can also easily test if a given member has been observed before. Just like with Bloom Filters, Count-min sketch saves a lot of space by using probabilistic techniques. To implement a counting mechanism, you need to use a hash function. Overall, the count-min structure works great whenever you’re looking for just approximate counts of the most essential elements. Operations: Insert: add an element to the data structure Merge: generate a structure which is a combination of two structures Cardinality: estimate the number of times a specific estimate was inserted As for the prominent applications, AT&T leverages the structure in network switches to analyze traffic in memory-constrained environments. The structure is also implemented as part of Twitter's Algebird library. To Sum It Up Working with Big Data is a challenge itself, let alone digging up answers from it. In this case, the best you can count on is an approximate answer. Probabilistic data structures allow you to conquer the beast and give you an estimated view of some data characteristics. You lose the accuracy of results, but get an enormous amount of storage space in exchange for that.

Apache

AT&T

Google

Twitter

2022 - HackerNoon Contributor of the Year - Coding Skills

2022 - HackerNoon Contributor of the Year - Mathematics

2022 - HackerNoon Contributor of the Year - Optimization

Nominated for 2022 - HackerNoon Contributor of the Year - Mathematics

Nominated for 2022 - HackerNoon Contributor of the Year - Optimization

Nominated for 2022 - HackerNoon Contributor of the Year - Coding Skills

Too Long; Didn't Read

In your car, at home, or at work — Bosch technology shapes many areas of life.

Probabilistic Data Structures And Algorithms In Big Data

Probabilistic Data Structures And Algorithms In Big Data

Too Long; Didn't Read

Companies Mentioned

Nikita Vasilev

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

Probabilistic Data Structures And Algorithms In Big Data

Too Long; Didn't Read

Companies Mentioned

Nikita Vasilev

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

RELATED STORIES