Our mission at Column Tax is to democratize access to financial help and meaningfully improve people’s lives by providing game-changing tax solutions

Column Tax

columntax

Strategies for Implementing Columnar Compression in Large PostgreSQL Databases

বড় পোস্টগ্রেএসকিউএল ডেটাবেসে কলামার কম্প্রেশন বাস্তবায়নের কৌশল

Strategien zur Implementierung der Spaltenkomprimierung in großen PostgreSQL-Datenbanken

Estrategias para implementar la compresión de columnas en grandes bases de datos PostgreSQL

Stratégies d'implémentation de la compression en colonnes dans les grandes bases de données PostgreSQL

बड़े PostgreSQL डेटाबेस में कॉलमर कम्प्रेशन लागू करने की रणनीतियाँ

大規模な PostgreSQL データベースに列圧縮を実装するための戦略

대규모 PostgreSQL 데이터베이스에서 열 형식 압축을 구현하기 위한 전략

Estratégias para implementar compactação colunar em grandes bancos de dados PostgreSQL

Стратегии реализации столбчатого сжатия в больших базах данных PostgreSQL

Büyük PostgreSQL Veritabanlarında Sütun Sıkıştırmasını Uygulama Stratejileri

Các chiến lược thực hiện nén cột trong cơ sở dữ liệu PostgreSQL lớn

在大型 PostgreSQL 数据库中实现列压缩的策略

Steganography: How to Hide Text in Images Using JavaScript

স্টেগানোগ্রাফি: জাভাস্ক্রিপ্ট ব্যবহার করে চিত্রগুলিতে পাঠ্য কীভাবে লুকাবেন

Steganographie: So verbergen Sie Text in Bildern mit JavaScript

Esteganografía: cómo ocultar texto en imágenes usando JavaScript

Stéganographie : comment masquer le texte dans les images à l'aide de JavaScript

स्टेग्नोग्राफ़ी: जावास्क्रिप्ट का उपयोग करके छवियों में टेक्स्ट कैसे छिपाएं

ステガノグラフィー: JavaScript を使用して画像内のテキストを隠す方法

스테가노그래피: JavaScript를 사용하여 이미지에서 텍스트를 숨기는 방법

Esteganografia: como ocultar texto em imagens usando JavaScript

Стеганография: как скрыть текст на изображениях с помощью JavaScript

Steganografi: JavaScript Kullanarak Görsellerdeki Metin Nasıl Gizlenir?

Steganography: Cách ẩn văn bản trong hình ảnh bằng JavaScript

隐写术：如何使用 JavaScript 隐藏图像中的文本

Apache Doris for Log and Time Series Data Analysis in NetEase: Why Not Elasticsearch and InfluxDB?

Authors:
(1) Sasun Hambardzumyan, Activeloop, Mountain View, CA, USA;
(2) Abhinav Tuli, Activeloop, Mountain View, CA, USA;
(3) Levon Ghukasyan, Activeloop, Mountain View, CA, USA;
(4) Fariz Rahman, Activeloop, Mountain View, CA, USA;.
(5) Hrant Topchyan, Activeloop, Mountain View, CA, USA;
(6) David Isayan, Activeloop, Mountain View, CA, USA;
(7) Mark McQuade, Activeloop, Mountain View, CA, USA;
(8) Mikayel Harutyunyan, Activeloop, Mountain View, CA, USA;
(9) Tatevik Hakobyan, Activeloop, Mountain View, CA, USA;
(10) Ivo Stranic, Activeloop, Mountain View, CA, USA;
(11) Davit Buniatyan, Activeloop, Mountain View, CA, USA. Table of Links Abstract and Intro
Current Challenges
Tensor Storage Format
Deep Lake System Overview
Machine Learning Use Cases
Performance Benchmarks
Discussion and Limitations
Related Work
Conclusions, Acknowledgement, and References 7. DISCUSSION AND LIMITATIONS Deep Lake’s primary use cases include (a) Deep Learning Model Training, (b) Data Lineage and Version Control, (c) Data Querying, and Analytics, (d) Data Inspection and Quality Control. We took NumPy [55] arrays as a fundamental block and implemented version control, streaming dataloaders, visualization engine from scratch. 7.1 Format Design Space The Tensor Storage Format (TSF) is a binary file format designed specifically for storing tensors, which are multi-dimensional arrays of numerical values used in many machine learning and deep learning algorithms. The TSF format is designed to be efficient and compact, allowing for fast and efficient storage and access of tensor data. One key advantage of the TSF format is that it supports a wide range of tensor data types, including dynamically shaped tensors. In comparison, the Parquet [79] and Arrow [13] formats are columnar file formats that are designed for storing and processing large analytical datasets. Unlike TSF, which is specifically designed for tensor data, Parquet and Arrow are optimized for efficient storage and querying of analytical workloads on tabular and time-series data. They use columnar storage and compression techniques to minimize storage space and improve performance, making them suitable for big data applications. However, TSF has some advantages over Parquet and Arrow when it comes to tensor data. TSF can support tensor operations and efficient streaming to deep learning frameworks. Other tensor formats [18, 52, 23, 57] are efficient for massively massively parallelizable workloads as they don’t require coordination across chunks. Tensor Storage Format key trade-off is enabling to store dynamically shape arrays inside a tensor without padding memory footprint. For example, in computer vision it is very common to store multiple images with different shapes or videos have dynamic length. To support the flexibility, minor overhead is introduced in the form of previously discussed chunk encoder that in practice we haven’t observed impact on production workloads. 7.2 Dataloader Deep Lake achieves state-of-the-art results in local and remote settings, as seen in benchmarks for iterating on large images Fig. 7. Primarily, it has been faster than FFCV [39], which claimed a reduction of ImageNet model training up to 98 cents per model training. Furthermore, Deep Lake achieves similar ingestion performance to WebDataset [19]. Deep Lake significantly outperforms on larger images. Parquet is optimized for small cells and analytical workloads, while Deep Lake is optimized for large, dynamically shaped tensorial data. Compared to other data lake solutions, its minimal python package design enables Deep Lake to be easily integrated into large-scale distributed training or inference workloads. 7.3 Future work The current implementation of Deep Lake has opportunities for further improvement. Firstly, the storage format does not support custom ordering for an even more efficient storage layout required for vector search or key-value indexing. Secondly, Deep Lake implements branch-based locks for concurrent access. Similar to Delta ACID transaction model [27], Deep Lake can be extended to highlyperformant parallel workloads. Thirdly, the current implementation of TQL only supports a subset of SQL operations (i.e., does not support operations such as join). Further work will focus on making it SQL-complete, extending to more numeric operations, running federated queries in external data sources and benchmarking against SQL engines. This paper is available on arxiv under CC 4.0 license. Authors: (1) Sasun Hambardzumyan, Activeloop, Mountain View, CA, USA; (2) Abhinav Tuli, Activeloop, Mountain View, CA, USA; (3) Levon Ghukasyan, Activeloop, Mountain View, CA, USA; (4) Fariz Rahman, Activeloop, Mountain View, CA, USA;. (5) Hrant Topchyan, Activeloop, Mountain View, CA, USA; (6) David Isayan, Activeloop, Mountain View, CA, USA; (7) Mark McQuade, Activeloop, Mountain View, CA, USA; (8) Mikayel Harutyunyan, Activeloop, Mountain View, CA, USA; (9) Tatevik Hakobyan, Activeloop, Mountain View, CA, USA; (10) Ivo Stranic, Activeloop, Mountain View, CA, USA; (11) Davit Buniatyan, Activeloop, Mountain View, CA, USA. Authors: Authors: (1) Sasun Hambardzumyan, Activeloop, Mountain View, CA, USA; (2) Abhinav Tuli, Activeloop, Mountain View, CA, USA; (3) Levon Ghukasyan, Activeloop, Mountain View, CA, USA; (4) Fariz Rahman, Activeloop, Mountain View, CA, USA;. (5) Hrant Topchyan, Activeloop, Mountain View, CA, USA; (6) David Isayan, Activeloop, Mountain View, CA, USA; (7) Mark McQuade, Activeloop, Mountain View, CA, USA; (8) Mikayel Harutyunyan, Activeloop, Mountain View, CA, USA; (9) Tatevik Hakobyan, Activeloop, Mountain View, CA, USA; (10) Ivo Stranic, Activeloop, Mountain View, CA, USA; (11) Davit Buniatyan, Activeloop, Mountain View, CA, USA. Table of Links Abstract and Intro Current Challenges Tensor Storage Format Deep Lake System Overview Machine Learning Use Cases Performance Benchmarks Discussion and Limitations Related Work Conclusions, Acknowledgement, and References Abstract and Intro Abstract and Intro Current Challenges Current Challenges Tensor Storage Format Tensor Storage Format Deep Lake System Overview Deep Lake System Overview Machine Learning Use Cases Machine Learning Use Cases Performance Benchmarks Performance Benchmarks Discussion and Limitations Discussion and Limitations Related Work Related Work Conclusions, Acknowledgement, and References Conclusions, Acknowledgement, and References 7. DISCUSSION AND LIMITATIONS Deep Lake’s primary use cases include (a) Deep Learning Model Training, (b) Data Lineage and Version Control, (c) Data Querying, and Analytics, (d) Data Inspection and Quality Control. We took NumPy [55] arrays as a fundamental block and implemented version control, streaming dataloaders, visualization engine from scratch. 7.1 Format Design Space The Tensor Storage Format (TSF) is a binary file format designed specifically for storing tensors, which are multi-dimensional arrays of numerical values used in many machine learning and deep learning algorithms. The TSF format is designed to be efficient and compact, allowing for fast and efficient storage and access of tensor data. One key advantage of the TSF format is that it supports a wide range of tensor data types, including dynamically shaped tensors. In comparison, the Parquet [79] and Arrow [13] formats are columnar file formats that are designed for storing and processing large analytical datasets. Unlike TSF, which is specifically designed for tensor data, Parquet and Arrow are optimized for efficient storage and querying of analytical workloads on tabular and time-series data. They use columnar storage and compression techniques to minimize storage space and improve performance, making them suitable for big data applications. However, TSF has some advantages over Parquet and Arrow when it comes to tensor data. TSF can support tensor operations and efficient streaming to deep learning frameworks. Other tensor formats [18, 52, 23, 57] are efficient for massively massively parallelizable workloads as they don’t require coordination across chunks. Tensor Storage Format key trade-off is enabling to store dynamically shape arrays inside a tensor without padding memory footprint. For example, in computer vision it is very common to store multiple images with different shapes or videos have dynamic length. To support the flexibility, minor overhead is introduced in the form of previously discussed chunk encoder that in practice we haven’t observed impact on production workloads. 7.2 Dataloader Deep Lake achieves state-of-the-art results in local and remote settings, as seen in benchmarks for iterating on large images Fig. 7. Primarily, it has been faster than FFCV [39], which claimed a reduction of ImageNet model training up to 98 cents per model training. Furthermore, Deep Lake achieves similar ingestion performance to WebDataset [19]. Deep Lake significantly outperforms on larger images. Parquet is optimized for small cells and analytical workloads, while Deep Lake is optimized for large, dynamically shaped tensorial data. Compared to other data lake solutions, its minimal python package design enables Deep Lake to be easily integrated into large-scale distributed training or inference workloads. 7.3 Future work The current implementation of Deep Lake has opportunities for further improvement. Firstly, the storage format does not support custom ordering for an even more efficient storage layout required for vector search or key-value indexing. Secondly, Deep Lake implements branch-based locks for concurrent access. Similar to Delta ACID transaction model [27], Deep Lake can be extended to highlyperformant parallel workloads. Thirdly, the current implementation of TQL only supports a subset of SQL operations (i.e., does not support operations such as join). Further work will focus on making it SQL-complete, extending to more numeric operations, running federated queries in external data sources and benchmarking against SQL engines. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Deep Lake, a Lakehouse for Deep Learning: Discussion and Limitations

গভীর লেক, গভীর শিক্ষার জন্য একটি লেকহাউস: আলোচনা এবং সীমাবদ্ধতা

Deep Lake, ein Lakehouse für Deep Learning: Diskussion und Einschränkungen

Deep Lake, una casa de lago para el aprendizaje profundo: discusión y limitaciones

Deep Lake, un Lakehouse pour le Deep Learning : discussion et limites

डीप लेक, डीप लर्निंग के लिए एक लेकहाउस: चर्चा और सीमाएँ

ディープラーニングのためのレイクハウス「Deep Lake」：議論と限界

딥 러닝을 위한 레이크하우스인 Deep Lake: 논의와 한계

Deep Lake, um Lakehouse para Deep Learning: Discussão e Limitações

Deep Lake, домик у озера для глубокого обучения: обсуждение и ограничения

Deep Lake, Derin Öğrenme için Bir Göl Evi: Tartışma ve Sınırlamalar

Deep Lake, Lakehouse cho Deep Learning: Thảo luận và hạn chế

Deep Lake，深度学习的 Lakehouse：讨论和局限性

Authors:
(1) Sasun Hambardzumyan, Activeloop, Mountain View, CA, USA;
(2) Abhinav Tuli, Activeloop, Mountain View, CA, USA;
(3) Levon Ghukasyan, Activeloop, Mountain View, CA, USA;
(4) Fariz Rahman, Activeloop, Mountain View, CA, USA;.
(5) Hrant Topchyan, Activeloop, Mountain View, CA, USA;
(6) David Isayan, Activeloop, Mountain View, CA, USA;
(7) Mark McQuade, Activeloop, Mountain View, CA, USA;
(8) Mikayel Harutyunyan, Activeloop, Mountain View, CA, USA;
(9) Tatevik Hakobyan, Activeloop, Mountain View, CA, USA;
(10) Ivo Stranic, Activeloop, Mountain View, CA, USA;
(11) Davit Buniatyan, Activeloop, Mountain View, CA, USA. Table of Links Abstract and Intro
Current Challenges
Tensor Storage Format
Deep Lake System Overview
Machine Learning Use Cases
Performance Benchmarks
Discussion and Limitations
Related Work
Conclusions, Acknowledgement, and References 2. CURRENT CHALLENGES In this section, we discuss the current and historical challenges of unstructured or complex data management. 2.1 Complex Data Types in a Databases It is generally not recommended to store binary data, such as images, directly in a database. This is because databases are not optimized for storing and serving large files and can cause performance issues. In addition, binary data does not fit well into a database’s structured format, making it difficult to query and manipulate. This can lead to slow load times for users. Databases are typically more expensive to operate and maintain than other types of storage, such as file systems or cloud storage services. Therefore, storing large amounts of binary data in a database can be more costly than other storage solutions. 2.2 Complex Data Along with Tabular Formats Increases in large-scale analytical and BI workloads motivated the development of compressed structured formats like Parquet, ORC, Avro, or transient in-memory formats like Arrow [79, 6, 20, 13]. As tabular formats gained adoption, attempts to extend those formats, such as Petastorm [18] or Feather [7] for deep learning, have emerged. To the best of our knowledge, these formats have yet to gain wide adoption. This approach primarily benefits from native integrations with Modern Data Stack (MDS). However, as discussed previously, upstream tools require fundamental modifications to adapt to deep learning applications. 2.3 Object Storage for Deep Learning The current cloud-native choice for storing large unstructured datasets is object storage such as AWS S3 [1], Google Cloud Storage (GCS) [3], or MinIO [17]. Object storage does offer three main benefits over distributed network file systems. They are (a) cost-efficient, (b) scalable, and (c) serve as a format-agnostic repository. However, cloud storages are not without drawbacks. Firstly, they introduce significant latency overhead, especially when iterating over many small files such as text or JSON. Next, unstructured data ingestion without metadata control can produce "data swamps". Furthermore, object storage has built-in version control; it is rarely used in data science workflows. Lastly, data on object storage gets copied to a virtual machine before training, thus resulting in storage overhead and additional costs. 2.4 Second Generation of Data Lakes The second-generation data lakes led by Delta, Iceberg, Hudi [27, 15, 10] extend object storage by managing tabular format files with the following primary properties. (1) Update operations: inserting or deleting a row on top of a tabular format file. (2) Streaming: downstream data ingestion with ACID properties and upstream integration with query engine exposing SQL interface. (3) Schema evolution: evolving columnar structure while preserving backward compatibility. (4) Time travel and audit log trailing: preserving historical state with rollback property where queries can be reproducible. Also, support for row-level control on data lineage. (5) Layout optimization: Built-in feature to optimize file sizes and data compaction with custom ordering support. Significantly speeds up querying. However, second-generation data lakes are still bound by the limitations of the inherent data formats to be used in deep learning, as previously discussed in section 2.2. Hence in this paper, we extend the second generation of data lake capabilities for deep learning use cases by rethinking the format and upstream features, including querying, visualization, and native integration to deep learning frameworks to complete the ML lifecycle as shown in Fig. 2. This paper is available on arxiv under CC 4.0 license. Authors: (1) Sasun Hambardzumyan, Activeloop, Mountain View, CA, USA; (2) Abhinav Tuli, Activeloop, Mountain View, CA, USA; (3) Levon Ghukasyan, Activeloop, Mountain View, CA, USA; (4) Fariz Rahman, Activeloop, Mountain View, CA, USA;. (5) Hrant Topchyan, Activeloop, Mountain View, CA, USA; (6) David Isayan, Activeloop, Mountain View, CA, USA; (7) Mark McQuade, Activeloop, Mountain View, CA, USA; (8) Mikayel Harutyunyan, Activeloop, Mountain View, CA, USA; (9) Tatevik Hakobyan, Activeloop, Mountain View, CA, USA; (10) Ivo Stranic, Activeloop, Mountain View, CA, USA; (11) Davit Buniatyan, Activeloop, Mountain View, CA, USA. Authors: Authors: (1) Sasun Hambardzumyan, Activeloop, Mountain View, CA, USA; (2) Abhinav Tuli, Activeloop, Mountain View, CA, USA; (3) Levon Ghukasyan, Activeloop, Mountain View, CA, USA; (4) Fariz Rahman, Activeloop, Mountain View, CA, USA;. (5) Hrant Topchyan, Activeloop, Mountain View, CA, USA; (6) David Isayan, Activeloop, Mountain View, CA, USA; (7) Mark McQuade, Activeloop, Mountain View, CA, USA; (8) Mikayel Harutyunyan, Activeloop, Mountain View, CA, USA; (9) Tatevik Hakobyan, Activeloop, Mountain View, CA, USA; (10) Ivo Stranic, Activeloop, Mountain View, CA, USA; (11) Davit Buniatyan, Activeloop, Mountain View, CA, USA. Table of Links Abstract and Intro Current Challenges Tensor Storage Format Deep Lake System Overview Machine Learning Use Cases Performance Benchmarks Discussion and Limitations Related Work Conclusions, Acknowledgement, and References Abstract and Intro Abstract and Intro Current Challenges Current Challenges Tensor Storage Format Tensor Storage Format Deep Lake System Overview Deep Lake System Overview Machine Learning Use Cases Machine Learning Use Cases Performance Benchmarks Performance Benchmarks Discussion and Limitations Discussion and Limitations Related Work Related Work Conclusions, Acknowledgement, and References Conclusions, Acknowledgement, and References 2. CURRENT CHALLENGES In this section, we discuss the current and historical challenges of unstructured or complex data management. 2.1 Complex Data Types in a Databases It is generally not recommended to store binary data, such as images, directly in a database. This is because databases are not optimized for storing and serving large files and can cause performance issues. In addition, binary data does not fit well into a database’s structured format, making it difficult to query and manipulate. This can lead to slow load times for users. Databases are typically more expensive to operate and maintain than other types of storage, such as file systems or cloud storage services. Therefore, storing large amounts of binary data in a database can be more costly than other storage solutions. 2.2 Complex Data Along with Tabular Formats Increases in large-scale analytical and BI workloads motivated the development of compressed structured formats like Parquet, ORC, Avro, or transient in-memory formats like Arrow [79, 6, 20, 13]. As tabular formats gained adoption, attempts to extend those formats, such as Petastorm [18] or Feather [7] for deep learning, have emerged. To the best of our knowledge, these formats have yet to gain wide adoption. This approach primarily benefits from native integrations with Modern Data Stack (MDS). However, as discussed previously, upstream tools require fundamental modifications to adapt to deep learning applications. 2.3 Object Storage for Deep Learning The current cloud-native choice for storing large unstructured datasets is object storage such as AWS S3 [1], Google Cloud Storage (GCS) [3], or MinIO [17]. Object storage does offer three main benefits over distributed network file systems. They are (a) cost-efficient, (b) scalable, and (c) serve as a format-agnostic repository. However, cloud storages are not without drawbacks. Firstly, they introduce significant latency overhead, especially when iterating over many small files such as text or JSON. Next, unstructured data ingestion without metadata control can produce "data swamps". Furthermore, object storage has built-in version control; it is rarely used in data science workflows. Lastly, data on object storage gets copied to a virtual machine before training, thus resulting in storage overhead and additional costs. 2.4 Second Generation of Data Lakes The second-generation data lakes led by Delta, Iceberg, Hudi [27, 15, 10] extend object storage by managing tabular format files with the following primary properties. (1) Update operations: inserting or deleting a row on top of a tabular format file. Update operations: (2) Streaming : downstream data ingestion with ACID properties and upstream integration with query engine exposing SQL interface. Streaming (3) Schema evolution: evolving columnar structure while preserving backward compatibility. Schema evolution: (4) Time travel and audit log trailing: preserving historical state with rollback property where queries can be reproducible. Also, support for row-level control on data lineage. Time travel and audit log trailing: (5) Layout optimization: Built-in feature to optimize file sizes and data compaction with custom ordering support. Significantly speeds up querying. Layout optimization: However, second-generation data lakes are still bound by the limitations of the inherent data formats to be used in deep learning, as previously discussed in section 2.2. Hence in this paper, we extend the second generation of data lake capabilities for deep learning use cases by rethinking the format and upstream features, including querying, visualization, and native integration to deep learning frameworks to complete the ML lifecycle as shown in Fig. 2. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Deep Lake, a Lakehouse for Deep Learning: Current Challenges

গভীর লেক, গভীর শিক্ষার জন্য একটি লেকহাউস: বর্তমান চ্যালেঞ্জ

Deep Lake, ein Lakehouse für Deep Learning: Aktuelle Herausforderungen

Deep Lake, un lago para el aprendizaje profundo: desafíos actuales

Deep Lake, un Lakehouse pour le Deep Learning : les défis actuels

डीप लेक, डीप लर्निंग के लिए एक लेकहाउस: वर्तमान चुनौतियाँ

ディープラーニングのためのレイクハウス「Deep Lake」：現在の課題

딥 러닝을 위한 레이크하우스인 Deep Lake: 현재의 과제

Deep Lake, um Lakehouse para Deep Learning: desafios atuais

Deep Lake, домик у озера для глубокого обучения: текущие проблемы

Deep Lake, Derin Öğrenme için Bir Göl Evi: Güncel Zorluklar

Deep Lake, Lakehouse cho Deep Learning: Những thách thức hiện tại

Deep Lake，深度学习的 Lakehouse：当前的挑战

Authors:
(1) Sasun Hambardzumyan, Activeloop, Mountain View, CA, USA;
(2) Abhinav Tuli, Activeloop, Mountain View, CA, USA;
(3) Levon Ghukasyan, Activeloop, Mountain View, CA, USA;
(4) Fariz Rahman, Activeloop, Mountain View, CA, USA;.
(5) Hrant Topchyan, Activeloop, Mountain View, CA, USA;
(6) David Isayan, Activeloop, Mountain View, CA, USA;
(7) Mark McQuade, Activeloop, Mountain View, CA, USA;
(8) Mikayel Harutyunyan, Activeloop, Mountain View, CA, USA;
(9) Tatevik Hakobyan, Activeloop, Mountain View, CA, USA;
(10) Ivo Stranic, Activeloop, Mountain View, CA, USA;
(11) Davit Buniatyan, Activeloop, Mountain View, CA, USA. Table of Links Abstract and Intro
Current Challenges
Tensor Storage Format
Deep Lake System Overview
Machine Learning Use Cases
Performance Benchmarks
Discussion and Limitations
Related Work
Conclusions, Acknowledgement, and References 3. TENSOR STORAGE FORMAT Deep Lake datasets follow columnar storage architecture, with tensors as columns, as shown in Fig. 3. Each tensor is a collection of chunks - binary blobs that contain the data samples. An index map associated with each tensor helps find the right chunk and index of the sample within that chunk for a given sample index. 3.1 Dataset A sample in a dataset represents a single row indexed across parallel tensors. As opposed to a document storage format, sample elements are logically independent, which enables partial access to samples for running performant queries or streaming selected tensors over the network to the GPU training instances. Multiple tensors can be grouped. Groups implement syntactic nesting and define how tensors are related to each other. Syntactic nesting avoids the format complication for hierarchical memory layout. Changes to the dataset’s schema are also tracked over time with version control, similar to dataset content changes. 3.2 Tensors Tensors are typed and can be appended or modified in-place. Default access to an index or a set of indices returns the data as NumPy arrays [55]. Instead of storing 1-D data as seen in Parquet [79] or series in Arrow [13], tensors can accommodate n-dimensional data, where typically the first dimension corresponds to the index or batch dimension. Tensors can contain dynamically shaped arrays, also called ragged tensors, as opposed to other statically chunked array formats such as Zarr [52]. 3.3 Types Htype defines the expectations on samples in a tensor such as data type (dtype as seen in NumPy [55]), shape, number of dimensions, or compression. Typed tensors make interacting with deep learning frameworks straightforward and enable sanity checks and efficient memory layout. By inheriting from a generic tensor htype, we can construct types such as image, video, audio, bbox, dicom, and others. For example, a tensor with image htype would expect samples being appended to it to have dtype as uint8 and shape length 3 (i.e. width, height and number of channels). We further expand on the notion of htypes allowing for meta types that support storing image sequences in tensors (sequence[image]), referencing to remotely stored images, while maintaining the regular behavior of a image tensor (link[image]), or even possible cross-format support. 3.4 Memory Layout A Deep Lake dataset contains a provenance file in JSON format and folders per tensor. A tensor contains chunks, chunk encoder, tile encoder, and tensor metadata. Tensors can be optionally hidden. For instance, hidden tensors can be used to maintain down-sampled versions of images or preserve shape information for fast queries. Tensors are stored in chunks at the storage level. While statically (inferred) shaped chunking avoids maintaining a chunk map table, it introduces significant user overhead during the specification of the tensor, custom compression usage limitations, underutilized storage for dynamically shaped tensors, and post-processing inefficiencies. Deep Lake chunks are constructed based on the lower and upper bound of the chunk size to fit a limited number of samples. This comes with a trade-off of having a compressed index map that preserves the sample index to chunk id mapping per tensor while enabling chunk sizes in the range optimal for streaming while accommodating mixed-shape samples. One could consider the approach taken in this paper as an optimized trade-off between file system page map and compute-defined map-less array storage system. For practical reasons, a single chunk encoder can be scaled to billions of images while maintaining a 150MB chunk encoder per 1PB tensor data. Further scaling can be introduced by sharding the chunk encoder. Chunks contain header information such as byte ranges, shapes of the samples, and the sample data itself. If a sample is larger than the upper bound chunk size, which is the case for large aerial or microscopy images, the sample is tiled into chunks across spatial dimensions. The only exception to tiling is videos. Videos are preserved due to efficient frame mapping to indices, key-frame-only decompression, and range-based requests while streaming. 3.5 Access Patterns The tensor storage format is optimized for deep learning training and inference, including sequential and random access. Sequential access is used for running scan queries, transforming tensors into other tensors, or running inference. Random access use cases include multiple annotators writing labels to the same image or models storing back predictions along with the dataset. While the strict mode is disabled, out-of-the-bounds indices of a tensor can be assigned, thus accommodating sparse tensors. However, random assignment over time will produce inefficiently stored data chunks. To fix the data layout, we implement an on-the-fly re-chunking algorithm to optimize the data layout. One of the key access patterns of Deep Lake is shuffled stream access for training machine learning models. It requires random or custom order access while streaming chunks into the training process. This is achieved by involving range-based requests to access sub-elements inside chunks, running complex queries before training to determine the order, and maintaining a buffer cache of fetched and unutilized data. This avoids having a separate compute cluster for running shuffling algorithm [50]. Each tensor has its own chunks, and the default chunk size is 8MB. A single chunk consists of data from multiple indices when the individual data points (image, label, annotation, etc.) are smaller than the chunk size. Conversely, when individual data points are larger than the chunk size, the data is split among multiple chunks (tiling). Exceptions to chunking logic are video data. Deep Lake format is optimized for maximizing throughput to GPU processing. It includes CPU pre-fetching, decompression or decoding, transformations, and GPU memory transfer in a deep learning framework’s expected layout. 3.6 Storage Providers Deep Lake can be plugged into any storage provider, including object storages such as AWS S3 [1], Google Cloud Storage (GCS) [3], POSIX compatible file systems, or local in-memory storage. Moreover, it constructs memory caching by chaining various storage providers together, for instance - the Least Recently Used (LRU) cache of remote S3 storage with local in-memory data. This paper is available on arxiv under CC 4.0 license. Authors: (1) Sasun Hambardzumyan, Activeloop, Mountain View, CA, USA; (2) Abhinav Tuli, Activeloop, Mountain View, CA, USA; (3) Levon Ghukasyan, Activeloop, Mountain View, CA, USA; (4) Fariz Rahman, Activeloop, Mountain View, CA, USA;. (5) Hrant Topchyan, Activeloop, Mountain View, CA, USA; (6) David Isayan, Activeloop, Mountain View, CA, USA; (7) Mark McQuade, Activeloop, Mountain View, CA, USA; (8) Mikayel Harutyunyan, Activeloop, Mountain View, CA, USA; (9) Tatevik Hakobyan, Activeloop, Mountain View, CA, USA; (10) Ivo Stranic, Activeloop, Mountain View, CA, USA; (11) Davit Buniatyan, Activeloop, Mountain View, CA, USA. Authors: Authors: (1) Sasun Hambardzumyan, Activeloop, Mountain View, CA, USA; (2) Abhinav Tuli, Activeloop, Mountain View, CA, USA; (3) Levon Ghukasyan, Activeloop, Mountain View, CA, USA; (4) Fariz Rahman, Activeloop, Mountain View, CA, USA;. (5) Hrant Topchyan, Activeloop, Mountain View, CA, USA; (6) David Isayan, Activeloop, Mountain View, CA, USA; (7) Mark McQuade, Activeloop, Mountain View, CA, USA; (8) Mikayel Harutyunyan, Activeloop, Mountain View, CA, USA; (9) Tatevik Hakobyan, Activeloop, Mountain View, CA, USA; (10) Ivo Stranic, Activeloop, Mountain View, CA, USA; (11) Davit Buniatyan, Activeloop, Mountain View, CA, USA. Table of Links Abstract and Intro Current Challenges Tensor Storage Format Deep Lake System Overview Machine Learning Use Cases Performance Benchmarks Discussion and Limitations Related Work Conclusions, Acknowledgement, and References Abstract and Intro Abstract and Intro Current Challenges Current Challenges Tensor Storage Format Tensor Storage Format Deep Lake System Overview Deep Lake System Overview Machine Learning Use Cases Machine Learning Use Cases Performance Benchmarks Performance Benchmarks Discussion and Limitations Discussion and Limitations Related Work Related Work Conclusions, Acknowledgement, and References Conclusions, Acknowledgement, and References 3. TENSOR STORAGE FORMAT Deep Lake datasets follow columnar storage architecture, with tensors as columns, as shown in Fig. 3. Each tensor is a collection of chunks - binary blobs that contain the data samples. An index map associated with each tensor helps find the right chunk and index of the sample within that chunk for a given sample index. 3.1 Dataset A sample in a dataset represents a single row indexed across parallel tensors. As opposed to a document storage format, sample elements are logically independent, which enables partial access to samples for running performant queries or streaming selected tensors over the network to the GPU training instances. Multiple tensors can be grouped. Groups implement syntactic nesting and define how tensors are related to each other. Syntactic nesting avoids the format complication for hierarchical memory layout. Changes to the dataset’s schema are also tracked over time with version control, similar to dataset content changes. 3.2 Tensors Tensors are typed and can be appended or modified in-place. Default access to an index or a set of indices returns the data as NumPy arrays [55]. Instead of storing 1-D data as seen in Parquet [79] or series in Arrow [13], tensors can accommodate n-dimensional data, where typically the first dimension corresponds to the index or batch dimension. Tensors can contain dynamically shaped arrays, also called ragged tensors, as opposed to other statically chunked array formats such as Zarr [52]. 3.3 Types Htype defines the expectations on samples in a tensor such as data type (dtype as seen in NumPy [55]), shape, number of dimensions, or compression. Typed tensors make interacting with deep learning frameworks straightforward and enable sanity checks and efficient memory layout. By inheriting from a generic tensor htype, we can construct types such as image, video, audio, bbox, dicom, and others. For example, a tensor with image htype would expect samples being appended to it to have dtype as uint8 and shape length 3 (i.e. width, height and number of channels). We further expand on the notion of htypes allowing for meta types that support storing image sequences in tensors (sequence[image]), referencing to remotely stored images, while maintaining the regular behavior of a image tensor (link[image]), or even possible cross-format support. 3.4 Memory Layout A Deep Lake dataset contains a provenance file in JSON format and folders per tensor. A tensor contains chunks, chunk encoder, tile encoder, and tensor metadata. Tensors can be optionally hidden. For instance, hidden tensors can be used to maintain down-sampled versions of images or preserve shape information for fast queries. Tensors are stored in chunks at the storage level. While statically (inferred) shaped chunking avoids maintaining a chunk map table, it introduces significant user overhead during the specification of the tensor, custom compression usage limitations, underutilized storage for dynamically shaped tensors, and post-processing inefficiencies. Deep Lake chunks are constructed based on the lower and upper bound of the chunk size to fit a limited number of samples. This comes with a trade-off of having a compressed index map that preserves the sample index to chunk id mapping per tensor while enabling chunk sizes in the range optimal for streaming while accommodating mixed-shape samples. One could consider the approach taken in this paper as an optimized trade-off between file system page map and compute-defined map-less array storage system. For practical reasons, a single chunk encoder can be scaled to billions of images while maintaining a 150MB chunk encoder per 1PB tensor data. Further scaling can be introduced by sharding the chunk encoder. Chunks contain header information such as byte ranges, shapes of the samples, and the sample data itself. If a sample is larger than the upper bound chunk size, which is the case for large aerial or microscopy images, the sample is tiled into chunks across spatial dimensions. The only exception to tiling is videos. Videos are preserved due to efficient frame mapping to indices, key-frame-only decompression, and range-based requests while streaming. 3.5 Access Patterns The tensor storage format is optimized for deep learning training and inference, including sequential and random access. Sequential access is used for running scan queries, transforming tensors into other tensors, or running inference. Random access use cases include multiple annotators writing labels to the same image or models storing back predictions along with the dataset. While the strict mode is disabled, out-of-the-bounds indices of a tensor can be assigned, thus accommodating sparse tensors. However, random assignment over time will produce inefficiently stored data chunks. To fix the data layout, we implement an on-the-fly re-chunking algorithm to optimize the data layout. One of the key access patterns of Deep Lake is shuffled stream access for training machine learning models. It requires random or custom order access while streaming chunks into the training process. This is achieved by involving range-based requests to access sub-elements inside chunks, running complex queries before training to determine the order, and maintaining a buffer cache of fetched and unutilized data. This avoids having a separate compute cluster for running shuffling algorithm [50]. Each tensor has its own chunks, and the default chunk size is 8MB. A single chunk consists of data from multiple indices when the individual data points (image, label, annotation, etc.) are smaller than the chunk size. Conversely, when individual data points are larger than the chunk size, the data is split among multiple chunks (tiling). Exceptions to chunking logic are video data. Deep Lake format is optimized for maximizing throughput to GPU processing. It includes CPU pre-fetching, decompression or decoding, transformations, and GPU memory transfer in a deep learning framework’s expected layout. 3.6 Storage Providers Deep Lake can be plugged into any storage provider, including object storages such as AWS S3 [1], Google Cloud Storage (GCS) [3], POSIX compatible file systems, or local in-memory storage. Moreover, it constructs memory caching by chaining various storage providers together, for instance - the Least Recently Used (LRU) cache of remote S3 storage with local in-memory data. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Deep Lake, a Lakehouse for Deep Learning: Tensor Storage Format

ডিপ লেক, গভীর শিক্ষার জন্য একটি লেকহাউস: টেনসর স্টোরেজ ফর্ম্যাট

Deep Lake, ein Lakehouse für Deep Learning: Tensor-Speicherformat

Deep Lake, una casa de lago para el aprendizaje profundo: formato de almacenamiento tensorial

Deep Lake, un Lakehouse pour le Deep Learning : format de stockage Tensor

डीप लेक, डीप लर्निंग के लिए एक लेकहाउस: टेंसर स्टोरेज फॉर्मेट

ディープラーニングのためのレイクハウス「Deep Lake」: テンソルストレージ形式

딥 러닝을 위한 레이크하우스인 Deep Lake: Tensor 스토리지 형식

Deep Lake, um Lakehouse para aprendizado profundo: formato de armazenamento tensor

Deep Lake, домик у озера для глубокого обучения: тензорный формат хранения

Deep Lake, Derin Öğrenmeye Yönelik Bir Göl Evi: Tensör Depolama Formatı

Deep Lake, một Lakehouse cho Deep Learning: Định dạng lưu trữ Tensor

Deep Lake，深度学习的 Lakehouse：张量存储格式

Deep Lake, a Lakehouse for Deep Learning: Conclusions, Acknowledgement, and References

গভীর লেক, গভীর শিক্ষার জন্য একটি লেকহাউস: উপসংহার, স্বীকৃতি, এবং উল্লেখ

Deep Lake, ein Lakehouse für Deep Learning: Schlussfolgerungen, Danksagungen und Referenzen

Deep Lake, una casa de lago para el aprendizaje profundo: conclusiones, reconocimientos y referencias

Deep Lake, un Lakehouse pour le Deep Learning : conclusions, remerciements et références

डीप लेक, डीप लर्निंग के लिए एक लेकहाउस: निष्कर्ष, स्वीकृति और संदर्भ

ディープラーニングのためのレイクハウス、Deep Lake: 結論、謝辞、参考文献

딥 러닝을 위한 레이크하우스인 Deep Lake: 결론, 감사의 말씀, 참고 자료

Deep Lake, um Lakehouse para Deep Learning: Conclusões, Reconhecimento e Referências

Deep Lake, домик у озера для глубокого обучения: выводы, признательность и ссылки

Deep Lake, Derin Öğrenme için Bir Göl Evi: Sonuçlar, Teşekkür ve Referanslar

Deep Lake, Lakehouse cho Deep Learning: Kết luận, Lời cảm ơn và Tài liệu tham khảo

Deep Lake，深度学习的 Lakehouse：结论、致谢和参考文献

Deep Lake, a Lakehouse for Deep Learning: Deep Lake System Overview

গভীর লেক, গভীর শিক্ষার জন্য একটি লেকহাউস: গভীর লেক সিস্টেম ওভারভিউ

Deep Lake, ein Lakehouse für Deep Learning: Übersicht zum Deep Lake-System

Deep Lake, una casa de lago para el aprendizaje profundo: descripción general del sistema Deep Lake

Deep Lake, une Lakehouse pour le Deep Learning : aperçu du système Deep Lake

डीप लेक, डीप लर्निंग के लिए एक लेकहाउस: डीप लेक सिस्टम अवलोकन

ディープラーニングのためのレイクハウス「Deep Lake」: Deep Lake システムの概要

딥 러닝을 위한 레이크하우스, Deep Lake: Deep Lake 시스템 개요

Deep Lake, um Lakehouse para Deep Learning: Visão geral do sistema Deep Lake

Deep Lake, дом для глубокого обучения: обзор системы Deep Lake

Deep Lake, Derin Öğrenme için Bir Göl Evi: Deep Lake Sistemine Genel Bakış

Deep Lake, Lakehouse cho Deep Learning: Tổng quan về hệ thống Deep Lake

Deep Lake，深度学习的Lakehouse：Deep Lake系统概述

How I'm Building an AI for Analytics Service

আমি কীভাবে অ্যানালিটিক্স পরিষেবার জন্য একটি এআই তৈরি করছি

Wie ich einen KI-Dienst für Analysen aufbaue

Cómo estoy construyendo una IA para el servicio de análisis

Comment je crée un service d'IA pour l'analyse

मैं एनालिटिक्स सेवा के लिए AI का निर्माण कैसे कर रहा हूँ

分析サービスのための AI の構築方法

분석 서비스용 AI를 구축하는 방법

Como estou construindo um serviço de IA para análise

Как я создаю ИИ для службы аналитики

Analitik Hizmeti için Nasıl Yapay Zeka Oluşturuyorum

Cách tôi xây dựng AI cho dịch vụ phân tích

我如何构建人工智能分析服务

Using Arrow Flight SQL Protocol in Apache Doris 2.1 For Super Fast Data Transfer

Many organizations that we’ve spoken to are in the exploration phase of using for AI-powered personalization, recommendations, semantic search and anomaly detection. The recent and astronomical improvements in accuracy and accessibility of large language models (LLMs) including BERT and OpenAI have made companies rethink how to build relevant search and analytics experiences. vector search In this blog, we capture engineering stories from 5 early adopters of vector search- Pinterest, Spotify, eBay, Airbnb and Doordash- who have integrated AI into their applications. We hope these stories will be helpful to engineering teams who are thinking through the full lifecycle of vector search all the way from generating embeddings to production deployments. What is vector search? Vector search is a method for efficiently finding and retrieving similar items from a large dataset based on representations of the data in a high-dimensional space. In this context, items can be anything, such as documents, images, or sounds, and are represented as vector embeddings. The similarity between items is computed using distance metrics, such as or , which quantify the closeness of two vector embeddings. cosine similarity Euclidean distance The vector search process usually involves: : Where relevant features are extracted from the raw data to create vector representations using models such as , or Generating embeddings word2vec BERT Universal Sentence Encoder : The vector embeddings are organized into a data structure that enables efficient search using algorithms such as or Indexing FAISS HNSW : Where the most similar items to a given query vector are retrieved based on a chosen distance metric like cosine similarity or Euclidean distance Vector search To better visualize vector search, we can imagine a 3D space where each axis corresponds to a feature. The time and the position of a point in the space is determined by the values of these features. In this space, similar items are located closer together and dissimilar items are farther apart. GITHUB JULIE-MILLS ^
  |          x Item 1
  |         /
  |        /
  |       /x Item 2
  |      /
  |     /
  |    /x Item 3
  |   /
  |  /
  | /
  +-------------------> Given a query, we can then find the most similar items in the dataset. The query is represented as a vector embedding in the same space as the item embeddings, and the distance between the query embedding and each item embedding is computed. The item embeddings with the shortest distance to the query embedding are considered the most similar. Query item:         x
                    |
  ^                 |
  |          x Item 1|
  |         /       |
  |        /        |
  |       /x Item 2 |
  |      /          |
  |     /           |
  |    /x Item 3    |
  |   /             |
  |  /              |
  | /               |
  +-------------------> This is obviously a simplified visualization as vector search operates in high-dimensional spaces. In the next sections, we’ll summarize 5 engineering blogs on vector search and highlight key implementation considerations. The full engineering blogs can be found below: by Jinfeng Zhuang at Pinterest PinText: A Multitask Text Embedding System in Pinterest by Alexandre Tamborrino at Spotify Introducing Natural Language Search for Podcast Episodes by Senthilkumar Gopal, Shubhangi Tandon, Christopher Miller, Deepika Srinivasan, Rui Kong, Selcuk Kopru and Srinivas Bhagavathula at eBay How eBay’s New Search Feature Was Inspired By Window Shopping by Mihajlo Grbovic at AirBnb Listing Embeddings in Search Ranking by Mitchell Koch, Aamir Manasawala, Raghav Ramesh at Doordash Personalized Store Feed with Vector Embeddings Pinterest: Interest search and discovery Pinterest uses for image search and discovery across multiple areas of its platform, including recommended content on the home feed, related pins and search using a multitask learning model. vector search A multi-task model is trained to perform multiple tasks simultaneously, often sharing underlying representations or features, which can improve generalization and efficiency across related tasks. In the case of Pinterest, the team trained and used the same model to drive recommended content on the homefeed, related pins and search. Pinterest trains the model by pairing a users search query (q) with the content they clicked on or pins they saved (p). Here is how Pinterest created the (q,p) pairs for each task: : Word embeddings are derived from the selected subject (q) and the pin clicked on or saved by the user (p). Related Pins : Word embeddings are created from the search query text (q) and the pin clicked on or saved by the user (p). Search : Word embeddings are generated based on the interest of the user (q) and the pin clicked on or saved by the user (p). Homefeed To obtain an overall entity embedding, Pinterest averages the associated word embeddings for related pins, search and the homefeed. Pinterest created and evaluated its own supervised Pintext-MTL (multi-task learning) against unsupervised learning models including GloVe, word2vec as well as a single-task learning model, PinText-SR on precision. PinText-MTL had higher precision than the other embedding models, meaning that it had a higher proportion of true positive predictions among all positive predictions. Pinterest also found that multi-task learning models had a higher recall, or a higher proportion of relevant instances correctly identified by the model, making them a better fit for search and discovery. To put this all together in production, Pinterest has a multitask model trained on streaming data from the homefeed, search and related pins. Once that model is trained, vector embeddings are created in a large batch job using either Kubernetes+Docker or a map-reduce system. The platform builds a search index of vector embeddings and runs a K-nearest neighbors (KNN) search to find the most relevant content for users. Results are cached to meet the performance requirements of the Pinterest platform. Spotify: Podcast search Spotify combines keyword and to retrieve relevant podcast episode results for users. As an example, the team highlighted the limitations of keyword search for the query “electric cars climate impact”, a query which yielded 0 results even though relevant podcast episodes exist in the Spotify library. To improve recall, the Spotify team used Approximate Nearest Neighbor (ANN) for fast, relevant podcast search. semantic search The team generates vector embeddings using the as it is multilingual, supporting a global library of podcasts, and produces high-quality vector embeddings. Other models were also evaluated including , a model trained on a big corpus of text data, but found that BERT was better suited for word embeddings than sentence embeddings and was pre-trained only in English. Universal Sentence Encoder CMLM model BERT Spotify builds the vector embeddings with the query text being the input embedding and a concatenation of textual metadata fields including title and description for the podcast episode embeddings. To determine the similarity, Spotify measured the cosine distance between the query and episode embeddings. To train the base Universal Sentence Encoder CMLM model, Spotify used positive pairs of successful podcast searches and episodes. They incorporated in-batch negatives, a technique highlighted in papers including and , to generate random negative pairings. Testing was also conducted using synthetic queries and manually written queries. Dense Passage Retrieval for Open-Domain Question Answering (DPR) Que2Search: Fast and Accurate Query and Document Understanding for Search at Facebook To incorporate vector search into serving podcast recommendations in production, Spotify used the following steps and technologies: : Spotify indexes the episode vectors offline in batch using , a search engine with native support for ANN. One of the reasons that Vespa was chosen is that it can also incorporate metadata filtering post-search on features like episode popularity. Index episode vectors Vespa : Spotify uses to generate a query vector. Vertex AI was chosen for its support for GPU inference, which is more cost effective when using large transformer models to generate embeddings, and for its query cache. After the query vector embedding is generated, it is used to retrieve the top 30 podcast episodes from Vespa. Online inference Google Cloud Vertex AI Semantic search contributes to the identification of pertinent podcast episodes, yet it is unable to fully supplant keyword search. This is due to the fact that semantic search falls short of exact term matching when users search an exact episode or podcast name. Spotify employs a hybrid search approach, merging semantic search in Vespa with keyword search in , followed by a conclusive re-ranking stage to establish the episodes displayed to users. Elasticsearch eBay: Image search Traditionally, search engines have displayed results by aligning the search query text with textual descriptions of items or documents. This method relies extensively on language to infer preferences and is not as effective in capturing elements of style or aesthetics. eBay introduces to help users find relevant, similar items that meet the style they’re looking for. image search eBay uses a multi-modal model which is designed to process and integrate data from multiple modalities or input types, such as text, images, audio, or video, to make predictions or perform tasks. eBay incorporates both text and images into its model, producing image embeddings utilizing a Convolutional Neural Network (CNN) model, specifically , and title embeddings using a text-based model such as . Every listing is represented by a vector embedding that combines both the image and title embeddings. Resnet-50 BERT Once the multi-modal model is trained using a large dataset of image-title listing pairs and recently sold listings, it is time to put it into production in the site search experience. Due to the large number of listings at eBay, the data is loaded in batches to HDFS, eBay’s data warehouse. eBay uses Apache Spark to retrieve and store the image and relevant fields required for further processing of listings, including generating listing embeddings. The listing embeddings are published to a columnar store such as HBase which is good at aggregating large-scale data. From HBase, the listing embedding is indexed and served in Cassini, a search engine created at eBay. The pipeline is managed using Apache Airflow, which is capable of scaling even when there is a high quantity and complexity of tasks. It also provides support for Spark, Hadoop, and Python, making it convenient for the machine learning team to adopt and utilize. Visual search allows users to find similar styles and preferences in the categories of furniture and home decor, where style and aesthetics are key to purchase decisions. In the future, eBay plans to expand visual search across all categories and also help users discover related items so they can establish the same look and feel across their home. AirBnb: Real-time personalized listings Search and similar listings features drive 99% of bookings on the AirBnb site. AirBnb built a to improve similar listing recommendations and provide real-time personalization in search rankings. listing embedding technique AirBnb realized early on that they could expand the application of embeddings beyond just word representations, encompassing user behaviors including clicks and bookings as well. To train the embedding models, AirBnb incorporated over 4.5M active listings and 800 million search sessions to determine the similarity based on what listings a user clicks and skips in a session. Listings that were clicked by the same user in a session are pushed closer together; listings that were skipped by the user are pushed further away. The team settled on the dimensionality of a listing embedding of d=32 given the tradeoff between offline performance and memory needed for online serving. https://youtu.be/aWjsUEX7B1I?si=GREROVrWxWqtqL-m&embedable=true AirBnb found that certain listings characteristics do not require learning, as they can be directly obtained from metadata, such as price. However, attributes like architecture, style, and ambiance are considerably more challenging to derive from metadata. Before moving to production, AirBnb validated their model by testing how well the model recommended listings that a user actually booked. The team also ran an A/B test comparing the existing listings algorithm against the vector embedding-based algorithm. They found that the algorithm with vector embeddings resulted in a 21% uptick in CTR and 4.9% increase in users discovering a listing that they booked. The team also realized that vector embeddings could be used as part of the model for real-time personalization in search. For each user, they collected and maintained in real time, using Kafka, a short-term history of user clicks and skips in the last two weeks. For every search conducted by the user, they ran two similarity searches: based on the geographic markets that were recently searched and then the similarity between the candidate listings and the ones the user has clicked/skipped Embeddings were evaluated in offline and online experiments and became part of the real-time personalization features. Doordash: Personalized store feeds Doordash has a wide variety of stores that users can choose to order from and being able to surface the most relevant stores using personalized preferences improves search and discovery. Doordash wanted to apply to its store feed algorithms using vector embeddings. This would enable Doordash to uncover similarities between stores that were not well-documented including if a store has sweet items, is considered trendy or features vegetarian options. latent information Doordash used a derivative of word2vec, an embedding model used in natural language processing, called store2vec that it adapted based on existing data. The team treated each store as a word and formed sentences using the list of stores viewed during a single user session, with a maximum limit of 5 stores per sentence. To create user vector embeddings, Doordash summed the vectors of the stores from which users placed orders in the past 6 months or up to 100 orders. As an example, Doordash used vector search to find similar restaurants for a user based on their recent purchases at popular, trendy joints 4505 Burgers and New Nagano Sushi in San Francisco. Doordash generated a list of similar restaurants measuring the cosine distance from the user embedding to store embeddings in the area. You can see that the stores that were closest in cosine distance include Kezar Pub and Wooden Charcoal Korean Village BBQ. Doordash incorporated store2vec distance feature as one of the features in its larger recommendation and personalization model. With vector search, Doordash was able to see a 5% increase in click-through-rate. The team is also experimenting with new models like , model optimizations and incorporating real-time onsite activity data from users. seq2seq Key considerations for vector search Pinterest, Spotify, eBay, Airbnb and Doordash create better search and discovery experiences with vector search. Many of these teams started out using text search and found limitations with fuzzy search or searches of specific styles or aesthetics. In these scenarios, adding vector search to the experience made it easier to find relevant, and often personalized, podcasts, pillows, rentals, pins and eateries. There are a few decisions that these companies made that are worth calling out when implementing vector search: : Many started out using an off-the-shelf model and then trained it on their own data. They also recognized that language models like word2vec could be used by swapping words and their descriptions with items and similar items that were recently clicked. Teams like AirBnb found that using derivatives of language models, rather than image models, could still work well for capturing visual similarities and differences. Embedding models : Many of these companies opted to train their models on past purchase and click through data, making use of existing large-scale datasets. Training : While many companies adopted ANN search, we saw that Pinterest was able to combine metadata filtering with KNN search for efficiency at scale. Indexing : Vector search rarely replaces text search. Many times, like in Spotify’s example, a final ranking algorithm is used to determine whether vector search or text search generated the most relevant result. Hybrid search : We’re seeing many teams use batch-based systems to create the vector embeddings, given that these embeddings are rarely updated. They employ a different system, frequently Elasticsearch, to compute the query vector embedding live and incorporate real-time metadata in their search. Productionizing Rockset, a real-time search and analytics database, recently added support for . Give vector search on Rockset a try for real-time personalization, recommendations, anomaly detection and more by starting a with $300 in credits today. vector search free trial

A Look Into 5 Use Cases for Vector Search from Major Tech Companies

প্রধান প্রযুক্তি কোম্পানি থেকে ভেক্টর অনুসন্ধানের জন্য 5টি ব্যবহারের ক্ষেত্রে একটি নজর৷

Ein Blick auf 5 Anwendungsfälle für die Vektorsuche bei großen Technologieunternehmen

Una mirada a cinco casos de uso de búsqueda de vectores de las principales empresas tecnológicas

Un aperçu de 5 cas d'utilisation de la recherche de vecteurs par de grandes entreprises technologiques

प्रमुख टेक कंपनियों द्वारा वेक्टर सर्च के लिए 5 उपयोग मामलों पर एक नज़र

大手テクノロジー企業によるベクトル検索の 5 つのユースケースを検証

주요 기술 회사의 벡터 검색에 대한 5가지 사용 사례 살펴보기

Uma análise de 5 casos de uso para pesquisa vetorial de grandes empresas de tecnologia

Взгляд на 5 вариантов использования векторного поиска от крупных технологических компаний

Büyük Teknoloji Şirketlerinden Vektör Aramaya Yönelik 5 Kullanım Durumuna Bir Bakış

Xem xét 5 trường hợp sử dụng tìm kiếm vectơ từ các công ty công nghệ lớn

了解主要科技公司矢量搜索的 5 个用例

Why PostgreSQL Is the Bedrock for the Future of Data

কেন PostgreSQL ডেটার ভবিষ্যতের ভিত্তি

Warum PostgreSQL das Fundament für die Zukunft der Daten ist

Por qué PostgreSQL es la base del futuro de los datos

Pourquoi PostgreSQL est le fondement de l'avenir des données

PostgreSQL डेटा के भविष्य का आधार क्यों है?

PostgreSQL がデータの未来の基盤となる理由

PostgreSQL이 데이터 미래의 기반인 이유

Por que o PostgreSQL é a base para o futuro dos dados

Почему PostgreSQL является основой будущего данных

PostgreSQL Neden Verinin Geleceğinin Temel Taşıdır?

Tại sao PostgreSQL là nền tảng cho tương lai của dữ liệu

为什么 PostgreSQL 是未来数据的基石

Python Bokeh is one of the best Python packages for data visualization. Today, we are going to see some Python Bokeh Examples. I have also provided the Python Bokeh project source code on GitHub. Learn this easy visualization tool and add it to your Python stack. What is Python Bokeh? Python is a data visualization tool, or we can also say Python Bokeh is used to plot various types of graphs. There are various other graph plotting libraries like matplotlib, but Python Bokeh graphs are dynamic in nature, which means you can interact with the generated graph. See the below examples… Bokeh Installation 💻: Python Bokeh can be easily installed using PIP. You can install the Python Bokeh easily by running the command: pip install bokeh Now everything is ready, let’s go through the examples 🏃‍♂️… 1. LinePlot from bokeh.plotting import figure, show, output_notebook
x = [1, 2, 3, 4, 5]
y = [4, 5, 5, 7, 3]
p = figure(title="{LinePlot Python Bokeh Example")
p.line(x, y, line_width=2)
output_notebook()
show(p) | | Live Preview Source Code 🌿 Contribute 2. Scatter Plot from bokeh.plotting import figure, show, output_notebook
x = [1, 2, 3, 4, 5]
y = [4, 5, 5, 7, 3]
p = figure(title="Scatter Plot Python Bokeh Example by PratikPathak.com")
p.circle(x, y, size=10, color="navy", alpha=0.5)
output_notebook()
show(p) 3. Bar Chart from bokeh.plotting import figure, show, output_notebook
categories = ["A", "B", "C", "D", "E"]
values = [10, 15, 8, 12, 6]
p = figure(x_range=categories, title="Bar Chart Python Bokeh Example by PratikPathak.com")
p.vbar(x=categories, top=values, width=0.9)
output_notebook()
show(p) 4. Histogram from bokeh.plotting import figure, show, output_notebook
import numpy as np
data = np.random.normal(0, 1, 1000)
p = figure(title="Histogram Python Bokeh Example by PratikPathak.com")
p.hist(data, bins=30, color="navy", alpha=0.5)
output_notebook()
show(p) 5. Pie Chart from bokeh.plotting import figure, show, output_notebook
labels = ["A", "B", "C", "D"]
values = [10, 15, 8, 12]
p = figure(title="Pie Chart Python Bokeh Example by PratikPathak.com")
p.wedge(x=0, y=0, radius=0.4, start_angle=0.6, end_angle=2.6, color=["red", "green", "blue", "yellow"], legend_label=labels)
output_notebook()
show(p) | | Live Preview Source Code 🌿 Contribute 6. Time Series Plot from bokeh.plotting import figure, show, output_notebook
from datetime import datetime, timedelta
start = datetime(2023, 1, 1)
end = start + timedelta(days=30)
x = [start + timedelta(days=i) for i in range((end-start).days)]
y = [10, 15, 8, 12, 6, 18, 9, 14, 7, 11, 5, 16, 8, 13, 6]
p = figure(x_axis_type="datetime", title="Time Series Plot Python Bokeh Example by PratikPathak.com")
p.line(x, y, line_width=2)
output_notebook()
show(p) 7. Linked Brushing from bokeh.plotting import figure, show, output_notebook
from bokeh.models import ColumnDataSource
x1 = [1, 2, 3, 4, 5]
y1 = [4, 5, 5, 7, 3]
x2 = [2, 3, 4, 5, 6]
y2 = [2, 4, 6, 8, 4]
source = ColumnDataSource(data=dict(x1=x1, y1=y1, x2=x2, y2=y2))
p1 = figure(title="Scatter Plot 1")
p1.circle('x1', 'y1', source=source)
p2 = figure(title="Scatter Plot 2 Python Bokeh Example by PratikPathak.com")
p2.circle('x2', 'y2', source=source)
output_notebook()
show(p1, p2) 8. Hover Tooltips from bokeh.plotting import figure, show, output_notebook
from bokeh.models import HoverTool
x = [1, 2, 3, 4, 5]
y = [4, 5, 5, 7, 3]
p = figure(title="Hover Tooltips Python Bokeh Example by PratikPathak.com")
p.circle(x, y, size=15, fill_color="navy", line_color="white", alpha=0.5)
hover = HoverTool(tooltips=[("(x,y)", "(@x, @y)")])
p.add_tools(hover)
output_notebook()
show(p) 9. Annotations from bokeh.plotting import figure, show, output_notebook
from bokeh.models import Arrow, VectorRenderer, Label
x = [1, 2, 3, 4, 5]
y = [4, 5, 5, 7, 3]
p = figure(title="Annotations Python Bokeh Example by PratikPathak.com")
p.circle(x, y, size=15, fill_color="navy", line_color="white", alpha=0.5)
arrow = Arrow(x_start=2, y_start=4, x_end=3, y_end=5, line_width=2, line_color="red")
label = Label(x=3, y=6, text="This is a label", render_mode='css', border_line_color='black', border_line_alpha=1.0, background_fill_color='white', background_fill_alpha=0.5)
p.add_layout(arrow)
p.add_layout(label)
output_notebook()
show(p) 10. Custom Glyphs from bokeh.plotting import figure, show, output_notebook
from bokeh.models.glyphs import Asterisk
x = [1, 2, 3, 4, 5]
y = [4, 5, 5, 7, 3]
p = figure(title="Custom Glyphs Python Bokeh Example by PratikPathak.com")
p.add_glyph(x, y, Asterisk(size=20, line_color="red", fill_color="yellow"))
output_notebook()
show(p) | | Live Preview Source Code 🌿 Contribute 11. Gridlines and Axes from bokeh.plotting import figure, show, output_notebook
x = [1, 2, 3, 4, 5]
y = [4, 5, 5, 7, 3]
p = figure(title="Gridlines and Axes Python Bokeh Example by PratikPathak.com", x_range=(0, 6), y_range=(0, 8))
p.grid.grid_line_color = "grey"
p.grid.grid_line_dash = [6, 4]
p.xaxis.axis_label = "X-axis"
p.yaxis.axis_label = "Y-axis"
p.line(x, y, line_width=2)
output_notebook()
show(p) 12. Legend from bokeh.plotting import figure, show, output_notebook
x1 = [1, 2, 3, 4, 5]
y1 = [4, 5, 5, 7, 3]
x2 = [2, 3, 4, 5, 6]
y2 = [2, 4, 6, 8, 4]
p = figure(title="Legend Python Bokeh Example by PratikPathak.com")
p.line(x1, y1, line_width=2, color="red", legend_label="Line 1")
p.line(x2, y2, line_width=2, color="blue", legend_label="Line 2")
p.legend.location = "top_left"
output_notebook()
show(p) 13. Categorical Plots from bokeh.plotting import figure, show, output_notebook
fruits = ["Apples", "Pears", "Nectarines", "Plums", "Grapes", "Strawberries"]
counts = [5, 3, 4, 2, 4, 6]
p = figure(x_range=fruits, title="Categorical Plots Python Bokeh Example by PratikPathak.com")
p.vbar(x=fruits, top=counts, width=0.9)
p.xaxis.axis_label = "Fruit"
p.yaxis.axis_label = "Count"
p.xaxis.axis_label_text_font_size = "12pt"
p.yaxis.axis_label_text_font_size = "12pt"
output_notebook()
show(p) 14. Subplots from bokeh.plotting import figure, show, output_notebook
from bokeh.layouts import gridplot
x1 = [1, 2, 3, 4, 5]
y1 = [4, 5, 5, 7, 3]
x2 = [2, 3, 4, 5, 6]
y2 = [2, 4, 6, 8, 4]
p1 = figure(title="Scatter Plot 1 Python Bokeh Example by PratikPathak.com")
p1.circle(x1, y1, size=10, color="navy", alpha=0.5)
p2 = figure(title="Scatter Plot 2 Python Bokeh Example by PratikPathak.com")
p2.circle(x2, y2, size=10, color="red", alpha=0.5)
grid = gridplot([[p1, p2]], plot_width=400, plot_height=400)
output_notebook()
show(grid) 15. Interactive Plots from bokeh.plotting import figure, show, output_notebook
from bokeh.models import ColumnDataSource, HoverTool, BoxSelectTool
import numpy as np
x = np.random.random(1000)
y = np.random.random(1000)
source = ColumnDataSource(data=dict(x=x, y=y))
p = figure(title="Interactive Plots Python Bokeh Example by PratikPathak.com", tools="hover,box_select")
p.circle('x', 'y', source=source, size=3, color="navy", alpha=0.5)
hover = HoverTool(tooltips=[("(x,y)", "(@x, @y)")])
p.add_tools(hover)
output_notebook()
show(p) | | Live Preview Source Code 🌿 Contribute 16. Linked Panning and Zooming from bokeh.plotting import figure, show, output_notebook
from bokeh.models import Range1d
x1 = [1, 2, 3, 4, 5]
y1 = [4, 5, 5, 7, 3]
x2 = [2, 3, 4, 5, 6]
y2 = [2, 4, 6, 8, 4]
p1 = figure(title="Scatter Plot 1 Python Bokeh Example by PratikPathak.com", x_range=Range1d(0, 6), y_range=Range1d(0, 8))
p1.circle(x1, y1, size=10, color="navy", alpha=0.5)
p2 = figure(title="Scatter Plot 2 Python Bokeh Example by PratikPathak.com", x_range=p1.x_range, y_range=p1.y_range)
p2.circle(x2, y2, size=10, color="red", alpha=0.5)
output_notebook()
show(p1, p2) 17. Linked Axis Ranges from bokeh.plotting import figure, show, output_notebook
from bokeh.models import Range1d
x1 = [1, 2, 3, 4, 5]
y1 = [4, 5, 5, 7, 3]
x2 = [2, 3, 4, 5, 6]
y2 = [2, 4, 6, 8, 4]
p1 = figure(title="Scatter Plot 1", x_range=(0, 6), y_range=(0, 8))
p1.circle(x1, y1, size=10, color="navy", alpha=0.5)
p2 = figure(title="Scatter Plot 2", x_range=p1.x_range, y_range=p1.y_range)
p2.circle(x2, y2, size=10, color="red", alpha=0.5)
output_notebook()
show(p1, p2) 18. DateTime Axis from bokeh.plotting import figure, show, output_notebook
from datetime import datetime, timedelta
start = datetime(2023, 1, 1)
end = start + timedelta(days=30)
x = [start + timedelta(days=i) for i in range((end-start).days)]
y = [10, 15, 8, 12, 6, 18, 9, 14, 7, 11, 5, 16, 8, 13, 6]
p = figure(x_axis_type="datetime", title="Datetime Axis Python Bokeh Example by PratikPathak.com")
p.line(x, y, line_width=2)
output_notebook()
show(p) 19. Heatmap from bokeh.plotting import figure, show, output_notebook
import numpy as np
data = np.random.rand(10, 10)
p = figure(title="Heatmap Python Bokeh Example by PratikPathak.com")
p.image(image=[data], x=0, y=0, dw=10, dh=10, palette="Viridis256")
output_notebook()
show(p) 20. Streamgraph from bokeh.plotting import figure, show, output_notebook
from bokeh.models import ColumnDataSource
import numpy as np
# Generate some sample data
x = np.linspace(0, 10, 100)
y1 = np.cumsum(np.random.randn(100))
y2 = np.cumsum(np.random.randn(100))
y3 = np.cumsum(np.random.randn(100))
y4 = np.cumsum(np.random.randn(100))
y5 = np.cumsum(np.random.randn(100))
source = ColumnDataSource(data=dict(x=x, y1=y1, y2=y2, y3=y3, y4=y4, y5=y5))
p = figure(title="Streamgraph Python Bokeh Example by PratikPathak.com")
p.area(x='x', y1='y1', y2='y2', y3='y3', y4='y4', y5='y5', source=source, line_color=None)
output_notebook()
show(p) | | Live Preview Source Code 🌿 Contribute 21. Polar Plot from bokeh.plotting import figure, show, output_notebook
import numpy as np
r = np.linspace(0.1, 1, 10)
theta = np.linspace(0, 2*np.pi, 10)
p = figure(title="Polar Plot Python Bokeh Example by PratikPathak.com", plot_width=600, plot_height=600, x_range=(-1.2, 1.2), y_range=(-1.2, 1.2), toolbar_location=None)
p.polar(r, theta, color="navy", alpha=0.5, line_width=2)
output_notebook()
show(p) 22. Donut Chart from bokeh.plotting import figure, show, output_notebook
from bokeh.models import ColumnDataSource
labels = ["A", "B", "C", "D"]
values = [10, 15, 8, 12]
source = ColumnDataSource(data=dict(labels=labels, values=values))
p = figure(title="Donut Chart Python Bokeh Example by PratikPathak.com", x_range=labels)
p.wedge(x=0, y=1, radius=0.4, start_angle=0.6, end_angle=2.6, color=["red", "green", "blue", "yellow"], legend_field="labels")
p.wedge(x=0, y=1, inner_radius=0.3, outer_radius=0.4, start_angle=0.6, end_angle=2.6, color="white")
p.axis.visible = False
p.grid.grid_line_color = None
output_notebook()
show(p) 23. Treemap from bokeh.plotting import figure, show, output_notebook
from bokeh.models import TreemapRenderer, ColumnDataSource
import pandas as pd

# Sample data
data = pd.DataFrame({
    'label': ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H'],
    'parent': ['', 'A', 'A', 'B', 'B', 'C', 'C', 'D'],
    'value': [10, 5, 4, 3, 2, 1, 1, 1]
})

source = ColumnDataSource(data)
p = figure(title="Treemap Python Bokeh Example by PratikPathak.com")
treemap = TreemapRenderer(
    source=source,
    label='label',
    parent='parent',
    value='value',
    color_discrete_sequence=["#e6550d", "#3182bd", "#31a354", "#756bb1", "#636363", "#9e9ac8", "#de2d26", "#fd8d3c"],
)
p.add_layout(treemap)
output_notebook()
show(p) If you get any error, make sure you have installed Python pandas by using the “pip install pandas” command 24. Network Plot from bokeh.plotting import figure, show, output_notebook
from bokeh.models import ColumnDataSource, HoverTool, NodesAndLinkedEdges
import networkx as nx

# Sample data
nodes = ['A', 'B', 'C', 'D', 'E', 'F']
edges = [('A', 'B'), ('A', 'C'), ('B', 'D'), ('B', 'E'), ('C', 'F')]
G = nx.Graph()
G.add_nodes_from(nodes)
G.add_edges_from(edges)

node_positions = nx.spring_layout(G)
node_x = [node_positions[node][0] for node in G.nodes()]
node_y = [node_positions[node][1] for node in G.nodes()]
node_color = ['red', 'green', 'blue', 'yellow', 'purple', 'orange']

source = ColumnDataSource(data=dict(
    x=node_x,
    y=node_y,
    color=node_color,
    node=nodes
))

p = figure(title="Network Plot Python Bokeh Example by PratikPathak.com", x_range=(-1.1, 1.1), y_range=(-1.1, 1.1))
p.add_tools(HoverTool(tooltips=[("Node", "@node")]))
p.circle('x', 'y', size=15, color='color', source=source)

edge_x = []
edge_y = []
for start, end in edges:
    x0, y0 = node_positions[start]
    x1, y1 = node_positions[end]
    edge_x.append(x0)
    edge_x.append(x1)
    edge_x.append(None)
    edge_y.append(y0)
    edge_y.append(y1)
    edge_y.append(None)

p.multi_line(edge_x, edge_y, color="black", alpha=0.5, line_width=1)

output_notebook()
show(p) 25. Sankey Diagram from bokeh.plotting import figure, show, output_notebook
from bokeh.models import Sankey, ColumnDataSource

# Sample data
flows = [
    ('A', 'X', 5),
    ('B', 'X', 3),
    ('A', 'Y', 2),
    ('B', 'Y', 4),
    ('X', 'C', 4),
    ('X', 'D', 3),
    ('Y', 'C', 2),
    ('Y', 'D', 4)
]

source = ColumnDataSource(data=dict(
    source=[flow[0] for flow in flows],
    target=[flow[1] for flow in flows],
    value=[flow[2] for flow in flows]
))

p = figure(plot_width=800, plot_height=400, title="Sankey Diagram Python Bokeh Example by PratikPathak.com")
sankey = Sankey(source=source, label_text_color="white", label_text_font_size="8pt")
p.renderers.append(sankey)

output_notebook()
show(p) | | Live Preview Source Code 🌿 Contribute 26. Chord Diagram from bokeh.plotting import figure, show, output_notebook
from bokeh.models import ColumnDataSource, CategoricalColorMapper
import numpy as np

# Sample data
nodes = ['A', 'B', 'C', 'D', 'E', 'F']
connections = [
    ('A', 'B', 0.5), ('A', 'C', 0.3), ('A', 'D', 0.2),
    ('B', 'C', 0.4), ('B', 'D', 0.3), ('B', 'E', 0.3),
    ('C', 'D', 0.2), ('C', 'E', 0.3), ('C', 'F', 0.5),
    ('D', 'E', 0.2), ('D', 'F', 0.3), ('E', 'F', 0.4)
]

source = ColumnDataSource(data=dict(
    start=np.array([con[0] for con in connections]),
    end=np.array([con[1] for con in connections]),
    value=np.array([con[2] for con in connections])
))

p = figure(title="Chord Diagram Python Bokeh Example by PratikPathak.com", plot_width=800, plot_height=800)
chord = p.chord(
    source=source, 
    start_angle=np.pi / 2,
    direction="clockwise",
    colors=CategoricalColorMapper(factors=nodes, palette="Spectral11")
)

output_notebook()
show(p) How to Contribute? Feel free to open a PR request on our GitHub repo. Steps to contribute: Fork the repo Make changes in the Forked repo and save Open a Pull Request That’s it 😄! 🌿 Contribute Conclusion In this Repo, I have shared 25+ Python Bokeh examples that can help you learn Python Bokeh. Feel free to contribute to our GitHub repo and keep it updated. Also published . here

Getting Started With Python Bokeh: 25+ Data Visualization Examples With Source Code

How to Use BigQuery Efficiently: Your Guide to Cost Optimization and Efficient Data Management

This New Data Type Is 8 Times Faster Than JSON: Improve Your Semi-Structured Data Analysis

Hello friends, if you have attended technical interviews then may know that the part is one of the toughest to crack. I’ve asked myself why this happens and I found that most developers are not familiar with essential System design topics or concepts. Earlier I shared which are also great for system design interviews. Today I’ll discuss key topics you must know to be prepared for system design interviews. System Design software architecture books As I have said before, System design interviews are a crucial part of the hiring process for software engineers and developers and you must prepare for it, leaving it to chance is not a good idea. These interviews assess your ability to design scalable and efficient systems to solve real-world problems. To excel in system design interviews, it's essential to have a strong grasp of the fundamental concepts and principles. This is the first part of the series and in this part, we will see the top 5 system interview topics. By the way, if you are and want to learn System Design in-depth then you can also refer to sites like , and , which have many great System design courses and tutorials. preparing for System design interviews ByteByteGo DesignGuru Here is a nice system design cheat sheet from DesignGuru for quick revision: 5 Essential System Design Topics for Interviews To stand out from the competition in system design interviews, it's crucial to have a solid understanding of key concepts that set you apart from other candidates. By mastering these ten system design concepts, you can position yourself ahead of 99% of candidates and impress interviewers with your expertise. 1. Scalability Scalability is crucial in system design as it ensures a system can handle increasing loads and maintain performance. To design scalable architectures, you should understand concepts like horizontal and vertical scaling, load balancing, and distributed systems. Those will help you with tech interviews. 2. Availability and Fault Tolerance Designing fault-tolerant systems is essential to maintain availability even in the face of failures. As a candidate, you should learn about replication, redundancy, failover mechanisms, and fault tolerance techniques like backups, checkpoints, and error handling. For high availability many companies create active-active or active-passive architecture as shown below, The best example is companies like AWS and Azure which have data centers in different parts of the world, and if one goes down then server clients from others. cloud computing 3. Data Storage and Databases Different applications require different types of data storage. You should familiarize yourself with different types of databases like relational databases, NoSQL databases, key-value stores, and columnar databases. You should understand their strengths, weaknesses, and use cases. One of the common questions and concepts to understand here is SQL vs NoSQL as they often come during different questions, here is a nice diagram that highlights the difference between SQL and NoSQL 4. Caching Caching helps improve performance by storing frequently accessed data closer to the users. In this topic, you should learn about caching techniques, caching strategies, cache eviction policies, and cache coherence to design efficient caching systems. You should also get yourself familiar with things like cache hit and cache miss and here is a nice diagram for quick reference: 5. Message Queues and Event-driven Architecture Message queues enable asynchronous communication between different components of a system. While preparing for this topic you should understand concepts like pub-sub (publish-subscribe) patterns, message brokers, event-driven architecture, and their applications in building scalable and loosely coupled systems. In the past, I wrote about Apache Kafka vs ActiveMQ vs RabbitMQ which is a good starting point for learning about message brokers and queues. Here is a nice diagram showing Event Driven Architecture using the pub-sub model and point-to-point model : Conclusion That's all about Mastering these ten essential system design concepts will significantly enhance your performance in system design interviews. 5 System design topics you must prepare for tech interviews. By understanding scalability, fault tolerance, data storage, caching, message queues, microservices, security, performance optimization, design patterns, and trade-offs, you'll be well-prepared to tackle complex design problems and impress interviewers with your comprehensive knowledge. Remember, practice and hands-on experience in designing real-world systems are equally important, so apply these concepts in practical scenarios to solidify your understanding. Good luck with your interviews and your future endeavors as a skilled system designer! By the way, if you are and want to learn System Design in-depth then you can also check out sites like , , and , which have many great System design courses. preparing for System design interviews ByteByteGo DesignGuru Exponent In the next part, we will take a look at 5 more topics that are important for system design interviews. Stay tuned!

Code Smell 244 - Incomplete Error Information and How to Fix It

Hi There! So, I believe each of us has encountered a situation when, after modifying an entity in the database, it's necessary to notify other systems about this change. For example, upon accepting an order, we need to inform all related services and teams about the arrival of a new order. The first solution that comes to mind is straightforward: have each service or database polled at regular intervals to check for new orders or status updates. But what if the logic becomes too complex and needs to be segmented? This is where we turn to the event-driven approach, where we broadcast what happened with our objects. is an architectural pattern that allows systems to asynchronously exchange messages or events. The main advantage of EDA is its ability to increase system scalability and flexibility by minimizing direct dependencies between its components. Event-driven architecture (EDA) This facilitates easier adaptation to changes and growth, ensures higher resilience, and simplifies the integration of diverse services. However, a challenge arises in synchronizing data between the database and the messaging system. If we first record data about a new order in the database and then attempt to send a message to Kafka, we cannot guarantee that the message will be sent. A potential producer error could lead to the loss of critical notifications. actually Attempting to wrap these operations in a single transaction (writing to the database and sending to Kafka, followed by a transaction commit) faces fundamental difficulties, as Kafka operates outside the transaction context of databases. This necessitates an alternative solution that ensures atomicity and delivery guarantees for messages without compromising the system's performance and reliability. Here is where the comes to the rescue. Outbox pattern https://microservices.io/patterns/data/transactional-outbox.html?embedable=true The Outbox pattern involves creating a special table (outbox) in the database, where an event intended for the messaging system is recorded within the same transaction as the main data changes. Thus, both operations - changing the state in the main table and creating an event in the outbox table - are guaranteed to be executed together. Subsequently, a dedicated connector (e.g., Kafka Connect) asynchronously reads data from the outbox table and sends it to Kafka, thereby ensuring the delivery of events to the end recipients. This approach achieves high reliability and data consistency without adding undue complexity to the system's architecture or compromising its performance. Let’s review it. The Problem ACID is an acronym describing key transaction properties in databases: A transaction is either executed in full or not executed at all. Atomicity: A transaction transitions the database from one consistent state to another. Consistency: The execution of one transaction doesn't impact the execution of others. Isolation: Completed transactions persist in the database, even in case of system failures. Durability: In distributed systems, the requirement to synchronize data across multiple services is a common challenge. The conventional Two-Phase Commit (2PC) approach is often deemed suboptimal due to its sluggish and blocking nature. Moreover, dealing with data integrity recovery in the event of a system failure poses significant difficulties. The Outbox Pattern emerges as a remedy to address the need for atomicity in writing operations to the database and dispatching messages to a message queue. The core of this solution involves incorporating an additional table within the service's database, referred to as the "Outbox." This table acts as a buffer, storing messages destined for dispatch to a message queue. How the Outbox Pattern Works: Local Transaction: When performing a business operation, data changes are written to the respective database tables. A message intended for other services is placed in the Outbox table within the same transaction. Message Publication: Pollers periodically scan the Outbox table for new messages. Each message from the Outbox table is published to the corresponding message queue. After successful publication, the message is either deleted from the Outbox table or marked as sent. Let’s Code Let's break down this process step by step with a simple example, utilizing Go. You can find the source code and additional materials at the following . link Step 1: Setting Up the Order Service We'll start by creating a service for handling orders. This service will be responsible for receiving orders and recording them in the database. A key component will be implementing the "Outbox" pattern to ensure that every change in order status is reliably transmitted to Kafka. For our case, we need to create only one method: curl -X POST
  http://localhost:8080/api/v1/create-order
  -H 'Content-Type: application/json'
  -d '{
    "name": "John",
    "surname": "Doe",
    "details": {
      "code": 123,
      "operation_code": 456,
      "transaction_code": "sdfsdfsdfs"
    }
  }' The core concept of this service involves writing to both the orders table and the outbox table within a single transaction. For instance, in Go, this would be implemented as follows: func (r *OrderRepository) CreateOrder(userUUID string, order domain.Order) (*int32, error) {
	det := order.Details
	var id int32

	tx, err := r.db.Begin()
	if err != nil {
		return nil, err
	}

	err = tx.QueryRow(
		query,
		order.Name, order.Surname, det.Code, det.OperationCode, det.TransactionCode, StatusCreated).Scan(&id)

	if err != nil {
		tx.Rollback()
		return nil, err
	}

	// Save to outbox
	err = r.createOutbox(userUUID, order)
	if err != nil {
		tx.Rollback()
		return nil, err
	}

	return &id, tx.Commit()
} As a result, we’ll expect messages like this: Step 2: Configuring Kafka JDBC Connector Next, we'll set up the Kafka JDBC Connector to automatically transfer messages from the outbox table to Kafka. This allows other services to subscribe to these events and react to changes in order statuses in real-time. For the Kafka connector, I'll be using the JDBC connector. Find a good way for you to install it: https://docs.confluent.io/kafka-connectors/jdbc/current/source-connector/overview.html?embedable=true#install-the-jdbc-sink-connector After installation, just call connector API: curl http://localhost:8083/connector-plugins We’ll expect to see: [{"class":"io.confluent.connect.jdbc.JdbcSinkConnector","type":"sink","version":"10.7.5"},
{"class":"io.confluent.connect.jdbc.JdbcSourceConnector","type":"source","version":"10.7.5"},
{"class":"org.apache.kafka.connect.mirror.MirrorCheckpointConnector","type":"source","version":"7.6.0-ccs"},
{"class":"org.apache.kafka.connect.mirror.MirrorHeartbeatConnector","type":"source","version":"7.6.0-ccs"},
{"class":"org.apache.kafka.connect.mirror.MirrorSourceConnector","type":"source","version":"7.6.0-ccs"}] Next, we need to create settings for the source. Be careful at this stage, as we are providing credentials for our connector. I recommend creating separate credentials with read-only permissions for the outbox table so that this service can only observe changes. Overall, there are plenty of best practices in this regard, but we won't delve into them here. {
  "name": "jdbc-source-connector",
  "config": {
    "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
    "connection.url": "jdbc:postgresql://postgres:5432/postgres",
    "connection.user": "postgres",
    "connection.password": "postgres",
    "topic.prefix": "microshop.order-",
    "mode": "incrementing",
    "incrementing.column.name": "id",
    "table.whitelist": "outbox",
    "poll.interval.ms": "5000",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "key.converter": "org.apache.kafka.connect.storage.StringConverter",
    "key.converter.schemas.enable": "false",
    "transforms": "createKey",
    "transforms.createKey.type": "org.apache.kafka.connect.transforms.ValueToKey",
    "transforms.createKey.fields": "user_id"
  }
} Set - our table. "table.whitelist": "outbox" Set - our topic. "topic.prefix": "microshop.order-" Set - our message key. "transforms.createKey.fields": "user_id" More about settings and fields: … here And send it. curl -X POST -H "Content-Type: application/json" --data @jdbc-source-config.json http://localhost:8083/connectors And that's it! Now, you just need to create an order, and check if everything is working and if messages are being recorded in the tables. To view what we have in Kafka, I use Kafka UI for such tasks, which allows us to quickly see what we wrote there and if it's working. Step 3: Creating the Background Processing Service We'll create a background service that listens for messages from Kafka and executes the appropriate processing logic. For instance, this service might be responsible for updating order statuses or initiating delivery processes. Full schema: The last service is a background (order-background-service). In our case, it’s just a simple service that can consume messages from Kafka. ...
	err = c.SubscribeTopics([]string{"microshop.order-outbox"}, nil)
	if err != nil {
		fmt.Fprintf(os.Stderr, "Failed to subscribe to topics: %s\n", err)
		os.Exit(1)
	}

...

	ev := c.Poll(cfg.Poll)
	if ev == nil {
	  continue
	}

	switch e := ev.(type) {
		case *kafka.Message:

		// Create an instance of Outbox to store the unmarshaled data
		var outbox domain.Outbox

		// Unmarshal the JSON message into the Outbox struct
		err := json.Unmarshal(e.Value, &outbox)
		if err != nil {
			fmt.Println("Error decoding message:", err)
			return
		}
		fmt.Println("Payload:", outbox.Payload)

		_, err = c.CommitMessage(e)
		if err != nil {	
			fmt.Printf("Failed to commit message: %v\n", err)
		} Be Careful! It is essential to pause here and reflect on the following: as we can observe, entries are now being recorded in tables atomically, ensuring ACID compliance. However, our pipeline has been expanded to include both a Kafka connector and Kafka itself. It is crucial to remember the delivery guarantees that Kafka promises. Apache Kafka offers developers various levels of message delivery guarantees, enabling a balance between performance and reliability according to application requirements. These guarantees are based on the concepts of atomicity, durability, and isolation. Here are the primary delivery guarantees provided by Kafka: : This foundational guarantee ensures a message is delivered at least once. It implies that, in the event of failures or retries, a message may be delivered or processed more than once. This guarantee is facilitated by retry mechanisms in case of errors and acknowledgments from consumers. At Least Once : In this mode, messages can be lost but will not be delivered more than once. This is the fastest and least reliable mode, as it does not account for retries in case of errors. This approach may be acceptable for scenarios where the loss of some messages is not critical. At Most Once : This is the most complex and desirable guarantee, ensuring that each message is delivered and processed “exactly once”, eliminating duplication or loss. Kafka has supported "exactly once" delivery since version 0.11, using a combination of techniques such as idempotent producers and transactions to ensure atomicity across various topics and consumers. Exactly Once The choice among these guarantees depends on your application's requirements for reliability and performance. If you need more information about it just let me know. In conclusion, implementing an Outbox pattern alongside a reliable Kafka connector like the JDBC connector offers a robust solution for orchestrating data changes across microservices. This approach ensures atomicity and consistency when updating multiple systems, significantly reducing the chances of data inconsistencies. This method serves as a crucial first step toward an Event-Driven Architecture (EDA). By introducing a reliable messaging system, like Kafka, and seamlessly integrating it with a transactional database, we establish a foundation for distributing events across the microservices landscape. This approach becomes particularly valuable when considering the need for analytics, as all relevant events are readily available for consumption. Analytics teams can effortlessly retrieve and process these events, contributing to the generation of valuable insights and reports. In essence, the Outbox pattern, coupled with Kafka integration, not only addresses immediate concerns of data consistency but also lays the groundwork for a scalable, event-driven ecosystem that is essential for modern microservices architectures. This facilitates the seamless flow of information between services, unlocking the potential for enhanced analytics and real-time decision-making. If you're interested in learning more about advanced topics such as managing deletions from the Outbox table and further insights into working with this setup, consider subscribing for future updates. Your subscription will help gauge the interest in this topic and guide the direction of upcoming articles. Stay tuned for more in-depth explorations into the intricacies of microservices architectures and event-driven systems! Useful Links Kafka UI: click here Kafka JDBC connector: click here Outbox pattern: click here

Unlocking Microservices Reliability With ACID and the Outbox Pattern

Stories, News & Mentions about COLUMNTAX

Columntax’s Stories on HackerNoon