Tasbu

tasbu

Telugu cinema news, Movie reviews, OTT News, OTT Release dates, Latest Movie reviews in Telugu, telugu movie reviews, Box office collections

Crew: Tentative OTT streaming date of Kareena Kapoor, Tabu, & Kriti Sanon starrer is here | Latest Telugu cinema news | Movie reviews | OTT Updates, OTT

Directed by Rajesh A Krishnan, Crew features Kareena Kapoor Khan, Kriti Sanon and Tabu in lead roles. The fiilm also stars Diljit Dosanjh and Kapil Sharma. , Bollywood News - Times Now

Crew Box Office Collection Day 33: Kareena, Kriti, Tabu's Heist Comedy Sees No Growth At All | Times Now

Authors:
(1) Troisemaine Colin, Department of Computer Science, IMT Atlantique, Brest, France., and Orange Labs, Lannion, France;
(2) Reiffers-Masson Alexandre, Department of Computer Science, IMT Atlantique, Brest, France.;
(3) Gosselin Stephane, Orange Labs, Lannion, France;
(4) Lemaire Vincent, Orange Labs, Lannion, France;
(5) Vaton Sandrine, Department of Computer Science, IMT Atlantique, Brest, France. Table of Links Abstract and Intro Related work Approaches Hyperparameter optimization Estimating the number of novel classes Full training procedure Experiments Conclusion Declarations References Appendix A: Additional result metrics Appendix B: Hyperparameters Appendix C: Cluster Validity Indices numerical results Appendix D: NCD k-means centroids convergence study Appendix A Additional result metrics Appendix B Hyperparameters The Table B3 shows the hyperparameters found by the full procedure described in Section 6. Appendix C Cluster Validity Indices numerical results An estimate of the number of clusters in the 7 datasets considered in this paper can be found in Table C4. Among the 6 CVIs reported here, the Silhouette coefficient performed the best. Furthermore, compared to the original feature space, its average estimation error significantly decreased in the latent space, validating our approach. For some datasets, the Davies-Bouldin index continued to decrease and the Dunn index continued to increase as the number of clusters increased, resulting in very large overestimations. Note that the estimates of the number of novel classes in Table C4 are not needed in the experiments of Section 7.2.2, since Algorithm 1 directly incorporates such estimates in the training procedure. This table has only helped us to identify the most appropriate CVI for our problem. The only exception is the TabularNCD method, which requires an a priori estimation of the number of novel classes in the original feature space. Appendix D NCD k-means centroids convergence study In this appendix, we aim to determine how to achieve the best performance with NCD k-means. Specifically, after the centroid initialization described in Section 3.2, we investigate: (1) whether it is more effective to update the centroids of both known and novel classes, or only the centroids of novel classes; (2) whether the centroids need to be updated using data from both known and novel classes, or only using data from novel classes. The results are presented in Table D5 and show that for 5 out of 7 datasets, the best results are obtained when only the centroids of the novel classes are updated on the unlabeled data. Updating the centroids of the known classes always leads to worse performance, as the class labels are not used in this process. Thus, the centroids of the known classes run the risk of capturing data from the novel classes (and vice versa). This paper is available on arxiv under CC 4.0 license. Authors: (1) Troisemaine Colin, Department of Computer Science, IMT Atlantique, Brest, France., and Orange Labs, Lannion, France; (2) Reiffers-Masson Alexandre, Department of Computer Science, IMT Atlantique, Brest, France.; (3) Gosselin Stephane, Orange Labs, Lannion, France; (4) Lemaire Vincent, Orange Labs, Lannion, France; (5) Vaton Sandrine, Department of Computer Science, IMT Atlantique, Brest, France. Authors: Authors: (1) Troisemaine Colin, Department of Computer Science, IMT Atlantique, Brest, France., and Orange Labs, Lannion, France; (2) Reiffers-Masson Alexandre, Department of Computer Science, IMT Atlantique, Brest, France.; (3) Gosselin Stephane, Orange Labs, Lannion, France; (4) Lemaire Vincent, Orange Labs, Lannion, France; (5) Vaton Sandrine, Department of Computer Science, IMT Atlantique, Brest, France. Table of Links Abstract and Intro Abstract and Intro Related work Related work Approaches Approaches Hyperparameter optimization Hyperparameter optimization Estimating the number of novel classes Estimating the number of novel classes Full training procedure Full training procedure Experiments Experiments Conclusion Conclusion Declarations Declarations References References Appendix A: Additional result metrics Appendix A: Additional result metrics Appendix B: Hyperparameters Appendix B: Hyperparameters Appendix C: Cluster Validity Indices numerical results Appendix C: Cluster Validity Indices numerical results Appendix D: NCD k-means centroids convergence study Appendix D: NCD k-means centroids convergence study Appendix A Additional result metrics Appendix B Hyperparameters The Table B3 shows the hyperparameters found by the full procedure described in Section 6. Appendix C Cluster Validity Indices numerical results An estimate of the number of clusters in the 7 datasets considered in this paper can be found in Table C4. Among the 6 CVIs reported here, the Silhouette coefficient performed the best. Furthermore, compared to the original feature space, its average estimation error significantly decreased in the latent space, validating our approach. For some datasets, the Davies-Bouldin index continued to decrease and the Dunn index continued to increase as the number of clusters increased, resulting in very large overestimations. Note that the estimates of the number of novel classes in Table C4 are not needed in the experiments of Section 7.2.2, since Algorithm 1 directly incorporates such estimates in the training procedure. This table has only helped us to identify the most appropriate CVI for our problem. The only exception is the TabularNCD method, which requires an a priori estimation of the number of novel classes in the original feature space. Appendix D NCD k-means centroids convergence study In this appendix, we aim to determine how to achieve the best performance with NCD k-means. Specifically, after the centroid initialization described in Section 3.2, we investigate: (1) whether it is more effective to update the centroids of both known and novel classes, or only the centroids of novel classes; (2) whether the centroids need to be updated using data from both known and novel classes, or only using data from novel classes. The results are presented in Table D5 and show that for 5 out of 7 datasets, the best results are obtained when only the centroids of the novel classes are updated on the unlabeled data. Updating the centroids of the known classes always leads to worse performance, as the class labels are not used in this process. Thus, the centroids of the known classes run the risk of capturing data from the novel classes (and vice versa). This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

A Practical Approach to Novel Class Discovery in Tabular Data: Appendix 

Authors:
(1) Troisemaine Colin, Department of Computer Science, IMT Atlantique, Brest, France., and Orange Labs, Lannion, France;
(2) Reiffers-Masson Alexandre, Department of Computer Science, IMT Atlantique, Brest, France.;
(3) Gosselin Stephane, Orange Labs, Lannion, France;
(4) Lemaire Vincent, Orange Labs, Lannion, France;
(5) Vaton Sandrine, Department of Computer Science, IMT Atlantique, Brest, France. Table of Links Abstract and Intro Related work Approaches Hyperparameter optimization Estimating the number of novel classes Full training procedure Experiments Conclusion Declarations References Appendix A: Additional result metrics Appendix B: Hyperparameters Appendix C: Cluster Validity Indices numerical results Appendix D: NCD k-means centroids convergence study 6 Full training procedure In the previous sections, we presented the models, the hyperparameter optimization and the estimation procedure of the number of novel classes independently. In this section, these components are brought together to form a complete training procedure. To ensure that no prior knowledge about the novel classes is ever used in this process, the number of novel classes is naturally estimated during the k-fold CV introduced in Section 4. As the whole process is quite complex, we try to summarize it in clear terms in this section and in Algorithm 1. This paper is available on arxiv under CC 4.0 license. Authors: (1) Troisemaine Colin, Department of Computer Science, IMT Atlantique, Brest, France., and Orange Labs, Lannion, France; (2) Reiffers-Masson Alexandre, Department of Computer Science, IMT Atlantique, Brest, France.; (3) Gosselin Stephane, Orange Labs, Lannion, France; (4) Lemaire Vincent, Orange Labs, Lannion, France; (5) Vaton Sandrine, Department of Computer Science, IMT Atlantique, Brest, France. Authors: Authors: (1) Troisemaine Colin, Department of Computer Science, IMT Atlantique, Brest, France., and Orange Labs, Lannion, France; (2) Reiffers-Masson Alexandre, Department of Computer Science, IMT Atlantique, Brest, France.; (3) Gosselin Stephane, Orange Labs, Lannion, France; (4) Lemaire Vincent, Orange Labs, Lannion, France; (5) Vaton Sandrine, Department of Computer Science, IMT Atlantique, Brest, France. Table of Links Abstract and Intro Abstract and Intro Related work Related work Approaches Approaches Hyperparameter optimization Hyperparameter optimization Estimating the number of novel classes Estimating the number of novel classes Full training procedure Full training procedure Experiments Experiments Conclusion Conclusion Declarations Declarations References References Appendix A: Additional result metrics Appendix A: Additional result metrics Appendix B: Hyperparameters Appendix B: Hyperparameters Appendix C: Cluster Validity Indices numerical results Appendix C: Cluster Validity Indices numerical results Appendix D: NCD k-means centroids convergence study Appendix D: NCD k-means centroids convergence study 6 Full training procedure In the previous sections, we presented the models, the hyperparameter optimization and the estimation procedure of the number of novel classes independently. In this section, these components are brought together to form a complete training procedure. To ensure that no prior knowledge about the novel classes is ever used in this process, the number of novel classes is naturally estimated during the k-fold CV introduced in Section 4. As the whole process is quite complex, we try to summarize it in clear terms in this section and in Algorithm 1. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

A Practical Approach to Novel Class Discovery in Tabular Data: Full training procedure

Authors:
(1) Troisemaine Colin, Department of Computer Science, IMT Atlantique, Brest, France., and Orange Labs, Lannion, France;
(2) Reiffers-Masson Alexandre, Department of Computer Science, IMT Atlantique, Brest, France.;
(3) Gosselin Stephane, Orange Labs, Lannion, France;
(4) Lemaire Vincent, Orange Labs, Lannion, France;
(5) Vaton Sandrine, Department of Computer Science, IMT Atlantique, Brest, France. Table of Links Abstract and Intro Related work Approaches Hyperparameter optimization Estimating the number of novel classes Full training procedure Experiments Conclusion Declarations References Appendix A: Additional result metrics Appendix B: Hyperparameters Appendix C: Cluster Validity Indices numerical results Appendix D: NCD k-means centroids convergence study Abstract The problem of Novel Class Discovery (NCD) consists in extracting knowledge from a labeled set of known classes to accurately partition an unlabeled set of novel classes. While NCD has recently received a lot of attention from the community, it is often solved on computer vision problems and under unrealistic conditions. In particular, the number of novel classes is usually assumed to be known in advance, and their labels are sometimes used to tune hyperparameters. Methods that rely on these assumptions are not applicable in real-world scenarios. In this work, we focus on solving NCD in tabular data when no prior knowledge of the novel classes is available. To this end, we propose to tune the hyperparameters of NCD methods by adapting the k-fold cross-validation process and hiding some of the known classes in each fold. Since we have found that methods with too many hyperparameters are likely to overfit these hidden classes, we define a simple deep NCD model. This method is composed of only the essential elements necessary for the NCD problem and performs impressively well under realistic conditions. Furthermore, we find that the latent space of this method can be used to reliably estimate the number of novel classes. Additionally, we adapt two unsupervised clustering algorithms (k-means and Spectral Clustering) to leverage the knowledge of the known classes. Extensive experiments are conducted on 7 tabular datasets and demonstrate the effectiveness of the proposed method and hyperparameter tuning process, and show that the NCD problem can be solved without relying on knowledge from the novel classes. Keywords: novel class discovery, clustering, tabular data, open world learning, transfer learning 1 Introduction Recently, remarkable progress has been achieved in supervised tasks, in part with the help of large and fully labeled sets such as ImageNet [1]. These advancements have predominantly focused on closed-world scenarios, where, during training, it is presumed that all classes are known in advance and have some labeled examples. However, in practical applications, obtaining labeled instances for all classes of interest can be a difficult task due to factors such as budget constraints or lack of comprehensive information. Furthermore, for models to be able to transfer learned concepts to new classes, they need to be designed with this in mind from the start, which is rarely the case. Yet this is an important skill that humans can use effortlessly. For example, having learnt to distinguish a few animals, a person will easily be able to recognise and “cluster” new species they have never seen before. The transposition of this human capacity to the field of machine learning could be a model capable of categorizing new products in novel categories. This observation has led researchers to formulate a new problem called Novel Class Discovery (NCD) [2, 3]. Here, we are given a labeled set of known classes and an unlabeled set of different but related classes that must be discovered. Lately, this task has received a lot of attention from the community, with many new methods such as AutoNovel [4], OpenMix [5] or NCL [6] and theoretical studies [7, 8]. However, most of these works tackle the NCD problem under the unrealistic assumption that the number of novel classes is known in advance, or that the target labels of the novel classes are available for hyperparameter optimization [9]. These assumptions render these methods impractical for real-world NCD scenarios. To address these challenges, we propose a general framework for optimizing the hyperparameters of NCD methods where the ground-truth labels of novel classes are never used, as they are not available in real-world NCD scenarios. Furthermore, we show that the latent spaces obtained by such methods can be used to accurately estimate the number of novel classes. We also introduce three new NCD methods. Two of them are unsupervised clustering algorithms modified to leverage the additional information available in the NCD setting. The first one improves the centroid initialization step of k-means, resulting in a fast and easy to use algorithm that can still give good results in many scenarios. The second method focuses on optimizing the parameters of the Spectral Clustering (SC) algorithm. This approach has a potentially higher learning capacity as the representation itself (i.e. the spectral embedding) is tuned to easily cluster the novel data. Finally, the last approach is a deep NCD method composed of only the essential components necessary for the NCD problem. Compared to SC, this method is more flexible in the definition of its latent space and effectively integrates the knowledge of the known classes. While these contributions can be applied to any type of data, our work focuses on tabular data. The NCD community has focused almost exclusively on computer vision problems and, to the best of our knowledge, only one paper [9] has tackled the problem of NCD in the tabular context. However, this work required the meticulous tuning of a large number of hyperparameters to achieve optimal results. Methods designed for tabular data cannot take advantage of powerful techniques commonly employed in computer vision. Examples include convolutions, data augmentation or Self-Supervised Learning methods such as DINO [10], which have been used with great success in NCD works [11–13], thanks to their strong ability to obtain representative latent spaces without any supervision. On the other hand, tabular data methods have to rely on finely tuned hyperparameters to achieve optimal results. For this reason, we believe that the field of tabular data will benefit the most from our contributions. By making the following contributions, we demonstrate the feasibility of solving the NCD problem with tabular data and under realistic conditions: • We develop a hyperparameter optimization procedure tailored to transfer the results from the known classes to the novel classes with good generalization. • We show that it is possible to accurately estimate the number of novel classes in the context of NCD, by applying simple clustering quality metrics in the latent space of NCD methods. • We modify two classical unsupervised clustering algorithms to effectively utilize the data available in the NCD setting. • We propose a simple and robust method, called PBN (for Projection-Based NCD), that learns a latent representation that incorporates the important features of the known classes, without overfitting on them. The code is available at https://github.com/Orange-OpenSource/PracticalNCD. This paper is available on arxiv under CC 4.0 license. Authors: (1) Troisemaine Colin, Department of Computer Science, IMT Atlantique, Brest, France., and Orange Labs, Lannion, France; (2) Reiffers-Masson Alexandre, Department of Computer Science, IMT Atlantique, Brest, France.; (3) Gosselin Stephane, Orange Labs, Lannion, France; (4) Lemaire Vincent, Orange Labs, Lannion, France; (5) Vaton Sandrine, Department of Computer Science, IMT Atlantique, Brest, France. Authors: Authors: (1) Troisemaine Colin, Department of Computer Science, IMT Atlantique, Brest, France., and Orange Labs, Lannion, France; (2) Reiffers-Masson Alexandre, Department of Computer Science, IMT Atlantique, Brest, France.; (3) Gosselin Stephane, Orange Labs, Lannion, France; (4) Lemaire Vincent, Orange Labs, Lannion, France; (5) Vaton Sandrine, Department of Computer Science, IMT Atlantique, Brest, France. Table of Links Abstract and Intro Abstract and Intro Related work Related work Approaches Approaches Hyperparameter optimization Hyperparameter optimization Estimating the number of novel classes Estimating the number of novel classes Full training procedure Full training procedure Experiments Experiments Conclusion Conclusion Declarations Declarations References References Appendix A: Additional result metrics Appendix A: Additional result metrics Appendix B: Hyperparameters Appendix B: Hyperparameters Appendix C: Cluster Validity Indices numerical results Appendix C: Cluster Validity Indices numerical results Appendix D: NCD k-means centroids convergence study Appendix D: NCD k-means centroids convergence study Abstract The problem of Novel Class Discovery (NCD) consists in extracting knowledge from a labeled set of known classes to accurately partition an unlabeled set of novel classes. While NCD has recently received a lot of attention from the community, it is often solved on computer vision problems and under unrealistic conditions. In particular, the number of novel classes is usually assumed to be known in advance, and their labels are sometimes used to tune hyperparameters. Methods that rely on these assumptions are not applicable in real-world scenarios. In this work, we focus on solving NCD in tabular data when no prior knowledge of the novel classes is available. To this end, we propose to tune the hyperparameters of NCD methods by adapting the k-fold cross-validation process and hiding some of the known classes in each fold. Since we have found that methods with too many hyperparameters are likely to overfit these hidden classes, we define a simple deep NCD model. This method is composed of only the essential elements necessary for the NCD problem and performs impressively well under realistic conditions. Furthermore, we find that the latent space of this method can be used to reliably estimate the number of novel classes. Additionally, we adapt two unsupervised clustering algorithms (k-means and Spectral Clustering) to leverage the knowledge of the known classes. Extensive experiments are conducted on 7 tabular datasets and demonstrate the effectiveness of the proposed method and hyperparameter tuning process, and show that the NCD problem can be solved without relying on knowledge from the novel classes. Keywords : novel class discovery, clustering, tabular data, open world learning, transfer learning Keywords 1 Introduction Recently, remarkable progress has been achieved in supervised tasks, in part with the help of large and fully labeled sets such as ImageNet [1]. These advancements have predominantly focused on closed-world scenarios, where, during training, it is presumed that all classes are known in advance and have some labeled examples. However, in practical applications, obtaining labeled instances for all classes of interest can be a difficult task due to factors such as budget constraints or lack of comprehensive information. Furthermore, for models to be able to transfer learned concepts to new classes, they need to be designed with this in mind from the start, which is rarely the case. Yet this is an important skill that humans can use effortlessly. For example, having learnt to distinguish a few animals, a person will easily be able to recognise and “cluster” new species they have never seen before. The transposition of this human capacity to the field of machine learning could be a model capable of categorizing new products in novel categories. This observation has led researchers to formulate a new problem called Novel Class Discovery (NCD) [2, 3]. Here, we are given a labeled set of known classes and an unlabeled set of different but related classes that must be discovered. Lately, this task has received a lot of attention from the community, with many new methods such as AutoNovel [4], OpenMix [5] or NCL [6] and theoretical studies [7, 8]. However, most of these works tackle the NCD problem under the unrealistic assumption that the number of novel classes is known in advance, or that the target labels of the novel classes are available for hyperparameter optimization [9]. These assumptions render these methods impractical for real-world NCD scenarios. To address these challenges, we propose a general framework for optimizing the hyperparameters of NCD methods where the ground-truth labels of novel classes are never used, as they are not available in real-world NCD scenarios. Furthermore, we show that the latent spaces obtained by such methods can be used to accurately estimate the number of novel classes. We also introduce three new NCD methods. Two of them are unsupervised clustering algorithms modified to leverage the additional information available in the NCD setting. The first one improves the centroid initialization step of k-means, resulting in a fast and easy to use algorithm that can still give good results in many scenarios. The second method focuses on optimizing the parameters of the Spectral Clustering (SC) algorithm. This approach has a potentially higher learning capacity as the representation itself (i.e. the spectral embedding) is tuned to easily cluster the novel data. Finally, the last approach is a deep NCD method composed of only the essential components necessary for the NCD problem. Compared to SC, this method is more flexible in the definition of its latent space and effectively integrates the knowledge of the known classes. While these contributions can be applied to any type of data, our work focuses on tabular data. The NCD community has focused almost exclusively on computer vision problems and, to the best of our knowledge, only one paper [9] has tackled the problem of NCD in the tabular context. However, this work required the meticulous tuning of a large number of hyperparameters to achieve optimal results. Methods designed for tabular data cannot take advantage of powerful techniques commonly employed in computer vision. Examples include convolutions, data augmentation or Self-Supervised Learning methods such as DINO [10], which have been used with great success in NCD works [11–13], thanks to their strong ability to obtain representative latent spaces without any supervision. On the other hand, tabular data methods have to rely on finely tuned hyperparameters to achieve optimal results. For this reason, we believe that the field of tabular data will benefit the most from our contributions. By making the following contributions, we demonstrate the feasibility of solving the NCD problem with tabular data and under realistic conditions: • We develop a hyperparameter optimization procedure tailored to transfer the results from the known classes to the novel classes with good generalization. • We show that it is possible to accurately estimate the number of novel classes in the context of NCD, by applying simple clustering quality metrics in the latent space of NCD methods. • We modify two classical unsupervised clustering algorithms to effectively utilize the data available in the NCD setting. • We propose a simple and robust method, called PBN (for Projection-Based NCD), that learns a latent representation that incorporates the important features of the known classes, without overfitting on them. The code is available at https://github.com/Orange-OpenSource/PracticalNCD. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

A Practical Approach to Novel Class Discovery in Tabular Data

ট্যাবুলার ডেটাতে নভেল ক্লাস ডিসকভারির একটি ব্যবহারিক পদ্ধতি

Ein praktischer Ansatz zur Entdeckung neuer Klassen in tabellarischen Daten

Un enfoque práctico para el descubrimiento de clases novedosas en datos tabulares

Une approche pratique de la découverte de nouvelles classes dans les données tabulaires

सारणीबद्ध डेटा में नवीन वर्ग खोज के लिए एक व्यावहारिक दृष्टिकोण

表形式データにおける新規クラスの発見への実践的アプローチ

표 형식 데이터에서 새로운 클래스 검색에 대한 실용적인 접근 방식

Uma abordagem prática para descoberta de novas classes em dados tabulares

Практический подход к обнаружению новых классов в табличных данных

Tablo Verilerinde Yeni Sınıf Keşfine Pratik Bir Yaklaşım

Một cách tiếp cận thực tế để khám phá lớp mới trong dữ liệu dạng bảng

表格数据中新类别发现的实用方法

The TechBeat: Building Effective Modern Data Architectures with Iceberg, Tabular and MinIO (4/28/2024)

The modern data landscape demands a new breed of infrastructure – one that seamlessly integrates structured and unstructured data, scales effortlessly, and empowers efficient AI/ML workloads. This is where come in, providing a central hub for all your data needs. However, building and managing an effective data lake can be complex. modern datalakes This blog post dives deep into three powerful tools that can optimize your current approach: , and MinIO. The steps below will walk you through how these services seamlessly combine to create a robust, cloud-native data lake architecture specifically optimized for AI/ML workloads. Apache Iceberg Tabular, What is Tabular? Tabular is a data platform created by the original creators of Apache Iceberg. It is designed to provide an independent, universal storage platform that connects to any compute layer, eliminating data vendor lock-in. This feature is critical to the modern data stack, it allows users to pick best-in-class compute and storage tools without being forced into a particular vendor's aged and or mismatched tool set. In an of MinIO and Iceberg and can be enhanced by Tabular. Tabular can be used to manage and query Iceberg data stored in MinIO, allowing for the storage and management of structured data in a scalable, high-performance, and cloud-native manner. These Kubernetes native, components work together smoothly with very little friction and build on each other’s capabilities to perform at scale. architecture Why S3FileIO instead of Hadoop's file-io? This implementation leverages Iceberg's S3FileIO. S3FileIO is considered better than Hadoop's file-io for several reasons. Some of which we’ve already discussed : elsewhere : Iceberg's S3FileIO is designed to work with cloud-native storage. Optimized for Cloud Storage Iceberg uses an to distribute files across multiple prefixes in a MinIO bucket, which helps to minimize throttling and maximize throughput for S3-related IO operations. Improved Throughput and Minimized Throttling: ObjectStoreLocationProvider Iceberg has been to fully leverage strict consistency by eliminating redundant consistency checks that could impact performance. Strict Consistency: updated Iceberg's S3FileIO implements a algorithm, which uploads data files parts in parallel as soon as each part is ready, reducing local disk usage and increasing upload speed. Progressive Multipart Upload: progressive multipart upload Iceberg allows for for S3 API writes to ensure the integrity of uploaded objects, which can be enabled by setting the appropriate catalog property. Checksum Verification: checksum validations Iceberg supports adding to objects during write and delete operations with the S3 API, which can be useful for cost tracking and management. Custom Tags: custom tags The FileIO interface in Iceberg does not require as strict guarantees as a Hadoop-compatible FileSystem, which allows it to that could otherwise degrade performance. Avoidance of Negative Caching: avoid negative caching In contrast, Hadoop's S3A FileSystem, which was used before S3FileIO, does not offer the same level of optimization for cloud storage. All this to say: Don’t hobble your future-facing data lake infrastructure with the trappings of the past. Prerequisites Before you begin, make sure your system meets the following requirements: Docker Docker Compose If you're starting from scratch, you can install both using the installer for your specific platform. It is often easier than downloading Docker and Docker Compose separately. Verify if Docker is installed by running the following command: Docker Desktop docker-compose --version Getting started To get started, clone or copy the YAML file in Tabular’s . You just need the YAML for this tutorial. Feel free to explore the rest of the repository at a later time. git repository Breaking it Down The provided YAML file is a Docker Compose configuration file. It defines a set of services and their configurations for a multi-container Docker application. In this case, there are two services: Spark-Iceberg and MinIO. Let's break down each section: 1. Spark-Iceberg Service: spark-iceberg:
    image: tabulario/spark-iceberg
    container_name: spark-iceberg
    build: spark/
    networks:
      iceberg_net:
    depends_on:
      - rest
      - minio
    volumes:
      - ./warehouse:/home/iceberg/warehouse
      - ./notebooks:/home/iceberg/notebooks/notebooks
    environment:
      - AWS_ACCESS_KEY_ID=admin
      - AWS_SECRET_ACCESS_KEY=password
      - AWS_REGION=us-east-1
    ports:
      - 8888:8888
      - 8080:8080
      - 10000:10000
      - 10001:10001
  rest:
    image: tabulario/iceberg-rest
    container_name: iceberg-rest
    networks:
      iceberg_net:
    ports:
      - 8181:8181
    environment:
      - AWS_ACCESS_KEY_ID=admin
      - AWS_SECRET_ACCESS_KEY=password
      - AWS_REGION=us-east-1
      - CATALOG_WAREHOUSE=s3://warehouse/
      - CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO
      - CATALOG_S3_ENDPOINT=http://minio:9000 Specifies the Docker image to use for the spark-iceberg service. In this case, it uses the tabulario/spark-iceberg:latest image. image: Specifies that the spark-iceberg service depends on the rest and minio services. depends_on: Assigns a specific name (spark-iceberg) to the container. container_name: Sets environment variables for the container, including Spark and AWS credentials. environment: Mounts local directories (./warehouse and ./notebooks) as volumes inside the container. volumes: Maps container ports to host ports for accessing Spark UI and other services. ports: 2. Minio Service: minio:
    image: minio/minio
    container_name: minio
    environment:
      - MINIO_ROOT_USER=admin
      - MINIO_ROOT_PASSWORD=password
      - MINIO_DOMAIN=minio
    networks:
      iceberg_net:
        aliases:
          - warehouse.minio
    ports:
      - 9001:9001
      - 9000:9000
    command: ["server", "/data", "--console-address", ":9001"] Specifies the Docker image for the MinIO service. image: Assigns a specific name (MinIO) to the container. container_name: Sets environment variables for configuring MinIO, including root user credentials. environment: Maps container ports to host ports for accessing MinIO UI. ports: Specifies the command to start the MinIO server with specific parameters. command: Another aspect of the MinIO service is , MinIO’s command line tool. mc mc:
    depends_on:
      - minio
    image: minio/mc
    container_name: mc
    networks:
      iceberg_net:
    environment:
      - AWS_ACCESS_KEY_ID=admin
      - AWS_SECRET_ACCESS_KEY=password
      - AWS_REGION=us-east-1
    entrypoint: >
      /bin/sh -c "
      until (/usr/bin/mc config host add minio http://minio:9000 admin password) do echo '...waiting...' && sleep 1; done;
      /usr/bin/mc rm -r --force minio/warehouse;
      /usr/bin/mc mb minio/warehouse;
      /usr/bin/mc policy set public minio/warehouse;
      tail -f /dev/null
      " Specifies that the mc service depends on the MinIO service. depends_on: Specifies the Docker image for the mc service. image: Assigns a specific name (mc) to the container. container_name: Sets environment variables for configuring the MinIO client. environment: Defines the entry point command for the container, including setup steps for the MinIO client. entrypoint: /usr/bin/mc rm -r --force minio/warehouse;
      /usr/bin/mc mb minio/warehouse;
      /usr/bin/mc policy set public minio/warehouse;
      tail -f /dev/null
      " This sequence of commands essentially performs the following tasks: Removes the existing warehouse directory and its contents from the MinIO server. Creates a new bucket named warehouse. Sets the access policy of the warehouse bucket to public. This Docker Compose file orchestrates a multi-container environment with services for Spark, PostgreSQL, MinIO. It sets up dependencies, environment variables, and commands necessary for running the services together. The services work in tandem to create a development environment for data processing using Spark and Iceberg with MinIO as the object storage backend. Starting Up In a terminal window, cd into the tabular-spark-setup directory in the repository and run the following command: docker-compose up Login into MinIO at with the credentials to see that the warehouse bucket has been created. http://127.0.0.1:9001 admin:password Once all the containers are up and running, you can access your Jupyter Notebook server by navigating to http://localhost:8888 Run one of the sample notebooks and return to MinIO at to see your warehouse populated with data. http://127.0.0.1:9001 Building Your Modern Datalake This tutorial on building a modern datalake with Iceberg, Tabular, and MinIO is just the beginning. This powerful trio opens doors to a world of possibilities. With these tools, you can seamlessly integrate and analyze all your data, structured and unstructured, to uncover hidden patterns and drive data-driven decisions that fuel innovation. Leverage the efficiency and flexibility of this architecture in production to expedite your AI/ML initiatives and unlock the true potential of your machine learning models, accelerating your path to groundbreaking discoveries. Reach out to us at or on our channel if you have any questions as you build. hello@min.io Slack

Building Effective Modern Data Architectures with Iceberg, Tabular and MinIO

আইসবার্গ, ট্যাবুলার এবং মিনিও সহ কার্যকরী আধুনিক ডেটা আর্কিটেকচার তৈরি করা

Erstellen effektiver moderner Datenarchitekturen mit Iceberg, Tabular und MinIO

Creación de arquitecturas de datos modernas y eficaces con Iceberg, Tabular y MinIO

Construire des architectures de données modernes et efficaces avec Iceberg, Tabular et MinIO

आइसबर्ग, टेबुलर और मिनियो के साथ प्रभावी आधुनिक डेटा आर्किटेक्चर का निर्माण

Iceberg、Tabular、MinIO を使用した効果的な最新データ アーキテクチャの構築

Iceberg, Tabular 및 MinIO를 사용하여 효과적인 최신 데이터 아키텍처 구축

Construindo arquiteturas de dados modernas e eficazes com Iceberg, Tabular e MinIO

Создание эффективных современных архитектур данных с помощью Iceberg, Tabular и MinIO

Iceberg, Tabular ve MinIO ile Etkili Modern Veri Mimarileri Oluşturma

Xây dựng kiến trúc dữ liệu hiện đại hiệu quả với Iceberg, Tabular và MinIO

使用 Iceberg、Tabular 和 MinIO 构建有效的现代数据架构

Simple Bar Charts in Python Based on Tabular Data import matplotlib.pyplot as plt
import pandas as pd

df = pd.DataFrame({'x': ['A', 'B', 'C', 'D', 'E'],
                   'y': [50, 30, 70, 80, 60]})

plt.bar(df['x'], df['y'], align='center', width=0.5, color='b', label='data')
plt.xlabel('X axis')
plt.ylabel('Y axis')
plt.title('Bar chart')
plt.legend()
plt.show() Stacked bar chart in Python Based on Tabular Data import matplotlib.pyplot as plt
import pandas as pd

df = pd.DataFrame({'x': ['A', 'B', 'C', 'D', 'E'],
                   'y1': [50, 30, 70, 80, 60],
                   'y2': [20, 40, 10, 50, 30]})

plt.bar(df['x'], df['y1'], align='center', width=0.5, color='b', label='Series 1')
plt.bar(df['x'], df['y2'], bottom=df['y1'], align='center', width=0.5, color='g', label='Series 2')
plt.xlabel('X axis')
plt.ylabel('Y axis')
plt.title('Stacked Bar Chart')
plt.legend()
plt.show() Grouped bar chart based on Tabular Data in Python import matplotlib.pyplot as plt
import pandas as pd
import numpy as np

# Prepare the data
df = pd.DataFrame({
    'group': ['G1', 'G2', 'G3', 'G4', 'G5'],
    'men_means': [20, 35, 30, 35, 27],
    'women_means': [25, 32, 34, 20, 25]
})
ind = np.arange(len(df))  # x-axis position
width = 0.35  # width of each bar

# Plot the bar chart
fig, ax = plt.subplots()
rects1 = ax.bar(ind, df['men_means'], width, color='r')
rects2 = ax.bar(ind + width, df['women_means'], width, color='y')

# Add labels, legend, and axis labels
ax.set_xticks(ind + width / 2)
ax.set_xticklabels(df['group'])
ax.legend((rects1[0], rects2[0]), ('Men', 'Women'))
ax.set_xlabel('Groups')
ax.set_ylabel('Scores')

# Display the plot
plt.show() Percent stacked bar chart based on Tabular Data in Python import matplotlib.pyplot as plt
import pandas as pd

# Prepare the data
df = pd.DataFrame({
    'x': ['Group 1', 'Group 2', 'Group 3', 'Group 4', 'Group 5'],
    'y1': [10, 20, 30, 25, 30],
    'y2': [20, 25, 30, 15, 20],
    'y3': [30, 30, 25, 20, 10]
})

# calculate percentage
y_percent = df.iloc[:, 1:].div(df.iloc[:, 1:].sum(axis=1), axis=0) * 100

# plot the chart
fig, ax = plt.subplots()
ax.bar(df['x'], y_percent.iloc[:, 0], label='Series 1', color='r')
ax.bar(df['x'], y_percent.iloc[:, 1], bottom=y_percent.iloc[:, 0], label='Series 2', color='g')
ax.bar(df['x'], y_percent.iloc[:, 2], bottom=y_percent.iloc[:, :2].sum(axis=1), label='Series 3', color='b')

# Display the plot
plt.show() Thank you for taking the time to explore data-related insights with me. I appreciate your engagement. If you find this information helpful, I invite you to follow me or connect with me on or (@Luca_DataTeam). You can also catch glimpses of my personal life on , Happy exploring!👋 LinkedIn X Instagram

Four Types of Bar Charts in Python - Based on Tabular Data

Google’s MLKit library finds text in images and this works well for simple text but for tabular data such as receipts or tables, it does not. If you’ve ever used an you might have noticed that it works reasonably well when scanning a page with uniform text but when dealing with any kind of tables or text formatting, it just falls flat. OCR app This article discusses an algorithm used in a mobile application I wrote that processes text to keep it in line. We’ll take a common example of a store receipt. In the example below you can see exactly the problem that I’m trying to solve. Lens does not prefer lines for text but will rather put together text that is physically closer together. This works sometimes but definitely does not work for receipts and other tabular data. See specifically how Google Lens put the price as a new line because it was physically far away from the lines on the left. Google 1,49€ is an application developed specifically to scan tabular data and the results of scanning a receipt are in the top right corner.  You can see how it respected the horizontal layout of the receipt and put the price together with the correct article. Line Text Scanner Google OCR library Here is a quick overview of how to use the OCR library: load an image and get text. Recognition of characters happens entirely on Google’s side. InputImage image = InputImage.fromBitmap(mSelectedImage, 0);
        TextRecognizer recognizer = TextRecognition.getClient();
        
        recognizer.process(image)
                .addOnSuccessListener(
                        new OnSuccessListener<Text>() {
                            @Override
                            public void onSuccess(Text texts) {
                                processTextRecognitionResult(texts);
                            }
                        })
                .addOnFailureListener(
                       // 
                        }); Full documentation is here and the diagram below is from this link. https://developers.google.com/ml-kit/vision/text-recognition/v2?embedable=true Once we have the Text object we can begin our own logic. We’ll try to process Text to do better than Google Lens. First, we need to look at the Text object. Text is a hierarchical representation of the scanned text as seen in the following diagram from Google’s documentation. The algorithm looks for characters and words in physical space and puts them into a hierarchical structure. As mentioned, this works well for easy text as in the diagram above but for tabular data like a receipt it does not. Below we have the full text of the receipt scanned by Google Lens. ENTRETIEN/BAZAR......
MPX FILM FRAICH30M
EPICERIE/BOISSONS.......
1,49€
265G POIS CHICHES HARICOTS ROUGES TR
0,95€
PECHE FRAICH FRTS
COOKIES SABLES CHO
1,25€
3,69€
2,39€
HYGIENE/BEAUTE.......
2,99€
20 ELAST COURTS WA
2 X BATISTE SHP SEC 6,59€ 13,18€
2è à -60%
-3,95€
MPX FIL DENTAIR 50 SOIN BLANCHEUR GOU
2,09€
7,29€
SURGELES/PRODUITS FRAIS...........
LAITIERE PV VANILL
1,69€
MALO F.FRAIS FRUIT COMTE AOP 6M 450G
2,35€
5,29€
QUICHE LORRAINE ESCAL.PANEES DE BL
2,79€
OEUFS PLEIN AIRX12
2,85€
MORIN MOUSSE CHOCO
3,25€
2,09€
LONGLEY FARM COT.C
1,99€
TOTAL HORS AVANTAGES
57,62€
18
NOMBRE D'ARTICLES I think that you’ll agree that this is not very useful. The algorithm picked the columns first rather than the rows. It made new lines for text that was far away. This is the difference in text processing that this article explores. Processing Text To demonstrate what MLKit does I’ve modified the application to only output the index of Text Elements (those are boxes with dark green border in the diagram from Google). In the screen below we can see text blocks 0 to 31. We can see how the MLKit library prefers physical distance to group words together into phrases. This is not exactly what Google Lens has outputted. We can see that Google Lens made the price of 1,49€ below the first three lines. It treated the text blocks differently but didn’t manage to output clear lines. As seen from this output, the OCR library makes up block 0 as ‘Entrentien/Bazar. MPX FILM FRAICH30M’. The second block is made up of multiple lines of the left column. The library continues to create text blocks from the left column and then moves to the right column to create the text blocks from prices. The image is correctly converted into characters and words. We’ll use these words but do it using a different algorithm to prefer lines to make our sentences. It’s clear that Google Lens also processes these blocks but obviously in yet another way. Breaking boxes Now to our algorithm. The first thing we’ll do is process all the TextBlocks from Text to extract the individual lines. Each line has one or more object and we extract them all and put them into a single  list. Text.Element List<Text.Element> textElements = new ArrayList<>();

        for (int i = 0; i < blocks.size(); i++) {
            Rect rect = blocks.get(i).getBoundingBox();
  
            List<Text.Line> lines = blocks.get(i).getLines();
  
            for (int j = 0; j < lines.size(); j++) {
                List<Text.Element> elements = lines.get(j).getElements();
                for (int k = 0; k < elements.size(); k++) {
                    Text.Element e = elements.get(k);
                    textElements.add(e);
                 }
            } At this point, we have an array of objects which all contain some text. In our example, the four in block 0 are 'ENTRETIEN/BAZAR......’, ‘MPX’, ‘FILM’, and ‘FRAICH30M’. All of the elements are put into a list. Text.Element Text.Elements Sorting boxes So now that we have all these objects in a list, how do we determine what goes after what? As seen from above, the OCR algorithm just recognized blocks of text but it made no effort to make correct sentences. First, we need to realize that object is bound by Rect object which has coordinates and dimensions. We use this fact to sort the . Text.Element Text.Element Text.Elements We take the top and left coordinates of the rectangles and sort them: public int compare(Text.Element t1, Text.Element t2) {
            int diffOfTops = t1.getBoundingBox().top - t2.getBoundingBox().top;
            int diffOfLefts = t1.getBoundingBox().left - t2.getBoundingBox().left;

            int height = (t1.getBoundingBox().height() + t2.getBoundingBox().height()) / 2;
            int verticalDiff = (int)(height * 0.35);

            int result = diffOfLefts;
            if (Math.abs(diffOfTops) > verticalDiff) {
                result =  diffOfTops;
            }
            return result;
        } The main decision in this algorithm is to decide whether one rectangle should go after another based on whether it is more to the right of it or more below it. The horizontal difference is easy, if one rectangle has coordinates more to the left of another rectangle then it will come before it. But this will be taken into consideration only if the difference in the tops is more than 35% height of the line. Basically, we allow for some skew in the picture-taking where the text might not align perfectly. But if a rectangle is 35% the height of the current rectangle below the next rectangle then we consider it as another line. Overall, in our algorithm, we prefer horizontal lines along which we can find the objects. Text.Element Making lines In this final processing step, we will make lines by processing the list above. Remember that the list just has rectangles and so we need to add a condition based on coordinates to decide whether to start a new line. To do this, we get the difference between the top of the boxes and take the average of their height. We don’t check the difference in location on the horizontal axis as we already did that when sorting boxes. Here we just have one decision to make: whether or not to make a new line. private boolean isSameLine(Text.Element t1, Text.Element t2) {
        int diffOfTops = t1.getBoundingBox().top - t2.getBoundingBox().top;
    
        int height = (t1.getBoundingBox().height() + t2.getBoundingBox().height()) * 0.35;

        if (Math.abs(diffOfTops) > height ) {
            return false;
        }
        return true;
    } This finally gives us the well-formatted tabular text. Here is the full text again. ENTRETIEN/BAZAR.  
MPX FILM FRAICH3OM 1,49€  
EPICERIE/BO1SSONS.  
265G POIS CHICHES 0,95€  
HARICOTS ROUGES TR 1,25e  
PECHE FRAICH FRTS 3,69e  
COOKIES SABLES CHO 2,39e  
HYGIENE/BEAUTE.....  
20 ELAST COURTS WA 2,99€  
2 X BATISTE SHP SEC 6,59€ 13,18€  
2è à -60% -3,95€  
MPX FIL DENTAIR 50 2,09e  
SOIN BLANCHEUR GOU 7,29€  
SURGELES/PRODUITS FRAIS....  
LAITIERE PV VANILL 1,69€  
MALO F.FRAIS FRUIT 2,35€  
COMTE AOP 6M 450G 5,29€  
QUICHE LORRAINE 2,79€  
ESCAL.PANEES DE BL 2,85e 
OEUFS PLEIN AIRX12 3,25€  
MORIN MOUSSE CHOCO 2,09e  
LONGLEY FARM COT.C 1,99e  
TOTAL HORS AVANTAGES 57,62e 
NOMBRE D'ARTICLES 18 Conclusion The solution to process the text in lines in this article is not without faults. The text should be relatively straight and not skewed too much. With some effort, we could calculate the vectors of lines even in the most crumpled receipt but that would be for another article. Here we show how a small tweak in treating the output of a large library can have a big effect for a very specific purpose. If you think that this is useful to you, the app is available in the . Google Play Store

Using Google MLKit Text Recognition to Scan Tabular Data

The abundance of tabular data being accumulated by enterprises internally and online represents a big challenge for data integration and analysis for companies that strive to be data-driven. While the main focus of AI/ML research and community during the last several years has been on processing unstructured data, tabular data still is where the most time and money are spent in the Data Integration world. These days, every company, whether big or small, has dozens of different systems (data sources). Many of those have great integration capabilities, but still, every day people export data from various systems and ingest it into proprietary enterprise Data Lakes, Data Warehouses, Reporting databases, etc. That requires a lot of time and qualified Data Engineers. To keep up with quickly changing business requirements and growing data size, there have been many great technologies and tools developed by companies and communities: big data stack, data wrangling tools, quality and observability, visualization, and NLU interfaces to the data. All those technologies allowed the collection and processing of unprecedented amounts of data. Though it also makes the modern data engineering stack very complex, which, in turn, results in long and expensive development cycles that don’t meet business requirements Tools assume that the user understands the data, knows how it’s structured, and how to cross-reference data that comes from different sources. And that represents probably the biggest problem in today’s corporate world. In order to do something with the data you have to understand it, know where it is, and how it’s structured. There is a misconception that there are people who fully understand all the nuances of companies' data and can efficiently manage it. And it becomes exponentially more difficult when a company grows and people change. In its work, Datuum specifically targets this challenge by applying state-of-the-art AI models to help Data Workers handle the variety and complexity of tabular data. Datuum speeds up and optimizes the integration of multiple data sources by helping data workers find the data they need and connect the dots between source and destination. As a result of interacting with your data, our AI learns more with every iteration and that data so the institutional knowledge is not lost. Why Is It Not Easy? Imagine that you have two different data sources with seemingly similar data stored in them. For instance, data from medical or insurance domains, in which some tables could contain hundreds or even thousands of columns. Our major task is to unify this data into a single source of truth. The cornerstone of this unification is the process of column matching – discovering combinations of columns from different data sources which store the same information (Fig. 1). While it could be trivial to match columns with generalized data like name, email, or address, it's not the case for columns containing domain-specific information – to map this kind of data, we usually need domain expertise and manual intervention. The other important phase of the unification pipeline would be record matching – the process in which records from different data sources are merged into a single entity. For highly regulated domains like healthcare, this step could be extremely important. And it's even harder to get right when large datasets are being processed, even for human annotators. Transformers to the Rescue! Thankfully, the latest research in large language models gives us the key to the simplification of the matching process. It doesn't yet fully remove the need for human experts (especially in complex domains) but could give them a substantial boost compared to standard settings. By definition, language modeling is the usage of various statistical and probabilistic techniques to determine the probability of a given sequence of words occurring in a sentence. Language models train by analyzing text to correctly predict the next or masked word in a sequence (Fig. 2). The task of language modeling is not new, but in recent years, it has become one of the main areas for research and development in the machine learning field. There are 3 reasons why this sub-field experienced such enormous progress: The nature of language modeling tasks allowed us to use tremendous amounts of data for unsupervised learning. Transformer attention-based architecture allowed the massive parallelism of training on sequential data. Transformers pre-trained on a language modeling task turned out to be great at solving other, unrelated to language modeling, tasks. As a result, transformer models have found use in a wide range of tasks, ranging from question answering to video understanding. Naturally, people also looked at applying this architecture to tabular data as well. Tabular data is textual in its nature. One could argue that numeric columns do not constitute text, but in fact, numerical values are nothing other than sequences of specialized text characters. This means that tabular data, similar to textual one, could be effectively modeled as a language using Transformers. Modeling tabular data with Transformers facilitates the solution of other downstream tasks, such as column type prediction or column name recovery. Some of the proposed models could also learn efficient dense representations of the table parts: columns and rows. Deep Learning Journey Machine learning approaches using tabular data can hardly be called new. For many years, tree-based solutions, such as random forests or gradient boosting algorithms, were used for various classification and regression tasks, such as credit score prediction or fraud detection. However, the complexity of the aforementioned models lets us learn and understand patterns of data on the observation (row) level. For a much deeper understanding of data, as relations between columns in a table or between tables in a database, neural networks became very handy. In the pre-transformer era, deep learning models had a limited capability in processing sequential data like column values or names and, hence, learning contextual information from it. As a result, they often relied on manually engineered features for model input and processed each column separately. One prominent example of such a model is Sherlock [1] – the first successful application of deep learning to the task of semantic type classification. Sherlock is a regular feed-forward neural network that uses a set of statistical features calculated from column values to predict one of 78 semantic column types. It was the first deep-learning model used to improve column matching in Datuum. Despite being quite effective for common column types, like name, address, or zip code, Sherlock had a set of limitations, such as high reliance on annotated data, especially in previously unseen domains, or inability to take table context and column name into account. Obviously, that represented an obstacle to successfully applying Datuum in new verticals fast. As that architecture required additional training on the new data. Fortunately enough, the Transformer-based model proved to be a perfect alternative to Sherlock. The Case for Transformers Our history with the application of transformers to the task of column matching started with extensive research of related work. As a result, we chose TABBIE [2] and TaBERT [3] models for further experiments. It should be noted that at the beginning of 2022, almost all SOTA approaches to the task of column-type classification were based on Transformers (Fig. 3). By fine-tuning these models on multiple relevant datasets, we proved the superiority of transformer-based methods over Sherlock. For example, the TABBIE F1-macro score on the TURL dataset [4] was almost 20% higher than Sherlock’s. In fact, the performance of transformers on classification tasks wasn’t what we were looking for. The property that caught our attention most was the representational ability of these models. For instance, by training transformer-based models like TABBIE on domain-specific datasets for the task of semantic column type classification, we were able to learn dense representations of the elements of these tables: rows and columns. This, in turn, allowed us to extend the application of the model to previously unseen domains without the need to annotate new data. Our column-matching pipeline is specifically designed for a system that has a regular inflow of new data from previously unseen domains (Fig. 4). Generally, our approach works the following way: We fine-tuneTABBIE model with domain-specific data for a task that does not require additional annotation. For example, corrupted cell detection or cell in-filling. Such training is called semi-supervised. The model learns dense representations of columns that leverage both column information and table context. At runtime, we encode columns from tables that need to be matched and combine them into matching pairs based on distance measures between dense representations in vector space. Human experts review low-confidence matches for correctness. This pipeline has a few important benefits compared to the pipeline based on the Sherlock model, namely: Matching is not limited to a defined set of column types. The need for annotation is eliminated. Ability to include encoded column data, metadata, and context into a single representation. Combined, these advantages set a new standard in the accuracy and adaptivity of column matching employed in Datuum. Conclusion The representational ability of Transformer-based language models is bringing new capabilities to almost all tasks that rely on textual data, and table cross-reference is no exception. By employing a Transformer model in our column-matching pipeline, we were able to increase the matching accuracy and generalization ability of the system that doesn’t depend on annotated data, which is especially useful in domains where data access is restricted. References https://hci.stanford.edu/\~cagatay/projects/table-understanding/Sherlock-KDD19.pdf https://arxiv.org/pdf/2105.02584.pdf https://arxiv.org/pdf/2005.08314.pdf https://arxiv.org/pdf/2006.14806.pdf Also published here

What You Need to Know About Tabular Data as a Challenge

Transformer-based models are a game-changer when it comes to using unstructured text data. As of September 2020, the top-performing models in the General Language Understanding Evaluation (GLUE) benchmark are all BERT transformer-based models. At , we often encounter scenarios where we have supporting tabular feature information and unstructured text data. We found that by using the tabular data in these models, we could further improve performance, so we set out to build a toolkit that makes it easier for others to do the same. Georgian Building on Top of Transformers The main benefits of using transformers are that they can learn long-range dependencies between text and can be trained in parallel (as opposed to sequence to sequence models), meaning they can be pre-trained on large amounts of data. Given these advantages, BERT is now a staple model in many real-world applications. Likewise, with libraries such as , it’s easy to build high-performance transformer models on common NLP problems. HuggingFace Transformers Currently, transformer models using unstructured text data are well understood. However, in the real-world, text data is often supported by rich structured data or other unstructured data like audio or visual information. Each one of these might provide signals that one alone would not. We call these different ways of experiencing data—audio, visual or text—modalities. Think about E-commerce reviews as an example. Beyond the review text itself, information about the seller, buyer and product is available as numerical and categorical features. We set out to explore how text and tabular data could be used together to provide stronger signals in our projects. We started by exploring the field known as multimodal learning, which focuses on how to process different modalities in machine learning. Multimodal Literature Review The current models for multimodal learning mainly focus on learning from the sensory modalities such as audio, visual, and text. Within multimodal learning, there are several branches of research. The MultiComp Lab at Carnegie Mellon University provides an excellent . Our problem falls under Multimodal Fusion—joining information from two or more modalities to make a prediction. taxonomy As text data is our primary modality, we focus on the literature that treats text as the main modality and introduces models that leverage the transformer architecture. Trivial Solution to Structured Data Before we dive into the literature, there is a simple solution that can be used where the structured data is treated as regular text and is appended to the standard text inputs. Taking the E-commerce reviews example, the input can be structured as follows: Review. Buyer Info. Seller Info. Numbers/Labels. Etc. One caveat with this approach, however, is that it is limited by the maximum token length that a transformer can handle. Transformer on Images and Text In the last couple of years, transformer extensions for image and text have advanced significantly. by Kiela et al. (2019) uses pre-trained ResNet and pre-trained BERT features on unimodal images and text respectively and feeds this into a Bidirectional transformer. The key innovation is adapting the image features as additional tokens to the transformer model. Supervised Multimodal Bitransformers for Classifying Images and Text Additionally, there are models, (Lu et al. 2019)  and (Su et al. 2020), which define pre-training tasks for images and text. Both models pre-train on the , which contains roughly 3.3 million image-caption pairs (web images with captions from alt text). In both cases, for a given image, a pre-trained object detection model like Faster R-CNN obtains vector representations for regions of the image, which count as input token embeddings to the transformer model. ViLBERT VLBert Conceptual Captions dataset As an example, ViLBert pre-trains on the following training objectives: : Mask input image and word tokens. For the image, the model tries to predict a vector capturing image features for the corresponding image region, while for text, it predicts the masked text based on the textual and visual clues. Masked multimodal modelling : Whether the image and text pair are actually from the same image and caption pair. Multimodal alignment An image of the pretraining tasks and an example of masked multimodal learning are shown below. Given the image and text, if we mask out dog, the model should be able to use the unmasked visual information to correctly predict the masked word to be dog. All these models use the bidirectional transformer model that is the backbone of BERT. The differences lie in the pretraining tasks the models are trained on and slight additions to the transformer. In the case of ViLBERT, the authors also introduce a co-attention transformer layer (shown below) to define the attention mechanism between the modalities explicitly. Finally, there’s also (Tan and Mohit 2019), another pre-trained transformer model that as of Transformers version 3.1.0, is implemented as part of the library. The input to LXMERT is the same as ViLBERT and VLBERT. However, LXMERT pre-trains on aggregated datasets, which also include visual question answering datasets. In total LXMERT pre-trains on 9.18 million image text pairs. LXMERT Transformers on Aligning Audio, Visual, and Text Beyond transformers for combining image and text, there are multimodal models for audio, video, and text modalities in which there is a natural ground truth temporal alignment. Papers along this direction include MulT, (Tsai et al. 2019), and the Multimodal Adaptation Gate (MAG) from (Rahman et al. 2020). Multimodal Transformer for Unaligned Multimodal Language Sequences Integrating Multimodal Information in Large Pretrained Transformers MuIT is similar to ViLBert in which co-attention is used between pairs of modalities. Meanwhile, MAG hopes to inject other modality information at certain transformer layers via a gating mechanism. Transformers with Text and Knowledge Graph Embeddings Some works have also identified knowledge graphs as a vital piece of information in addition to text data. (Ostendorff et al. 2019) uses features from the author entities in the Wikidata knowledge graph in addition to metadata features for book category classification. In this case, the model is a simple concatenation of these features and BERT output text features of the book title and description before some final classification layers. Enriching BERT with Knowledge Graph Embeddings for Document Classification Key Takeaway The main takeaway for adapting transformers for multimodal data is to ensure that there's an attention or weighting mechanism between the multiple modalities. The attention mechanisms can occur at different points of the transformer architecture, as encoded input embeddings, injected in the middle, or combined after the transformer encodes the text data. Multimodal Transformers Toolkit Using what we’ve learned from the literature review and the comprehensive HuggingFace library of state of the art transformers, we’ve developed a toolkit. The package extends any HuggingFace transformer for tabular data. To see the code, documentation, and working examples, check out . multimodal-transformers the project repo At a high level, the outputs of a transformer model on text data and tabular features containing categorical and numerical data are combined in a combining module. Since there is no alignment in our data, we choose to combine the text features after the transformer’s output. The combining module implements several methods for integrating the modalities, including, attention and gating methods inspired by the literature survey. More details of these methods are available . here Walkthrough Let's work through an example where we classify clothing review recommendations. We'll use a simplified version of the example included in the Colab notebook. We will use the from Kaggle, which contains 23,000 customer reviews. Women's E-Commerce Clothing Reviews In this dataset, we have text data in the and columns. We also have categorical features from the , , and columns and numerical features from the and . Title Review Text Clothing ID Division Name Department Name Class Name Rating Positive Feedback Count Loading The Dataset We first load our data into a , which works with PyTorch's data loaders that include the text inputs for HuggingFace Transformers and our specified categorical feature columns and numerical feature columns. For this, we also need to load our HuggingFace tokenizer. TorchTabularTextDataset Loading Transformer with Tabular Model Now we load our transformer with a tabular model. We first specify our tabular configurations in a object. This config is then set as the member variable of a HuggingFace transformer config object. Here, we also specify how we want to combine the tabular features with the text features. In this example, we will use a weighted sum method. TabularConfig tabular_config Once we have the set, we can load the model using the same API as HuggingFace. See the for the list of currently supported transformer models that include the tabular combination module. tabular_config documentation Training For training, we can use HuggingFace’s class. We also need to specify the training arguments, and in this case, we will use the default. trainer Let's take a look at our models in training! Results Using this toolkit, we also ran our experiments on the dataset for recommendation prediction and the dataset for price prediction. The former is a classification task, while the latter is a regression task. Our results are in the table below. The text_only combine method is a baseline that uses only the transformer and is essentially the same as a HuggingFace for SequenceClassification model. Women’s E-Commerce Clothing Reviews Melbourne Airbnb Open Data We can see that incorporating tabular features improves performance over the text_only method. The performance gains also depend on how strong the training signals from the tabular data are. For example, in the review recommendation case, the text_only model is already a strong baseline. Next Steps We've already used the toolkit successfully in our projects. Feel free to try it out on your next machine learning project! Check out the and the included script for how to do evaluation and inference. Also, if you want support for your favourite transformer, feel free to add transformer support . documentation main here Appendix Readers should check out and , for a well summarized overview of transformers and BERT. The Illustrated Transformer The Illustrated BERT Here’s a quick taxonomy of papers we reviewed. Transformer on Image and Text Supervised Multimodal Bitransformers for Classifying Images and Text  (Kiela et al. 2019) ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks (Lu et al. 2019) VL-BERT: Pre-training of Generic Visual-Linguistic Representations (Su et al. ICLR 2020) LXMERT: Learning Cross-Modality Encoder Representations from Transformers (Tan et al. EMNLP 2019) Transformers on Aligning Audio, Visual, and Text Multimodal Transformer for Unaligned Multimodal Language Sequences (Tsai et al. ACL 2019) Integrating Multimodal Information in Large Pretrained Transformers (Rahman et al. ACL 2020) Transformers with Knowledge Graph Embeddings Enriching BERT with Knowledge Graph Embeddings for Document Classification (Ostendorff et al. 2019) ERNIE: Enhanced Language Representation with Informative Entities (Zhang et al. 2019 [ Colab Tutorial ] [ Github ] Also published at https://medium.com/georgian-impact-blog/how-to-incorporate-tabular-data-with-huggingface-transformers-b70ac45fcfb4

A Beginner Guide to Incorporating Tabular Data via HuggingFace Transformers

Tree-based models like Random Forest and XGBoost have become very popular in solving tabular(structured) data problems and gained a lot of tractions in Kaggle competitions lately. It has its very deserving reasons. However, in this article, I want to introduce a different approach from fast.ai’s module leveraging. Tabular (Embeddings other than word representations can be used) Motivation Tree-based models like Random Forest and XGBoost have become very popular in solving tabular(structured) data problems and gained a lot of tractions in Kaggle competitions lately. It has its very deserving reasons. However, in this article, I want to introduce a different approach from fast.ai’s module leveraging: Tabular Deep Learning and Embedding Layers. This is a bit against industry consensus that Deep Learning is more for unstructured data like image, audio or NLP, and usually not suitable for handling tabular data. Yet, the introduction of embedding layers for the categorical data changed this perspective and we’ll try to use ’s tabular module on the on and see how far this approach can go. fast.ai Blue Book Bulldozers Competition Kaggle You can find the Kaggle Notebook 📔: here . Loading Data First, let’s import the necessary modules. The core one here is : fastai.tabular fastai * fastai.tabular * from import from import Then we’ll read the data into a Pandas DataFrame. You can find the specific code in the Kaggle Notebook link on top of this article but for here, I’ll only show necessary code snippets to keep things as concise as possible. We will read in the CSV file into and this will be the DataFrame we’ll be mainly working on. We will also read in which is the test set. train_df test_df Let’s take a brief look at the data we’re dealing with: len(train_df), len(test_df)
( , ) 401125 12457 Sorting the Training Set This is to create a good validation set. It cannot be emphasized enough how important a good validation set is in making a successful model. Since we are predicting sales price data in the future, we need to make a validation set that all of its data is collected in the ‘future’ of the training set. So we need to sort the training set first, then split the ‘future’ part as the validation set. train_df = train_df.sort_values(by= , ascending= )
train_df = train_df.reset_index(drop= ) 'saledate' False True Data Pre-Processing The competition’s evaluation methods use RMSLE (root mean squared log error). So if we take the log of our prediction, we can just use the good old RMSE as our loss function. It’s just easier this way. train_df.SalePrice = np.log(train_df.SalePrice) For , since we will be using deep learning to tackle the problem and it is very good at feature extraction, we’ll only do it on the . This is the advantage of using a Deep Learning approach, it requires way less feature engineering and less dependent on domain knowledge. We’ll use the fast.ai’s function to for adding some more features related to the sale date. Feature Engineering saledate add_datepart add_datepart(train_df, , drop= )
add_datepart(test_df, , drop= ) # The only feature engineering we do is add some meta-data from the sale date column, using 'add_datepart' function in fast.ai "saledate" False "saledate" False What does is it takes the column and added a bunch of other columns like , , whether it is the start or end of a month, quarter and year, etc. These added features will offer more insights into the date and are relevant to user purchasing behaviors. For example, at the end of the year, the company will usually run promotions and prices will usually decrease. add_datepart saledate day of week day of month Let’s check whether all these date related features got added into our DataFrame: pd.option_context( , , , ): 
        display(df)
        
display_all(train_df.tail( ).T) # check and see whether all date related meta data is added. : def display_all (df) with "display.max_rows" 1000 "display.max_columns" 1000 10 They did get added. Good. Now we need to do some data pre-processing since this DataFrame has quite some missing data and we also want to categorify and normalize the columns. With the fast.ai library, this is rather simple. We just specify the pre-processing methods we want into a Python list, like so: procs=[FillMissing, Categorify, Normalize] # Defining pre-processing we want for our fast.ai DataBunch This variable procs will later be used to create the fast.ai DataBunch for training. Building the Model Let’s look at the data types of each column to decide which ones are categorical and which ones are continuous: procs=[FillMissing, Categorify, Normalize] # Defining pre-processing we want for our fast.ai DataBunch Here are the results: Then we’ll put all categorical columns into a list and all continuous columns into a list . These two variables will also be used to construct fast.ai DataBunch. cat_vars cont_vars cat_vars = [ , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ]
cont_vars = [ , ] # prepare categorical and continous data columns for building Tabular DataBunch. 'SalesID' 'YearMade' 'MachineID' 'ModelID' 'datasource' 'auctioneerID' 'UsageBand' 'fiModelDesc' 'fiBaseModel' 'fiSecondaryDesc' 'fiModelSeries' 'fiModelDescriptor' 'ProductSize' 'fiProductClassDesc' 'state' 'ProductGroup' 'ProductGroupDesc' 'Drive_System' 'Enclosure' 'Forks' 'Pad_Type' 'Ride_Control' 'Stick' 'Transmission' 'Turbocharged' 'Blade_Extension' 'Blade_Width' 'Enclosure_Type' 'Engine_Horsepower' 'Hydraulics' 'Pushblock' 'Ripper' 'Scarifier' 'Tip_Control' 'Tire_Size' 'Coupler' 'Coupler_System' 'Grouser_Tracks' 'Hydraulics_Flow' 'Track_Type' 'Undercarriage_Pad_Width' 'Stick_Length' 'Thumb' 'Pattern_Changer' 'Grouser_Type' 'Backhoe_Mounting' 'Blade_Type' 'Travel_Controls' 'Differential_Type' 'Steering_Controls' 'saleYear' 'saleMonth' 'saleWeek' 'saleDay' 'saleDayofweek' 'saleDayofyear' 'saleIs_month_end' 'saleIs_month_start' 'saleIs_quarter_end' 'saleIs_quarter_start' 'saleIs_year_end' 'saleIs_year_start' 'MachineHoursCurrentMeter' 'saleElapsed' We’ll create another DataFrame df to feed into the DataBunch. We also specify the dependent variable as dep_var . dep_var = df = train_df[cat_vars + cont_vars + [dep_var, ]].copy() # rearrange training set before feed into the databunch 'SalePrice' 'saledate' Now is the time to create our validation set. We do this by cutting out a block of the most recent entries from the training set. How big the block should be? Well, the same size as the test set. Let’s see the code: test_df[ ].min(), test_df[ ].max() cut = train_df[ ][(train_df[ ] == train_df[ ][len(test_df)])].index.max()
cut valid_idx = range(cut) # Look at the time period of test set, make sure it's more recent 'saledate' 'saledate' # Calculate where we should cut the validation set. We pick the most recent 'n' records in training set where n is the number of entries in test set. 'saledate' 'saledate' 'saledate' 12621 # specify the valid_idx variable as the cut out range. We first look at the time period of the test set and make sure it is more recent than all our training set. Then we calculate how many records we need to cut out. Finally, let’s construct our DataBunch for training using fast.ai’s datablock API: data = (TabularList.from_df(df, path=path, cat_names=cat_vars, cont_names=cont_vars, procs=procs)
                   .split_by_idx(valid_idx)
                   .label_from_df(cols=dep_var, label_cls=FloatList)
                   .databunch()) # Use fast.ai datablock api to put our training data into the DataBunch, getting ready for training Building the Model We will fire up a fast.ai from the DataBunch we just created. We want to limit the price range for our prediction to be within the history sale price range, so we need to calculate the . Note that we multiplied the maximum of by 1.2 so when we apply sigmoid, the upper limit will also be covered. This is a small trick to squeeze a bit more performance out of the model. tabular.learner y_range SalePrice max_y = np.max(train_df[ ])* y_range = torch.tensor([ , max_y], device=defaults.device)
y_range
tensor([ , ], device= ) 'SalePrice' 1.2 0 0.0000 14.2363 'cuda:0' Now we can create our learner: learn = tabular_learner(data, layers=[ , ], ps=[ , ], emb_drop= , y_range=y_range, metrics=rmse) # Create our tabular learner. The dense layer is 1000 and 500 two layer NN. We used dropout, hai 1000 500 0.001 0.01 0.04 The single most important thing about fast.ai is the use of embedding layers for categorical data. This is the ‘ ’ that enables Deep Learning to be competitive in handling tabular data. With one embedding layer for each categorical variable, we introduced good interaction for the categorical variables and leverage Deep Learning’s biggest strength: Automatic Feature Extraction. We also used Drop Out for both the dense layers and embedding layers for better regularization. The metrics of the learner is RMSE since we’ve already taken the log of SalePrice. Let’s look at the model. tabular_learner secret sauce TabularModel(
  (embeds): ModuleList(
    ( ): Embedding( , )
    ( ): Embedding( , )
    ( ): Embedding( , )
    ( ): Embedding( , )
   ...
    ( ): Embedding( , )
    ( ): Embedding( , )
    ( ): Embedding( , )
  )
  (emb_drop): Dropout(p= , inplace= )
  (bn_cont): BatchNorm1d( , eps= , momentum= , affine= , track_running_stats= )
  (layers): Sequential(
    ( ): Linear(in_features= , out_features= , bias= )
    ( ): ReLU(inplace= )
    ( ): BatchNorm1d( , eps= , momentum= , affine= , track_running_stats= )
    ( ): Dropout(p= , inplace= )
    ( ): Linear(in_features= , out_features= , bias= )
    ( ): ReLU(inplace= )
    ( ): BatchNorm1d( , eps= , momentum= , affine= , track_running_stats= )
    ( ): Dropout(p= , inplace= )
    ( ): Linear(in_features= , out_features= , bias= )
  )
) 0 388505 600 1 72 18 2 331868 600 3 5155 192 60 3 3 61 2 2 62 3 3 0.04 False 2 1e-05 0.1 True True 0 2102 1000 True 1 True 2 1000 1e-05 0.1 True True 3 0.001 False 4 1000 500 True 5 True 6 500 1e-05 0.1 True True 7 0.01 False 8 500 1 True As can be seen from the above, we have embedding layers for categorical columns, then followed by a drop out layer. We have a batch norm layer for the continuous columns, then we concatenate all of them (categorical embeddings + continuous variables) together and throw them into two fully connected layers with 1000 and 500 nodes, with Relu, BatchNorm, and Dropout in between. Quite standard stuff. Now that we have the model, let’s use fast.ai’s learning rate finder to find a good learning rate: learn.lr_find()
learn.recorder.plot() We’ll pick the learning rate at the end of the biggest learning rate curve slope: le-02 Let’s do some training using fast.ai’s One-Cycle Training approach. Note that we added some weight decay (0.2) for regularization. learn.fit_one_cycle( , , wd= ) 2 1e-2 0.2 We can train some more cycles with smaller learning rate: learn.fit_one_cycle( , , wd= ) 5 3e-4 0.2 We’ve reached a score of on our validation set. Since the competition is not accepting submissions, we can only look at the leaderboard to get a rough idea of how well this model performs: 0.223 The top place is . Compare to this model’s . We don’t know how well it works on the test set but overall I think the result we got isn’t bad at all. 0.229 0.223 A Couple of More Words on Embedding Layers What makes everything click here is the embedding layers. Embedding is just a fancy word of saying mapping something to a vector. Like the word embedding that is getting more popular in NLP, it means using a vector (the size is arbitrary though, depending on tasks) to represent words, and those vectors are weights and can be trained via back-prop. Similarly, for our case, we used embeddings on our categorical variables. Each column gets an embedding matrix that can be trained. And each unique column value gets a specific vector mapped to it. The beautiful thing about this is: , ‘semantics’ in the form of weights that matters to our sale price and can be extracted and trained via our deep neural network. The model will have the ‘ ’ it needs to fit a big data set well. with embedding, we can now develop ‘semantics’ to the variable depth But don’t take my words for it, or just look at the results of my humble little project. In a more glory case, there was by the folks who came 3rd in a Kaggle competition for something called (prediction future sales). Among the top teams in the leaderboard, everyone else used some kind of heavy feature engineering, but by using embedding layers, they managed to score 3rd place with way less feature engineering. this paper Rossman What’s more interesting is, with embedding layers, you can actually visualize the variable projection in the embedding matrix space. Take the Rossman project as an example. They took a two-dimensional projection of the embedding matrix for the German states. And if you circle some states on the embedding space and same states on the actual map. You’ll find out that they are scarily similar. The embedding layer actually discovered geography.

Gain State-Of-The-Art Results on Tabular Data with Deep Learning & Embedding Layers [A How To Guide]

I originally wrote this post for the . SocialCops engineering blog Photo by on Carles Rabada Unsplash The PDF ( ) was born out of to create “a universal way to communicate documents across a wide variety of machine configurations, operating systems and communication networks”. Basically, the goal was to make documents viewable on any display and printable on any modern printer. PDF was built on top of (a page description language), which had already solved this “view and print anywhere” problem. PDF encapsulates the components required to create a “view and print anywhere” document. These include characters, fonts, graphics and images. Portable Document Format The Camelot Project PostScript A PDF file defines instructions to place characters (and other components) at precise coordinates relative to the bottom-left corner of the page. Words are simulated by placing some characters closer than others. Similarly, spaces are simulated by placing words relatively far apart. How are tables simulated then? You guessed it correctly — by placing words as they would appear in a spreadsheet. x,y The PDF format has no internal representation of a table structure, which makes it difficult to extract tables for analysis. Sadly, a lot of open data is stored in PDFs, which was not designed for tabular data in the first place! Camelot: PDF table extraction for humans Today, we’re pleased to announce the release of Camelot, a Python library and command-line tool that makes it easy for anyone to extract data tables trapped inside PDF files! You can check out the documentation at and follow the development on . Read the Docs GitHub How to install Camelot Installation is easy! After , you can install Camelot using pip (the recommended tool for installing Python packages): installing the dependencies $ pip install camelot-py How to use Camelot Extracting tables from a PDF using Camelot is very simple. Here’s how you do it. ( used in the following example.) Here’s the PDF >>> import camelot>>> tables = camelot.read_pdf('foo.pdf')>>> tables<TableList n=1>>>> tables.export('foo.csv', f='csv', compress=True) # json, excel, html>>> tables[0]<Table shape=(7, 7)>>>> tables[0].parsing_report{'accuracy': 99.02,'whitespace': 12.24,'order': 1,'page': 1}>>> tables[0].to_csv('foo.csv') # to_json, to_excel, to_html>>> tables[0].df # get a pandas DataFrame! You can also check out the . command-line interface Why use Camelot? Camelot gives you complete control over table extraction by letting you tweak its settings. Bad tables can be discarded based on metrics like accuracy and whitespace, without ever having to manually look at each table. Each table is a , which seamlessly integrates into . pandas DataFrame ETL and data analysis workflows You can export tables to multiple formats, including CSV, JSON, Excel and HTML. Okay, but why another PDF table extraction library? TL;DR: Total control for better table extraction Many people use open ( , ) and closed-source ( , ) tools to extract tables from PDFs. But they either give a nice output or fail miserably. There is no in between. This is not helpful since everything in the real world, including PDF table extraction, is fuzzy. This leads to the creation of ad-hoc table extraction scripts for each type of PDF table. Tabula pdf-table-extract smallpdf pdftables We created Camelot to offer users complete control over table extraction. If you can’t get your desired output with the default settings, you can tweak them and get the job done! You can check out a . comparison of Camelot’s output with other open-source PDF table extraction libraries The longer read We’ve often needed to extract data trapped inside PDFs. The first tool that we tried was , which has nice user and command-line interfaces, but it either worked perfectly or failed miserably. When it failed, it was difficult to tweak the settings — such as the image thresholding parameters, which influence table detection and can lead to a better output. Tabula We also tried closed-source tools like and , which worked slightly better than Tabula. But then again, they also didn’t allow tweaking and cost money. (We wrote a blog post about how we went about extracting tables from PDFs back in 2015, titled .) smallpdf pdftables “PDF is evil” When these full-blown PDF table extraction tools didn’t work, we tried (an open-source command-line utility). pdftotext extracts text from a PDF while preserving the layout, using spaces. After getting the text, we had to write Python scripts with complicated regexes ( ) to convert the text into tables. This wasn’t scalable, since we had to change the regexs for each new table layout. pdftotext regular expressions We clearly needed a tweakable PDF table extraction tool, so we started developing one in December 2015. We started with the idea of giving the tool back to the community, which had given us so many open-source tools to work with. We knew that Tabula classifies PDF tables into two classes. It has two methods to extract these different classes: Lattice (to extract tables with clearly defined lines between cells) and Stream (to extract tables with spaces between cells). We named Camelot’s table extraction flavors, Lattice and Stream, after Tabula’s methods. For Lattice, , an image processing technique to detect lines. Since we wanted to use Python, was the obvious choice to do image processing. However, OpenCV’s returned only line equations. After more exploration, we settled on , which gave the exact line segments. From here, representing the table trapped inside a PDF was straightforward. Tabula uses Hough Transform OpenCV Hough Line Transform morphological transformations To get more information on how Lattice and Stream work in Camelot, check out the section of the documentation. “How It Works” How we use Camelot We’ve battle tested Camelot by using it in a variety of projects, both for one-off and automated table extraction. Earlier this year, we developed our to help organizations track and measure their contribution to . For India, we identified open data sources (primarily PDF reports) for each of the 17 Sustainable Development Goals. For example, one of our sources for Goal 3 (“Good Health and Well-Being for People”) is the released by . To get data from these PDF sources, we created an internal web interface built on top of Camelot, where our data analysts could upload PDF reports and extract tables in their preferred format. UN SDG Solution Agenda 2030 National Family Health Survey (NFHS) report IIPS Note: We became finalists for the UN SDG Action Awards in February 2018 . We also set up an . The workflow scrapes the website for weekly PDFs of disease outbreak data, and then it extracts tables from the PDFs using Camelot, sends alerts to our team, and loads the data into a data warehouse. ETL workflow using Apache Airflow to track disease outbreaks in India Integrated Disease Surveillance Programme (IDSP) To infinity and beyond! Camelot has some limitations. (We’re developing solutions!) Here are a couple of them: When using Stream, tables aren’t autodetected. Stream treats the whole page as a single table, which gives bad output when there are multiple tables on the page. Camelot only works with text-based PDFs and not scanned documents. (As Tabula , “If you can click-and-drag to select text in your table in a PDF viewer… then your PDF is text-based”.) explains You can check out the for more information. GitHub repository You can help too — every contribution counts! Check out the for guidelines around contributing code, documentation or tests, reporting issues and proposing enhancements. You can also head to the and look for issues labeled “help wanted” and “good first issue”. Contributor’s Guide issue tracker We urge organizations to release open data in a “data friendly” format like the . But while tables are trapped inside PDF files, there’s Camelot :) CSV

Announcing Camelot, a Python Library to Extract Tabular Data from PDFs

Court Reveals Wright's 2024 MYOB Email Was Manipulated to Support False Claims

Author:
(1) David Novoa-Paradela, Universidade da Coruña, CITIC, Campus de Elviña s/n, 15008, A Coruña, Spain & Corresponding author (Email: david.novoa@udc.es);
(2) Oscar Fontenla-Romero, Universidade da Coruña, CITIC, Campus de Elviña s/n, 15008, A Coruña, Spain (Email: oscar.fontenla@udc.es);
(3) Bertha Guijarro-Berdiñas, Universidade da Coruña, CITIC, Campus de Elviña s/n, 15008, A Coruña, Spain (Email: berta.guijarro@udc.es). Table of Links Abstract and Introduction Related work The proposed pipeline Evaluation Conclusion & Acknowledgements References & Appendix 3. The proposed pipeline The purpose of the proposed pipeline is, given the text reviews of an Amazon product, to classify them as normal if they refer to it, and as anomalous if they describe different products or if they are so generic that they do not provide useful information to consumers. These classifications will be accompanied by a normality score and explanations that justify them. Figure 1 shows the modules that are part of the proposed pipeline to solve this task. In the first phase, we encode the text reviews of the target product using the pretrained MPNet transformer [25]. In the second phase, a DAEF [26] autoencoder is trained using these embeddings. Using the reconstruction errors issued by the network and a predefined threshold error, we can classify reviews as normal (error greater than the threshold) or anomalous (error less than the threshold). For the last phase, we propose a method based on the most frequent normal terms to generate an explanation associated with each classification. Therefore, the proposed pipeline is composed of three main modules: (1) Text encoding; (2) Anomaly detection; (3) Explainability. The following sections describe each of them in detail. 3.1. Text encoding Although there are models capable of dealing directly with images or text, machine learning models are usually designed to be trained using numerical vectors that represent the sample data (tabular data). This input format of the data is the usual one for classic anomaly detection methods. In the NLP area, there are multiple techniques to represent texts using vectors of real numbers, which are known as embeddings. These techniques allow the generation of vector spaces that try to represent the relationships and semantic similarities of the language, so that, for example, two synonymous words will be found at a shorter distance in the vector space than two unrelated words. Embeddings can be calculated independently for each word of the language (word embeddings), which led to models such as word2vec [27] or Global Vectors (GloVe) [28]. The representation of a sentence (sentence embedding) or a document (document embedding) will therefore be the sum of all the individual representations of the terms that make it up. To obtain representations of fixed length, it is usual to perform operations such as the mean. In certain cases, these operations between embeddings can worsen or even invalidate the final embedding, so specific models have been developed capable of understanding and representing a text as a whole, instead of just encoding it word by word. Among these models are those based on transformers [29], such as BERT [30], XLNet [31], GPT-3 [32] or GPT-4 [33], which have been trained over large-scale datasets and can solve different tasks, including sentence embedding. Since it inherits the advantages of the BERT and XLNet models while overcoming their limitations, we will use the pre-trained MPNet [25] model to calculate the embeddings of the reviews. MPNet combines Masked Language Modeling (MLM) and Permuted Language Modeling (PLM) to predict token dependencies, using auxiliary position information as input to enable the model to view a complete sentence and reduce position differences. Models like GPT-3 or GPT-4 are more advanced, but in addition to not being open source, they demand much higher computational resources, unaffordable for a significant part of the scientific community, as is our particular case. MPNet maps sentences and paragraphs to a 768 dimensional dense vector space e ∈ R 768×1 , providing fast and quality encodings. Computing time is a critical aspect in the area in which this work is framed, since in e-commerce platforms (and online reviews in general) we can find a huge number of products and reviews to deal with. Therefore, the proposed pipeline uses MPNet to calculate the embeddings of the reviews, which allows them to be represented as numeric vectors of fixed size. This model and others are available in the Hugging Face repository [34]. 3.2. Anomaly detection Anomaly detection is a field with a large number of algorithms that solve the problem of distinguishing between normal and anomalous instances in a wide variety of ways [4, 35]. Depending on the assumptions made and the processes they employ, we can distinguish between five main types of methods: probabilistic, distance-based, information theory-based, boundarybased, and reconstruction-based methods. Among the latter are autoencoder networks (AE) [36], one of the most widely used models. AE are a type of self-associative neural network whose output layer seeks to reproduce the data presented to the input layer after having gone through a dimensional compression phase. In this way, they manage to obtain a representation of the input data in a space with a dimension smaller than the original, learning a compact representation of the data, retaining the important information, and compressing the redundant one. One of the requirements of our proposed pipeline is to generate a normality score for each review. This score will allow us, among other things, to order the different reviews by their level of normality. When AE networks are used in anomaly detection scenarios, the classification is usually carried out based on the reconstruction error that they emit to reproduce in its output the embeddings of the reviews it receives as inputs, which represents the level of normality of the evaluated instance. This reconstruction error can be used as the normality score we are looking for. Due to the speed of its training, we have decided to use DAEF (Deep Autoencoder for Federated learning) [26] as our anomaly detection model. Unlike traditional neural networks, DAEF trains a deep AE network in a non-iterative way, which drastically reduces its training time without losing the performance of traditional (iterative) AEs. The proposed pipeline uses DAEF to calculate the embeddings of the reviews, being able to issue a normality score associated with each of the review classifications. 3.3. Explainability As black-box ML models are mainly being employed to make important predictions in critical contexts, the demand for transparency is increasing. The danger is in creating and using decisions that are not justifiable, legitimate, or that do not allow obtaining detailed explanations of their behaviour. Explanations supporting the output of a model are crucial, moreover in fields such as medicine, autonomous vehicles, security, or finance. When ML models do not meet any of the criteria imposed to declare them explainable, a separate method must be applied to the model to explain its decisions. This is the purpose of post-hoc explainability techniques [37], which aim at communicating understandable information about how an already developed model produces its predictions for any given input. Within these techniques, the so-called model-agnostic techniques are those designed to be plugged into any model with the intent of extracting some information from its prediction procedure. To generate the explanations we have implemented an approach based on a statistical analysis of the dataset. Based on the definitions of normal and anomalous review presented in Section 2, Our hypothesis assumes that normal reviews will always refer directly or indirectly to the target product so that there will be a list of terms used very frequently among normal texts. The appearance of one or more of these “normal” terms in a review would justify its classification by the system as normal. In the same way, anomalous reviews may be justified with the non-presence of said terms. The original text reviews classified as normal using the AD model will be processed (tokenization, lemmatization and stopword removal) and analyzed to obtain the list of the top-n most frequent terms. This list of terms will be compared with the lists of frequent terms of other products in order to remove the terms they have in common, so that the final list for a product does not contain generic terms. This final list of terms will be used to generate the explanations. New reviews classified as normal will be searched for the presence of any of these terms. To make this search more flexible, it will also be analyzed if there are terms in the review semantically close to any of the words on the list. To quantify the closeness of two words, the cosine similarity emitted by the MPNet transformer model itself will be used as a measure of distance. Reviews classified as anomalous will be explained by the non-occurrence of terms from the list. Figure 2 represents an overall view of the pipeline considering the product “chocolate bars” as the normal class. In summary, in this work, we propose the use of this statistical analysis of the terms to generate the explanations associated with the classification of the reviews. This paper is available on arxiv under CC 4.0 license. Author: (1) David Novoa-Paradela, Universidade da Coruña, CITIC, Campus de Elviña s/n, 15008, A Coruña, Spain & Corresponding author (Email: david.novoa@udc.es); (2) Oscar Fontenla-Romero, Universidade da Coruña, CITIC, Campus de Elviña s/n, 15008, A Coruña, Spain (Email: oscar.fontenla@udc.es); (3) Bertha Guijarro-Berdiñas, Universidade da Coruña, CITIC, Campus de Elviña s/n, 15008, A Coruña, Spain (Email: berta.guijarro@udc.es). Author: Author: (1) David Novoa-Paradela, Universidade da Coruña, CITIC, Campus de Elviña s/n, 15008, A Coruña, Spain & Corresponding author (Email: david.novoa@udc.es); (2) Oscar Fontenla-Romero, Universidade da Coruña, CITIC, Campus de Elviña s/n, 15008, A Coruña, Spain (Email: oscar.fontenla@udc.es); (3) Bertha Guijarro-Berdiñas, Universidade da Coruña, CITIC, Campus de Elviña s/n, 15008, A Coruña, Spain (Email: berta.guijarro@udc.es). Table of Links Abstract and Introduction Abstract and Introduction Related work Related work The proposed pipeline The proposed pipeline Evaluation Evaluation Conclusion & Acknowledgements Conclusion & Acknowledgements References & Appendix References & Appendix 3. The proposed pipeline The purpose of the proposed pipeline is, given the text reviews of an Amazon product, to classify them as normal if they refer to it, and as anomalous if they describe different products or if they are so generic that they do not provide useful information to consumers. These classifications will be accompanied by a normality score and explanations that justify them. Figure 1 shows the modules that are part of the proposed pipeline to solve this task. In the first phase, we encode the text reviews of the target product using the pretrained MPNet transformer [25]. In the second phase, a DAEF [26] autoencoder is trained using these embeddings. Using the reconstruction errors issued by the network and a predefined threshold error, we can classify reviews as normal (error greater than the threshold) or anomalous (error less than the threshold). For the last phase, we propose a method based on the most frequent normal terms to generate an explanation associated with each classification. Therefore, the proposed pipeline is composed of three main modules: (1) Text encoding; (2) Anomaly detection; (3) Explainability. The following sections describe each of them in detail. 3.1. Text encoding 3.1. Text encoding Although there are models capable of dealing directly with images or text, machine learning models are usually designed to be trained using numerical vectors that represent the sample data (tabular data). This input format of the data is the usual one for classic anomaly detection methods. In the NLP area, there are multiple techniques to represent texts using vectors of real numbers, which are known as embeddings. These techniques allow the generation of vector spaces that try to represent the relationships and semantic similarities of the language, so that, for example, two synonymous words will be found at a shorter distance in the vector space than two unrelated words. Embeddings can be calculated independently for each word of the language (word embeddings), which led to models such as word2vec [27] or Global Vectors (GloVe) [28]. The representation of a sentence (sentence embedding) or a document (document embedding) will therefore be the sum of all the individual representations of the terms that make it up. To obtain representations of fixed length, it is usual to perform operations such as the mean. In certain cases, these operations between embeddings can worsen or even invalidate the final embedding, so specific models have been developed capable of understanding and representing a text as a whole, instead of just encoding it word by word. Among these models are those based on transformers [29], such as BERT [30], XLNet [31], GPT-3 [32] or GPT-4 [33], which have been trained over large-scale datasets and can solve different tasks, including sentence embedding. Since it inherits the advantages of the BERT and XLNet models while overcoming their limitations, we will use the pre-trained MPNet [25] model to calculate the embeddings of the reviews. MPNet combines Masked Language Modeling (MLM) and Permuted Language Modeling (PLM) to predict token dependencies, using auxiliary position information as input to enable the model to view a complete sentence and reduce position differences. Models like GPT-3 or GPT-4 are more advanced, but in addition to not being open source, they demand much higher computational resources, unaffordable for a significant part of the scientific community, as is our particular case. MPNet maps sentences and paragraphs to a 768 dimensional dense vector space e ∈ R 768×1 , providing fast and quality encodings. Computing time is a critical aspect in the area in which this work is framed, since in e-commerce platforms (and online reviews in general) we can find a huge number of products and reviews to deal with. Therefore, the proposed pipeline uses MPNet to calculate the embeddings of the reviews, which allows them to be represented as numeric vectors of fixed size. This model and others are available in the Hugging Face repository [34]. 3.2. Anomaly detection 3.2. Anomaly detection Anomaly detection is a field with a large number of algorithms that solve the problem of distinguishing between normal and anomalous instances in a wide variety of ways [4, 35]. Depending on the assumptions made and the processes they employ, we can distinguish between five main types of methods: probabilistic, distance-based, information theory-based, boundarybased, and reconstruction-based methods. Among the latter are autoencoder networks (AE) [36], one of the most widely used models. AE are a type of self-associative neural network whose output layer seeks to reproduce the data presented to the input layer after having gone through a dimensional compression phase. In this way, they manage to obtain a representation of the input data in a space with a dimension smaller than the original, learning a compact representation of the data, retaining the important information, and compressing the redundant one. One of the requirements of our proposed pipeline is to generate a normality score for each review. This score will allow us, among other things, to order the different reviews by their level of normality. When AE networks are used in anomaly detection scenarios, the classification is usually carried out based on the reconstruction error that they emit to reproduce in its output the embeddings of the reviews it receives as inputs, which represents the level of normality of the evaluated instance. This reconstruction error can be used as the normality score we are looking for. Due to the speed of its training, we have decided to use DAEF (Deep Autoencoder for Federated learning) [26] as our anomaly detection model. Unlike traditional neural networks, DAEF trains a deep AE network in a non-iterative way, which drastically reduces its training time without losing the performance of traditional (iterative) AEs. The proposed pipeline uses DAEF to calculate the embeddings of the reviews, being able to issue a normality score associated with each of the review classifications. 3.3. Explainability 3.3. Explainability As black-box ML models are mainly being employed to make important predictions in critical contexts, the demand for transparency is increasing. The danger is in creating and using decisions that are not justifiable, legitimate, or that do not allow obtaining detailed explanations of their behaviour. Explanations supporting the output of a model are crucial, moreover in fields such as medicine, autonomous vehicles, security, or finance. When ML models do not meet any of the criteria imposed to declare them explainable, a separate method must be applied to the model to explain its decisions. This is the purpose of post-hoc explainability techniques [37], which aim at communicating understandable information about how an already developed model produces its predictions for any given input. Within these techniques, the so-called model-agnostic techniques are those designed to be plugged into any model with the intent of extracting some information from its prediction procedure. To generate the explanations we have implemented an approach based on a statistical analysis of the dataset. Based on the definitions of normal and anomalous review presented in Section 2, Our hypothesis assumes that normal reviews will always refer directly or indirectly to the target product so that there will be a list of terms used very frequently among normal texts. The appearance of one or more of these “normal” terms in a review would justify its classification by the system as normal. In the same way, anomalous reviews may be justified with the non-presence of said terms. The original text reviews classified as normal using the AD model will be processed (tokenization, lemmatization and stopword removal) and analyzed to obtain the list of the top-n most frequent terms. This list of terms will be compared with the lists of frequent terms of other products in order to remove the terms they have in common, so that the final list for a product does not contain generic terms. This final list of terms will be used to generate the explanations. New reviews classified as normal will be searched for the presence of any of these terms. To make this search more flexible, it will also be analyzed if there are terms in the review semantically close to any of the words on the list. To quantify the closeness of two words, the cosine similarity emitted by the MPNet transformer model itself will be used as a measure of distance. Reviews classified as anomalous will be explained by the non-occurrence of terms from the list. Figure 2 represents an overall view of the pipeline considering the product “chocolate bars” as the normal class. In summary, in this work, we propose the use of this statistical analysis of the terms to generate the explanations associated with the classification of the reviews. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Explainable AI in Action: Generating Insights from Review Anomalies

Decentralized Society + UBI

Authors:
(1) P Aditya Sreekar, Amazon and these authors contributed equally to this work {sreekarp@amazon.com};
(2) Sahil Verm, Amazon and these authors contributed equally to this work {vrsahil@amazon.com;}
(3) Varun Madhavan, Indian Institute of Technology, Kharagpur. Work done during internship at Amazon {varunmadhavan@iitkgp.ac.in};
(4) Abhishek Persad, Amazon {persadap@amazon.com}. Table of Links Abstract and Introduction
Related Works
Methodology
Experiment
Conclusion and Future Work
References 2. Related Works Tree-based algorithms are widely used in machine learning for tabular data. Decision trees recursively split the data into multiple parts based on axis-aligned hyper-planes (Hastie et al., 2009). Random Forests (RF) (Breiman, 2001) and Gradient Boosted Decision Trees (GBDT) (Friedman, 2001) are the most commonly used tree based ensembles. RF fits multiple decision trees on random subsets of the data and averages/polls the predictions to alleviate the overfitting characteristic of decision trees. GBDT, XGBoost (Chen and Guestrin, 2016), and CatBoost (Prokhorenkova et al., 2018) are boosted ensemble models that sequentially build decision trees to correct errors made by previous trees, leading to improved performance on complex datasets with non-linear relations. Recently, there has been a lot of interest in deep learning models for tabular data. Some methods introduce differentiable approximations of decision functions used in decision trees to make them differentiable (Hazimeh et al., 2020; Popov et al., 2019). These methods outperform pure tree based problem for some problem statements, however, they are not consistently better (Gorishniy et al., 2021). Other methods have used attention mechanisms to adapt DL methods to tabular data (Arik et al., 2019; Huang et al., 2020; Gorishniy et al., 2021; Somepalli et al., 2021; Chen et al., 2022). TabNet (Arik et al., 2019) proposes a sparse attention mechanism that is stacked in multiple layers to mimic the recursive splitting of decision trees. Inspired from the success of self-attention transformers (Vaswani et al., 2017) in many domains (Devlin et al., 2019; Dosovitskiy et al., 2021; Gong et al., 2021) methods like TabTransformer (Huang et al., 2020), FT-Transformer (Gorishniy et al., 2021) and SAINT (Somepalli et al., 2021) were proposed. TabTransformer embeds all categorical variables into a unified embedding space, and a sentence of categorical embeddings is passed through self-attention transformer layers. FT-Transformer further extends this by attending to numerical features as well, by using continuous embedding. SAINT builds on FT-Transformer by proposing a new kind of attention which captures interactions between samples of a batch. However, SAINT does not provide any advantage over FT-Transformer for our problem statement, because intersample attention is only effective when the number of dimensions is higher in comparision to the number of samples, thus we do not compare RCT against SAINT (Somepalli et al., 2021). This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) P Aditya Sreekar, Amazon and these authors contributed equally to this work {sreekarp@amazon.com}; (2) Sahil Verm, Amazon and these authors contributed equally to this work {vrsahil@amazon.com;} (3) Varun Madhavan, Indian Institute of Technology, Kharagpur. Work done during internship at Amazon {varunmadhavan@iitkgp.ac.in}; (4) Abhishek Persad, Amazon {persadap@amazon.com}. Authors: Authors: (1) P Aditya Sreekar, Amazon and these authors contributed equally to this work {sreekarp@amazon.com}; (2) Sahil Verm, Amazon and these authors contributed equally to this work {vrsahil@amazon.com;} (3) Varun Madhavan, Indian Institute of Technology, Kharagpur. Work done during internship at Amazon {varunmadhavan@iitkgp.ac.in}; (4) Abhishek Persad, Amazon {persadap@amazon.com}. Table of Links Abstract and Introduction Related Works Methodology Experiment Conclusion and Future Work References Abstract and Introduction Abstract and Introduction Related Works Related Works Methodology Methodology Experiment Experiment Conclusion and Future Work Conclusion and Future Work References References 2. Related Works Tree-based algorithms are widely used in machine learning for tabular data. Decision trees recursively split the data into multiple parts based on axis-aligned hyper-planes (Hastie et al., 2009). Random Forests (RF) (Breiman, 2001) and Gradient Boosted Decision Trees (GBDT) (Friedman, 2001) are the most commonly used tree based ensembles. RF fits multiple decision trees on random subsets of the data and averages/polls the predictions to alleviate the overfitting characteristic of decision trees. GBDT, XGBoost (Chen and Guestrin, 2016), and CatBoost (Prokhorenkova et al., 2018) are boosted ensemble models that sequentially build decision trees to correct errors made by previous trees, leading to improved performance on complex datasets with non-linear relations. Recently, there has been a lot of interest in deep learning models for tabular data. Some methods introduce differentiable approximations of decision functions used in decision trees to make them differentiable (Hazimeh et al., 2020; Popov et al., 2019). These methods outperform pure tree based problem for some problem statements, however, they are not consistently better (Gorishniy et al., 2021). Other methods have used attention mechanisms to adapt DL methods to tabular data (Arik et al., 2019; Huang et al., 2020; Gorishniy et al., 2021; Somepalli et al., 2021; Chen et al., 2022). TabNet (Arik et al., 2019) proposes a sparse attention mechanism that is stacked in multiple layers to mimic the recursive splitting of decision trees. Inspired from the success of self-attention transformers (Vaswani et al., 2017) in many domains (Devlin et al., 2019; Dosovitskiy et al., 2021; Gong et al., 2021) methods like TabTransformer (Huang et al., 2020), FT-Transformer (Gorishniy et al., 2021) and SAINT (Somepalli et al., 2021) were proposed. TabTransformer embeds all categorical variables into a unified embedding space, and a sentence of categorical embeddings is passed through self-attention transformer layers. FT-Transformer further extends this by attending to numerical features as well, by using continuous embedding. SAINT builds on FT-Transformer by proposing a new kind of attention which captures interactions between samples of a batch. However, SAINT does not provide any advantage over FT-Transformer for our problem statement, because intersample attention is only effective when the number of dimensions is higher in comparision to the number of samples, thus we do not compare RCT against SAINT (Somepalli et al., 2021). This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

Unveiling the Power of Self-Attention for Shipping Cost Prediction: Related Works

শিপিং খরচ পূর্বাভাসের জন্য স্ব-মনোযোগের শক্তি উন্মোচন: সম্পর্কিত কাজ

Die Macht der Selbstaufmerksamkeit zur Vorhersage von Versandkosten enthüllen: Verwandte Arbeiten

Revelando el poder de la autoatención para la predicción de costos de envío: trabajos relacionados

Dévoiler le pouvoir de l’auto-attention pour la prévision des coûts d’expédition : travaux connexes

शिपिंग लागत की भविष्यवाणी के लिए स्व-ध्यान की शक्ति का अनावरण: संबंधित कार्य

配送コスト予測における自己注意の力を明らかにする: 関連研究

배송비 예측을 위한 자기주의의 힘 공개: 관련 연구

Revelando o poder da autoatenção para previsão de custos de envio: trabalhos relacionados

Раскрытие силы самовнимания для прогнозирования стоимости доставки: сопутствующие работы

Nakliye Maliyeti Tahmininde Kişisel Dikkatin Gücünü Ortaya Çıkarma: İlgili Çalışmalar

Khám phá sức mạnh của sự chú ý trong việc dự đoán chi phí vận chuyển: Các tác phẩm liên quan

揭示自注意力机制在运输成本预测中的作用：相关研究

Authors:
(1) P Aditya Sreekar, Amazon and these authors contributed equally to this work {sreekarp@amazon.com};
(2) Sahil Verm, Amazon and these authors contributed equally to this work {vrsahil@amazon.com;}
(3) Varun Madhavan, Indian Institute of Technology, Kharagpur. Work done during internship at Amazon {varunmadhavan@iitkgp.ac.in};
(4) Abhishek Persad, Amazon {persadap@amazon.com}. Table of Links Abstract and Introduction
Related Works
Methodology
Experiment
Conclusion and Future Work
References References Alexander Amini, Wilko Schwarting, Ava Soleimany, and Daniela Rus. Deep evidential regression. CoRR, abs/1910.02600, 2019. URL http://arxiv.org/abs/1910.02600. Sercan O Arik, Engin Gedik, Kenan Guney, and Umut Atilla. Tabnet: Attentive interpretable tabular learning. In Advances in Neural Information Processing Systems, pages 10951–10961, 2019. Christopher M Bishop. Pattern recognition and machine learning. In Springer, chapter 2, pages 36–43. 2006. Leo Breiman. Random forests. Machine learning, 45(1):5–32, 2001. Jintai Chen, Kuanlun Liao, Yao Wan, Danny Z Chen, and Jian Wu. Danets: Deep abstract networks for tabular data classification and regression. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 3930–3938, 2022. Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 785–794, 2016. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171–4186. Association for Computational Linguistics, 2019. doi: 10.18653/v1/n19-1423. URL https://doi.org/10.18653/v1/n19-1423. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021. Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, and Alexander Smola. Autogluon-tabular: Robust and accurate automl for structured data, 2020. William Falcon and The PyTorch Lightning team. PyTorch Lightning, 3 2019. URL https: //github.com/Lightning-AI/lightning. Jerome H Friedman. Greedy function approximation: a gradient boosting machine. Annals of statistics, pages 1189–1232, 2001. Yuan Gong, Yu-An Chung, and James Glass. AST: Audio Spectrogram Transformer. In Proc. Interspeech 2021, pages 571–575, 2021. doi: 10.21437/Interspeech.2021-698. Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems, 34:18932–18943, 2021. Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The elements of statistical learning: data mining, inference, and prediction. Springer Science & Business Media, 2009. Hussein Hazimeh, Natalia Ponomareva, Petros Mol, Zhenyu Tan, and Rahul Mazumder. The tree ensemble layer: Differentiability meets conditional computation. In International Conference on Machine Learning, pages 4138–4148. PMLR, 2020. Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. Tabtransformer: Tabular data modeling using contextual embeddings. arXiv preprint arXiv:2012.06678, 2020. Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30:3146–3154, 2017. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. Daniele Micci-Barreca. A preprocessing scheme for high-cardinality categorical attributes in classification and prediction problems. SIGKDD Explor. Newsl., 3(1):27–32, jul 2001. ISSN 1931-0145. doi: 10.1145/507533.507538. URL https://doi.org/10.1145/507533. 507538. Sergei Popov, Stanislav Morozov, and Artem Babenko. Neural oblivious decision ensembles for deep learning on tabular data. arXiv preprint arXiv:1909.06312, 2019. Liudmila Prokhorenkova, Gleb Gusev, Alexey Vorobev, Anna Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. Advances in Neural Information Processing Systems, 31:6638–6648, 2018. Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C Bayan Bruss, and Tom Goldstein. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint arXiv:2106.01342, 2021. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008, 2017. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) P Aditya Sreekar, Amazon and these authors contributed equally to this work {sreekarp@amazon.com}; (2) Sahil Verm, Amazon and these authors contributed equally to this work {vrsahil@amazon.com;} (3) Varun Madhavan, Indian Institute of Technology, Kharagpur. Work done during internship at Amazon {varunmadhavan@iitkgp.ac.in}; (4) Abhishek Persad, Amazon {persadap@amazon.com}. Authors: Authors: (1) P Aditya Sreekar, Amazon and these authors contributed equally to this work {sreekarp@amazon.com}; (2) Sahil Verm, Amazon and these authors contributed equally to this work {vrsahil@amazon.com;} (3) Varun Madhavan, Indian Institute of Technology, Kharagpur. Work done during internship at Amazon {varunmadhavan@iitkgp.ac.in}; (4) Abhishek Persad, Amazon {persadap@amazon.com}. Table of Links Abstract and Introduction Related Works Methodology Experiment Conclusion and Future Work References Abstract and Introduction Abstract and Introduction Related Works Related Works Methodology Methodology Experiment Experiment Conclusion and Future Work Conclusion and Future Work References References References Alexander Amini, Wilko Schwarting, Ava Soleimany, and Daniela Rus. Deep evidential regression. CoRR, abs/1910.02600, 2019. URL http://arxiv.org/abs/1910.02600. Sercan O Arik, Engin Gedik, Kenan Guney, and Umut Atilla. Tabnet: Attentive interpretable tabular learning. In Advances in Neural Information Processing Systems, pages 10951–10961, 2019. Christopher M Bishop. Pattern recognition and machine learning. In Springer, chapter 2, pages 36–43. 2006. Leo Breiman. Random forests. Machine learning, 45(1):5–32, 2001. Jintai Chen, Kuanlun Liao, Yao Wan, Danny Z Chen, and Jian Wu. Danets: Deep abstract networks for tabular data classification and regression. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 3930–3938, 2022. Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 785–794, 2016. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171–4186. Association for Computational Linguistics, 2019. doi: 10.18653/v1/n19-1423. URL https://doi.org/10.18653/v1/n19-1423. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021. Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, and Alexander Smola. Autogluon-tabular: Robust and accurate automl for structured data, 2020. William Falcon and The PyTorch Lightning team. PyTorch Lightning, 3 2019. URL https: //github.com/Lightning-AI/lightning. Jerome H Friedman. Greedy function approximation: a gradient boosting machine. Annals of statistics, pages 1189–1232, 2001. Yuan Gong, Yu-An Chung, and James Glass. AST: Audio Spectrogram Transformer. In Proc. Interspeech 2021, pages 571–575, 2021. doi: 10.21437/Interspeech.2021-698. Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems, 34:18932–18943, 2021. Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The elements of statistical learning: data mining, inference, and prediction. Springer Science & Business Media, 2009. Hussein Hazimeh, Natalia Ponomareva, Petros Mol, Zhenyu Tan, and Rahul Mazumder. The tree ensemble layer: Differentiability meets conditional computation. In International Conference on Machine Learning, pages 4138–4148. PMLR, 2020. Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. Tabtransformer: Tabular data modeling using contextual embeddings. arXiv preprint arXiv:2012.06678, 2020. Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30:3146–3154, 2017. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. Daniele Micci-Barreca. A preprocessing scheme for high-cardinality categorical attributes in classification and prediction problems. SIGKDD Explor. Newsl., 3(1):27–32, jul 2001. ISSN 1931-0145. doi: 10.1145/507533.507538. URL https://doi.org/10.1145/507533. 507538. Sergei Popov, Stanislav Morozov, and Artem Babenko. Neural oblivious decision ensembles for deep learning on tabular data. arXiv preprint arXiv:1909.06312, 2019. Liudmila Prokhorenkova, Gleb Gusev, Alexey Vorobev, Anna Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. Advances in Neural Information Processing Systems, 31:6638–6648, 2018. Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C Bayan Bruss, and Tom Goldstein. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint arXiv:2106.01342, 2021. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008, 2017. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

Unveiling the Power of Self-Attention for Shipping Cost Prediction: References

শিপিং খরচ পূর্বাভাসের জন্য স্ব-মনোযোগের শক্তি উন্মোচন: তথ্যসূত্র

Die Macht der Selbstaufmerksamkeit zur Vorhersage von Versandkosten enthüllen: Referenzen

Revelando el poder de la autoatención para la predicción de costos de envío: referencias

Dévoiler le pouvoir de l’auto-attention pour la prévision des coûts d’expédition : références

शिपिंग लागत की भविष्यवाणी के लिए स्व-ध्यान की शक्ति का अनावरण: संदर्भ

配送コスト予測における自己注意の力を明らかにする: 参考文献

배송비 예측을 위한 자기주의의 힘 공개: 참고자료

Revelando o poder da autoatenção para previsão de custos de envio: referências

Раскрытие силы самовнимания для прогнозирования стоимости доставки: ссылки

Nakliye Maliyeti Tahmininde Kişisel Dikkatin Gücünü Ortaya Çıkarma: Referanslar

Khám phá sức mạnh của sự chú ý trong việc dự đoán chi phí vận chuyển: Tài liệu tham khảo

揭示自注意力机制在运输成本预测中的作用：参考文献

Authors:
(1) P Aditya Sreekar, Amazon and these authors contributed equally to this work {sreekarp@amazon.com};
(2) Sahil Verm, Amazon and these authors contributed equally to this work {vrsahil@amazon.com;}
(3) Varun Madhavan, Indian Institute of Technology, Kharagpur. Work done during internship at Amazon {varunmadhavan@iitkgp.ac.in};
(4) Abhishek Persad, Amazon {persadap@amazon.com}. Table of Links Abstract and Introduction
Related Works
Methodology
Experiment
Conclusion and Future Work
References Abstract Amazon ships billions of packages to its customers annually within the United States. Shipping cost of these packages are used on the day of shipping (day 0) to estimate profitability of sales. Downstream systems utilize these days 0 profitability estimates to make financial decisions, such as pricing strategies and delisting loss-making products. However, obtaining accurate shipping cost estimates on day 0 is complex for reasons like delay in carrier invoicing or fixed cost components getting recorded at monthly cadence. Inaccurate shipping cost estimates can lead to bad decision, such as pricing items too low or high, or promoting the wrong product to the customers. Current solutions for estimating shipping costs on day 0 rely on tree-based models that require extensive manual engineering efforts. In this study, we propose a novel architecture called the Rate Card Transformer (RCT) that uses self-attention to encode all package shipping information such as package attributes, carrier information and route plan. Unlike other transformer-based tabular models, RCT has the ability to encode a variable list of one-to-many relations of a shipment, allowing it to capture more information about a shipment. For example, RCT can encode properties of all products in a package. Our results demonstrate that cost predictions made by the RCT have 28.82% less error compared to tree-based GBDT model. Moreover, the RCT outperforms the state-of-the-art transformer-based tabular model, FTTransformer, by 6.08%. We also illustrate that the RCT learns a generalized manifold of the rate card that can improve the performance of tree-based models. 1. Introduction Amazon ships packages in the order of billions annually to its customers in the United States alone. The route planning for these packages is done on the day of shipping, day 0. As part of this plan, the shipping cost for each package is estimated by breaking down the package journey into smaller legs, and calculating the cost of each leg using a rate card. Day cost estimates are used to compute initial profitability estimates for accounting purposes, e.g.the estimate of profit/loss for each item as a result of a specific sale to a customer. These profitability estimates are used by several downstream services for decision making and planning. However, the day 0 estimates may differ from the actual cost due to factors like improper rate-card configuration, incorrect package dimensions, wrong delivery address, etc. Inaccurate cost estimates cause skewed profitability estimates, which in turn leads to suboptimal financial decisions by downstream systems. For example, if the shipping cost of an item is consistently overestimated, then the item could be removed from the catalog. On the other hand, underestimated cost can lead pricing systems to lower the price of the item, leading to losses. Further, inaccurate estimation also leads us to promote wrong products to the customer, causing bad customer experience. To improve these shipping cost estimates, we propose a Transformer based deep learning model that accurately predicts the shipping cost at day 0. In the context of shipping, a package is characterized by its physical dimensions, weight, and contents. It also include details about the carrier responsible for transporting it and the intended route. Additionally, a package is associated with a variable number of attributes that describe the item(s) inside and the various charges related to its shipment. Collectively, we refer to these attributes as the rate card associated with the package. For tabular datasets like package rate cards, tree based models like Gradient Boosted Decision Trees (GBDT), XGBoost (Chen and Guestrin, 2016), etc., are considered as state-of-the-art (SOTA) models. However, their effectiveness heavily relies on high-quality input features (Arik et al., 2019) which can require extensive feature engineering. For our use case, this problem is further accentuated by the fact that the target concept depends on high order combinatorial interactions between rate card attributes. For example, if the rate card is improperly configured for large containers with flammable substances shipped from Washington DC to New York by ABC carrier, then the model has to learn to associate property combination < size = large, item = flammable, source = W ashington, destination = New Y ork, carrier = ABC > with high deviation between estimated and actual costs. When dealing with feature combinations, considering all possible higher-order interactions between package properties may be impractical due to the exponential increase in the number of interactions with each increase in order, leading to the curse of dimensionality (Bishop, 2006). Another shortcoming of tree based models is their inability to handle a variable length list of features. A package may contain multiple items, and its ship cost can be broken down into multiple charges types. Previous experiments demonstrated that adding features engineered from multiple items and charges improved GBDT’s performance. However, due to inability of tree based models to handle variable list of features, complete information from them could not be learned. In this paper, inspired by the recent success of transformers in tabular domain (Huang et al., 2020; Somepalli et al., 2021; Gorishniy et al., 2021), we propose a novel architecture called the Rate Card Transformer (RCT) to predict ship cost on day 0. The proposed model is specifically designed to learn an embedding of rate card associated with a package. The RCT leverages self-attention mechanisms to effectively capture the interdependencies between various components in a rate card by learning interactions between input features. Specifically, our contributions in this work include: • Propose a novel architecture, Rate Card Transformer (RCT), which leverages transformer architecture to learn a manifold of the rate card, to predict shipping cost on Day 0. Further, it is demonstrated that RCT outperforms both GBDTs and the stateof-the-art tabular transformer, FT-Transformer, (Gorishniy et al., 2021) in shipping cost prediction. • Extensive experiments are performed to show that the learned embeddings are a sufficient representation of the rate card manifold, and self-attention layers are effective feature interaction learners. Ablation studies are performed to analyze the impact of number of transformer layers and attention heads on model performance. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) P Aditya Sreekar, Amazon and these authors contributed equally to this work {sreekarp@amazon.com}; (2) Sahil Verm, Amazon and these authors contributed equally to this work {vrsahil@amazon.com;} (3) Varun Madhavan, Indian Institute of Technology, Kharagpur. Work done during internship at Amazon {varunmadhavan@iitkgp.ac.in}; (4) Abhishek Persad, Amazon {persadap@amazon.com}. Authors: Authors: (1) P Aditya Sreekar, Amazon and these authors contributed equally to this work {sreekarp@amazon.com}; (2) Sahil Verm, Amazon and these authors contributed equally to this work {vrsahil@amazon.com;} (3) Varun Madhavan, Indian Institute of Technology, Kharagpur. Work done during internship at Amazon {varunmadhavan@iitkgp.ac.in}; (4) Abhishek Persad, Amazon {persadap@amazon.com}. Table of Links Abstract and Introduction Related Works Methodology Experiment Conclusion and Future Work References Abstract and Introduction Abstract and Introduction Related Works Related Works Methodology Methodology Experiment Experiment Conclusion and Future Work Conclusion and Future Work References References Abstract Amazon ships billions of packages to its customers annually within the United States. Shipping cost of these packages are used on the day of shipping (day 0) to estimate profitability of sales. Downstream systems utilize these days 0 profitability estimates to make financial decisions, such as pricing strategies and delisting loss-making products. However, obtaining accurate shipping cost estimates on day 0 is complex for reasons like delay in carrier invoicing or fixed cost components getting recorded at monthly cadence. Inaccurate shipping cost estimates can lead to bad decision, such as pricing items too low or high, or promoting the wrong product to the customers. Current solutions for estimating shipping costs on day 0 rely on tree-based models that require extensive manual engineering efforts. In this study, we propose a novel architecture called the Rate Card Transformer (RCT) that uses self-attention to encode all package shipping information such as package attributes, carrier information and route plan. Unlike other transformer-based tabular models, RCT has the ability to encode a variable list of one-to-many relations of a shipment, allowing it to capture more information about a shipment. For example, RCT can encode properties of all products in a package. Our results demonstrate that cost predictions made by the RCT have 28.82% less error compared to tree-based GBDT model. Moreover, the RCT outperforms the state-of-the-art transformer-based tabular model, FTTransformer, by 6.08%. We also illustrate that the RCT learns a generalized manifold of the rate card that can improve the performance of tree-based models. 1. Introduction Amazon ships packages in the order of billions annually to its customers in the United States alone. The route planning for these packages is done on the day of shipping, day 0. As part of this plan, the shipping cost for each package is estimated by breaking down the package journey into smaller legs, and calculating the cost of each leg using a rate card. Day cost estimates are used to compute initial profitability estimates for accounting purposes, e.g.the estimate of profit/loss for each item as a result of a specific sale to a customer. These profitability estimates are used by several downstream services for decision making and planning. However, the day 0 estimates may differ from the actual cost due to factors like improper rate-card configuration, incorrect package dimensions, wrong delivery address, etc. Inaccurate cost estimates cause skewed profitability estimates, which in turn leads to suboptimal financial decisions by downstream systems. For example, if the shipping cost of an item is consistently overestimated, then the item could be removed from the catalog. On the other hand, underestimated cost can lead pricing systems to lower the price of the item, leading to losses. Further, inaccurate estimation also leads us to promote wrong products to the customer, causing bad customer experience. To improve these shipping cost estimates, we propose a Transformer based deep learning model that accurately predicts the shipping cost at day 0. In the context of shipping, a package is characterized by its physical dimensions, weight, and contents. It also include details about the carrier responsible for transporting it and the intended route. Additionally, a package is associated with a variable number of attributes that describe the item(s) inside and the various charges related to its shipment. Collectively, we refer to these attributes as the rate card associated with the package. For tabular datasets like package rate cards, tree based models like Gradient Boosted Decision Trees (GBDT), XGBoost (Chen and Guestrin, 2016), etc., are considered as state-of-the-art (SOTA) models. However, their effectiveness heavily relies on high-quality input features (Arik et al., 2019) which can require extensive feature engineering. For our use case, this problem is further accentuated by the fact that the target concept depends on high order combinatorial interactions between rate card attributes. For example, if the rate card is improperly configured for large containers with flammable substances shipped from Washington DC to New York by ABC carrier, then the model has to learn to associate property combination < size = large, item = flammable, source = W ashington, destination = New Y ork, carrier = ABC > with high deviation between estimated and actual costs. When dealing with feature combinations, considering all possible higher-order interactions between package properties may be impractical due to the exponential increase in the number of interactions with each increase in order, leading to the curse of dimensionality (Bishop, 2006). Another shortcoming of tree based models is their inability to handle a variable length list of features. A package may contain multiple items, and its ship cost can be broken down into multiple charges types. Previous experiments demonstrated that adding features engineered from multiple items and charges improved GBDT’s performance. However, due to inability of tree based models to handle variable list of features, complete information from them could not be learned. In this paper, inspired by the recent success of transformers in tabular domain (Huang et al., 2020; Somepalli et al., 2021; Gorishniy et al., 2021), we propose a novel architecture called the Rate Card Transformer (RCT) to predict ship cost on day 0. The proposed model is specifically designed to learn an embedding of rate card associated with a package. The RCT leverages self-attention mechanisms to effectively capture the interdependencies between various components in a rate card by learning interactions between input features. Specifically, our contributions in this work include: • Propose a novel architecture, Rate Card Transformer (RCT), which leverages transformer architecture to learn a manifold of the rate card, to predict shipping cost on Day 0. Further, it is demonstrated that RCT outperforms both GBDTs and the stateof-the-art tabular transformer, FT-Transformer, (Gorishniy et al., 2021) in shipping cost prediction. • Extensive experiments are performed to show that the learned embeddings are a sufficient representation of the rate card manifold, and self-attention layers are effective feature interaction learners. Ablation studies are performed to analyze the impact of number of transformer layers and attention heads on model performance. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

Unveiling the Power of Self-Attention for Shipping Cost Prediction: Abstract and Introduction

শিপিং খরচ পূর্বাভাসের জন্য স্ব-মনোযোগের শক্তি উন্মোচন: বিমূর্ত এবং ভূমিকা

Die Macht der Selbstaufmerksamkeit zur Vorhersage von Versandkosten enthüllen: Zusammenfassung und Einführung

Revelando el poder de la autoatención para la predicción de costos de envío: resumen e introducción

Dévoiler le pouvoir de l’auto-attention pour la prévision des coûts d’expédition : résumé et introduction

शिपिंग लागत की भविष्यवाणी के लिए स्व-ध्यान की शक्ति का अनावरण: सार और परिचय

配送コスト予測における自己注意の力を明らかにする: 概要と概要

배송비 예측을 위한 자기주의의 힘 공개: 요약 및 소개

Revelando o poder da autoatenção para previsão de custos de envio: resumo e introdução

Раскрытие силы самовнимания для прогнозирования стоимости доставки: Краткое содержание и введение

Nakliye Maliyeti Tahmininde Kişisel Dikkatin Gücünü Ortaya Çıkarma: Özet ve Giriş

Khám phá sức mạnh của sự chú ý trong việc dự đoán chi phí vận chuyển: Tóm tắt và giới thiệu

揭示自注意力机制在运输成本预测中的作用：摘要和简介

Authors:
(1) Sasun Hambardzumyan, Activeloop, Mountain View, CA, USA;
(2) Abhinav Tuli, Activeloop, Mountain View, CA, USA;
(3) Levon Ghukasyan, Activeloop, Mountain View, CA, USA;
(4) Fariz Rahman, Activeloop, Mountain View, CA, USA;.
(5) Hrant Topchyan, Activeloop, Mountain View, CA, USA;
(6) David Isayan, Activeloop, Mountain View, CA, USA;
(7) Mark McQuade, Activeloop, Mountain View, CA, USA;
(8) Mikayel Harutyunyan, Activeloop, Mountain View, CA, USA;
(9) Tatevik Hakobyan, Activeloop, Mountain View, CA, USA;
(10) Ivo Stranic, Activeloop, Mountain View, CA, USA;
(11) Davit Buniatyan, Activeloop, Mountain View, CA, USA. Table of Links Abstract and Intro
Current Challenges
Tensor Storage Format
Deep Lake System Overview
Machine Learning Use Cases
Performance Benchmarks
Discussion and Limitations
Related Work
Conclusions, Acknowledgement, and References 7. DISCUSSION AND LIMITATIONS Deep Lake’s primary use cases include (a) Deep Learning Model Training, (b) Data Lineage and Version Control, (c) Data Querying, and Analytics, (d) Data Inspection and Quality Control. We took NumPy [55] arrays as a fundamental block and implemented version control, streaming dataloaders, visualization engine from scratch. 7.1 Format Design Space The Tensor Storage Format (TSF) is a binary file format designed specifically for storing tensors, which are multi-dimensional arrays of numerical values used in many machine learning and deep learning algorithms. The TSF format is designed to be efficient and compact, allowing for fast and efficient storage and access of tensor data. One key advantage of the TSF format is that it supports a wide range of tensor data types, including dynamically shaped tensors. In comparison, the Parquet [79] and Arrow [13] formats are columnar file formats that are designed for storing and processing large analytical datasets. Unlike TSF, which is specifically designed for tensor data, Parquet and Arrow are optimized for efficient storage and querying of analytical workloads on tabular and time-series data. They use columnar storage and compression techniques to minimize storage space and improve performance, making them suitable for big data applications. However, TSF has some advantages over Parquet and Arrow when it comes to tensor data. TSF can support tensor operations and efficient streaming to deep learning frameworks. Other tensor formats [18, 52, 23, 57] are efficient for massively massively parallelizable workloads as they don’t require coordination across chunks. Tensor Storage Format key trade-off is enabling to store dynamically shape arrays inside a tensor without padding memory footprint. For example, in computer vision it is very common to store multiple images with different shapes or videos have dynamic length. To support the flexibility, minor overhead is introduced in the form of previously discussed chunk encoder that in practice we haven’t observed impact on production workloads. 7.2 Dataloader Deep Lake achieves state-of-the-art results in local and remote settings, as seen in benchmarks for iterating on large images Fig. 7. Primarily, it has been faster than FFCV [39], which claimed a reduction of ImageNet model training up to 98 cents per model training. Furthermore, Deep Lake achieves similar ingestion performance to WebDataset [19]. Deep Lake significantly outperforms on larger images. Parquet is optimized for small cells and analytical workloads, while Deep Lake is optimized for large, dynamically shaped tensorial data. Compared to other data lake solutions, its minimal python package design enables Deep Lake to be easily integrated into large-scale distributed training or inference workloads. 7.3 Future work The current implementation of Deep Lake has opportunities for further improvement. Firstly, the storage format does not support custom ordering for an even more efficient storage layout required for vector search or key-value indexing. Secondly, Deep Lake implements branch-based locks for concurrent access. Similar to Delta ACID transaction model [27], Deep Lake can be extended to highlyperformant parallel workloads. Thirdly, the current implementation of TQL only supports a subset of SQL operations (i.e., does not support operations such as join). Further work will focus on making it SQL-complete, extending to more numeric operations, running federated queries in external data sources and benchmarking against SQL engines. This paper is available on arxiv under CC 4.0 license. Authors: (1) Sasun Hambardzumyan, Activeloop, Mountain View, CA, USA; (2) Abhinav Tuli, Activeloop, Mountain View, CA, USA; (3) Levon Ghukasyan, Activeloop, Mountain View, CA, USA; (4) Fariz Rahman, Activeloop, Mountain View, CA, USA;. (5) Hrant Topchyan, Activeloop, Mountain View, CA, USA; (6) David Isayan, Activeloop, Mountain View, CA, USA; (7) Mark McQuade, Activeloop, Mountain View, CA, USA; (8) Mikayel Harutyunyan, Activeloop, Mountain View, CA, USA; (9) Tatevik Hakobyan, Activeloop, Mountain View, CA, USA; (10) Ivo Stranic, Activeloop, Mountain View, CA, USA; (11) Davit Buniatyan, Activeloop, Mountain View, CA, USA. Authors: Authors: (1) Sasun Hambardzumyan, Activeloop, Mountain View, CA, USA; (2) Abhinav Tuli, Activeloop, Mountain View, CA, USA; (3) Levon Ghukasyan, Activeloop, Mountain View, CA, USA; (4) Fariz Rahman, Activeloop, Mountain View, CA, USA;. (5) Hrant Topchyan, Activeloop, Mountain View, CA, USA; (6) David Isayan, Activeloop, Mountain View, CA, USA; (7) Mark McQuade, Activeloop, Mountain View, CA, USA; (8) Mikayel Harutyunyan, Activeloop, Mountain View, CA, USA; (9) Tatevik Hakobyan, Activeloop, Mountain View, CA, USA; (10) Ivo Stranic, Activeloop, Mountain View, CA, USA; (11) Davit Buniatyan, Activeloop, Mountain View, CA, USA. Table of Links Abstract and Intro Current Challenges Tensor Storage Format Deep Lake System Overview Machine Learning Use Cases Performance Benchmarks Discussion and Limitations Related Work Conclusions, Acknowledgement, and References Abstract and Intro Abstract and Intro Current Challenges Current Challenges Tensor Storage Format Tensor Storage Format Deep Lake System Overview Deep Lake System Overview Machine Learning Use Cases Machine Learning Use Cases Performance Benchmarks Performance Benchmarks Discussion and Limitations Discussion and Limitations Related Work Related Work Conclusions, Acknowledgement, and References Conclusions, Acknowledgement, and References 7. DISCUSSION AND LIMITATIONS Deep Lake’s primary use cases include (a) Deep Learning Model Training, (b) Data Lineage and Version Control, (c) Data Querying, and Analytics, (d) Data Inspection and Quality Control. We took NumPy [55] arrays as a fundamental block and implemented version control, streaming dataloaders, visualization engine from scratch. 7.1 Format Design Space The Tensor Storage Format (TSF) is a binary file format designed specifically for storing tensors, which are multi-dimensional arrays of numerical values used in many machine learning and deep learning algorithms. The TSF format is designed to be efficient and compact, allowing for fast and efficient storage and access of tensor data. One key advantage of the TSF format is that it supports a wide range of tensor data types, including dynamically shaped tensors. In comparison, the Parquet [79] and Arrow [13] formats are columnar file formats that are designed for storing and processing large analytical datasets. Unlike TSF, which is specifically designed for tensor data, Parquet and Arrow are optimized for efficient storage and querying of analytical workloads on tabular and time-series data. They use columnar storage and compression techniques to minimize storage space and improve performance, making them suitable for big data applications. However, TSF has some advantages over Parquet and Arrow when it comes to tensor data. TSF can support tensor operations and efficient streaming to deep learning frameworks. Other tensor formats [18, 52, 23, 57] are efficient for massively massively parallelizable workloads as they don’t require coordination across chunks. Tensor Storage Format key trade-off is enabling to store dynamically shape arrays inside a tensor without padding memory footprint. For example, in computer vision it is very common to store multiple images with different shapes or videos have dynamic length. To support the flexibility, minor overhead is introduced in the form of previously discussed chunk encoder that in practice we haven’t observed impact on production workloads. 7.2 Dataloader Deep Lake achieves state-of-the-art results in local and remote settings, as seen in benchmarks for iterating on large images Fig. 7. Primarily, it has been faster than FFCV [39], which claimed a reduction of ImageNet model training up to 98 cents per model training. Furthermore, Deep Lake achieves similar ingestion performance to WebDataset [19]. Deep Lake significantly outperforms on larger images. Parquet is optimized for small cells and analytical workloads, while Deep Lake is optimized for large, dynamically shaped tensorial data. Compared to other data lake solutions, its minimal python package design enables Deep Lake to be easily integrated into large-scale distributed training or inference workloads. 7.3 Future work The current implementation of Deep Lake has opportunities for further improvement. Firstly, the storage format does not support custom ordering for an even more efficient storage layout required for vector search or key-value indexing. Secondly, Deep Lake implements branch-based locks for concurrent access. Similar to Delta ACID transaction model [27], Deep Lake can be extended to highlyperformant parallel workloads. Thirdly, the current implementation of TQL only supports a subset of SQL operations (i.e., does not support operations such as join). Further work will focus on making it SQL-complete, extending to more numeric operations, running federated queries in external data sources and benchmarking against SQL engines. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Deep Lake, a Lakehouse for Deep Learning: Discussion and Limitations

গভীর লেক, গভীর শিক্ষার জন্য একটি লেকহাউস: আলোচনা এবং সীমাবদ্ধতা

Deep Lake, ein Lakehouse für Deep Learning: Diskussion und Einschränkungen

Deep Lake, una casa de lago para el aprendizaje profundo: discusión y limitaciones

Deep Lake, un Lakehouse pour le Deep Learning : discussion et limites

डीप लेक, डीप लर्निंग के लिए एक लेकहाउस: चर्चा और सीमाएँ

ディープラーニングのためのレイクハウス「Deep Lake」：議論と限界

딥 러닝을 위한 레이크하우스인 Deep Lake: 논의와 한계

Deep Lake, um Lakehouse para Deep Learning: Discussão e Limitações

Deep Lake, домик у озера для глубокого обучения: обсуждение и ограничения

Deep Lake, Derin Öğrenme için Bir Göl Evi: Tartışma ve Sınırlamalar

Deep Lake, Lakehouse cho Deep Learning: Thảo luận và hạn chế

Deep Lake，深度学习的 Lakehouse：讨论和局限性

Stories, News & Mentions about TASBU

Tasbu’s Stories on HackerNoon

Tasbu's News & Mentions