Data Lake develops and implements cutting-edge technologies to help advance ethical medical research, and improve access to treatment.

Data Lake

Alex Merced is a developer advocate for Dremio and co-Author of O’Reilly’s “Apache Iceberg: The Definitive Guide” Apache Iceberg is disrupting the data landscape, offering a new paradigm where data is not confined to the storage system of a chosen data warehouse vendor. Instead, it resides in your own storage, accessible by multiple tools. A built on your data lake as the storage layer, offers limitless configuration possibilities. Among the various options for constructing an Apache Iceberg lakehouse, the Dremio Data Lakehouse Platform stands out as one of the most straightforward, rapid, and cost-effective choices. This platform has gained popularity for on-premises migrations, implementing data mesh strategies, enhancing BI dashboards, and more. data lakehouse, which is essentially a modular data warehouse In this article, we will explore ten reasons why the combination of Apache Iceberg and the Dremio platform is exceptionally powerful. We will delve into five reasons to choose Apache Iceberg over other table formats and five reasons to opt for the Dremio platform when considering Semantic Layers, Query Engines, and Lakehouse Management. 5 Reasons to Choose Apache Iceberg Over Other Table Formats Apache Iceberg is not the only table format available; . All three formats provide a core set of features, enabling database-like tables on your data lake with capabilities such as ACID transactions, time travel, and schema evolution. However, there are several unique aspects that make Apache Iceberg a noteworthy option to consider. Delta Lake and Apache Hudi are also key players in this domain 1. Partition Evolution Apache Iceberg distinguishes itself with a feature known as partition evolution, which allows users to modify their partitioning scheme at any time without the need to rewrite the entire table. This capability is unique to Iceberg and carries significant implications, particularly for tables at the petabyte scale, where altering partitioning can be a complex and costly process. Partition evolution facilitates the optimization of data management, as it enables users to easily revert any changes to the partitioning scheme by simply rolling back to a previous snapshot of the table. This flexibility is a considerable advantage in managing large-scale data efficiently. 2. Hidden Partitioning Apache Iceberg introduces a unique feature called hidden partitioning, which significantly simplifies the workflows for both data engineers and data analysts. In traditional partitioning approaches, data engineers often need to create additional partitioning columns derived from existing ones, which not only increases storage requirements but also complicates the data ingestion process. Additionally, data analysts must be cognizant of these extra columns; failing to filter by the partition column could lead to a full table scan, undermining efficiency. However, with Apache Iceberg's hidden partitioning, the system can partition tables based on the transformed value of a column, with this transformation tracked in the metadata, eliminating the need for physical partitioning columns in the data files. This means that analysts can apply filters directly on the original columns and still benefit from the optimized performance of partitioning. This feature streamlines operations for both data engineers and analysts, making the process more efficient and less prone to error. 3. Versioning Versioning is an invaluable feature that facilitates isolating changes, executing rollbacks, simultaneously publishing numerous changes across different objects, and creating zero-copy environments for experimentation and development. While each table format records a single chain of changes, allowing for rollbacks, Apache Iceberg uniquely incorporates branching, tagging, and merging as integral aspects of its core table format. Furthermore, Apache Iceberg stands out as the sole format presently compatible with Nessie, an open-source project that extends versioning capabilities to include commits, branches, tags, and merges at the multi-table catalog , thereby unlocking a plethora of new possibilities. level These advanced versioning features in Apache Iceberg are accessible through ergonomic SQL interfaces, making them user-friendly and easily integrated into data workflows. In contrast, other formats typically rely on file-level versioning, which necessitates the use of command-line interfaces (CLIs) and imperative programming for management, making them less approachable and more cumbersome to use. This distinction underscores Apache Iceberg's advanced capabilities and its potential to significantly enhance data management practices. 4. Lakehouse Management Apache Iceberg is attracting a growing roster of vendors eager to assist in managing tables, offering services such as compaction, sorting, snapshot cleanup, and more. This support makes using Iceberg tables as straightforward as utilizing tables in a traditional database or data warehouse. In contrast, other table formats typically rely on a single tool or vendor for data management, which can lead to vendor lock-in. With Iceberg, however, there is a diverse array of vendors, including Dremio, Tabular, Upsolver, AWS, and Snowflake, each providing varying levels of table management features. This variety gives users the flexibility to choose a vendor that best fits their needs, enhancing Iceberg's appeal as a versatile and user-friendly data management solution. 5. Open Culture One of the most persuasive arguments for adopting Apache Iceberg is its dynamic open-source culture, which permeates its development and ecosystem. Development discussions take place on publicly accessible mailing lists and emails, enabling anyone to participate in and influence the format's evolution. The ecosystem is expanding daily, with an increasing number of tools offering both read and write support, reflecting the growing enthusiasm among vendors. This open environment provides vendors with the confidence to invest their resources in supporting Iceberg, knowing they are not at the mercy of a single vendor who could unpredictably alter or restrict access to the format. This level of transparency and inclusivity not only fosters innovation and collaboration but also ensures a level of stability and predictability that is highly valued in the tech industry. Dremio Dremio is a comprehensive data lakehouse platform that consolidates numerous functionalities, typically offered by different vendors, into a single solution. It unifies data analytics through data virtualization and a semantic layer, streamlining the integration and interpretation of data from diverse sources. Dremio's robust SQL query engine is capable of federating queries across various data sources, offering transparent acceleration to enhance performance. Additionally, Dremio's suite of lakehouse management features includes a Nessie-powered data catalog, which ensures data is versioned and easily transportable, alongside automated table maintenance capabilities. This integration of multiple key features into one platform simplifies the data management process, making Dremio a powerful and efficient tool for organizations looking to harness the full potential of their data lakehouse. 5. Apache Arrow One of the key reasons Dremio's SQL query engine outperforms other distributed query engines and data warehouses is its core reliance on . Apache Arrow facilitates the swift and efficient loading of data from various sources into a unified format optimized for speedy processing. Moreover, it introduces a transport protocol known as Apache Arrow Flight, which significantly reduces serialization/deserialization bottlenecks often encountered when transferring data over the network within a distributed system or between different systems. This integration of Apache Arrow at the heart of Dremio's architecture enhances its query performance, making it a powerful tool for data analytics. Apache Arrow, an in-memory data format increasingly recognized as the de facto standard for analytical processing 4. Columnar Cloud Cache One common bottleneck in querying a data lake based on object storage is the latency experienced when retrieving a large number of objects from storage. Additionally, each individual request can incur a cost, contributing to the overall storage access expenses. Dremio addresses these challenges with its C3 (Columnar Cloud Cache) feature, which caches frequently accessed data on the NVMe memory of nodes within the Dremio cluster. This caching mechanism enables rapid access to data during subsequent query executions that require the same information. As a result, the more queries that are run, the more efficient Dremio becomes. This not only enhances query performance over time but also reduces costs, making Dremio an increasingly cost-effective and faster solution as usage grows. This anti-fragile nature of Dremio, where it strengthens and improves with stress or demand, is a significant advantage for organizations looking to optimize their data querying capabilities. 3. Reflections Other engines often rely on materialized views, and BI extracts to accelerate queries, which can require significant manual effort to maintain. This process creates a broader array of objects that data analysts must track and understand when to use. Moreover, many platforms cannot offer this acceleration across all their compatible data sources. In contrast, Dremio introduces a unique feature called Reflections, which simplifies query acceleration without adding to the management workload of engineers or expanding the number of namespaces analysts need to be aware of. Reflections can be applied to any table or view within Dremio, allowing for the materialization of rows or the aggregation of calculations on the dataset. For data engineers, Dremio automates the management of these materializations, treating them as Iceberg tables that can be intelligently substituted when a query that would benefit from them is detected. Data analysts, on the other hand, continue to query tables and build dashboards as usual without needing to navigate additional namespaces. They will, however, experience noticeable performance improvements immediately without any extra effort. This streamlined approach not only enhances efficiency but also significantly reduces the complexity typically associated with optimizing query performance. 2. Semantic Layer Many query engines and data warehouses lack the capability to offer an organized, user-friendly interface for end users, a feature known as a semantic layer. This layer is crucial for providing logical, intuitive views for understanding and discovering data. Without this feature, organizations often find themselves needing to integrate services from additional vendors, which can introduce a complex web of dependencies and potential conflicts to manage. Dremio stands out by incorporating an easy-to-use semantic layer within its lakehouse platform. This feature allows users to organize and document data from all sources into a single, coherent layer, facilitating data discovery. Beyond organization, Dremio enables robust data governance through role-based, column-based, and row-based access controls, ensuring users can only access the data they are permitted to view. This semantic layer enhances collaboration across data teams, offering a unified access point that supports the implementation of data-centric architectures like data mesh. By streamlining data access and collaboration, Dremio not only makes data more discoverable and understandable but also ensures a secure and controlled data environment, aligning with best practices in data management and governance. 1. Hybrid Architecture Many contemporary data tools focus predominantly on cloud-based data, sidelining the vast reserves of on-premise data that cannot leverage these modern solutions. Dremio, however, stands out by offering the capability to access on-premise data sources in addition to cloud data. This flexibility allows Dremio to unify on-premise and cloud data sources, facilitating seamless migrations between different systems. With Dremio, organizations can enhance their on-premise data by integrating it with the wealth of data available in cloud data marketplaces, all without the need for data movement. This approach not only broadens the scope of data resources available to businesses but also enables a more integrated and comprehensive data strategy, accommodating the needs of organizations with diverse data environments. Conclusion Apache Iceberg and Dremio are spearheading a transformative shift in data management and analysis. Apache Iceberg's innovative features, such as partition evolution, hidden partitioning, advanced versioning, and open-source culture, set it apart in the realm of data table formats, offering flexibility, efficiency, and a collaborative development environment. On the other hand, Dremio's data lakehouse platform leverages these strengths and further enhances the data management experience with its integrated SQL query engine, semantic layer, and unique features like Reflections and the C3 Columnar Cloud Cache. By providing a unified platform that addresses the challenges of both on-premise and cloud data, Dremio eliminates the complexity and fragmentation often associated with data analytics. Its ability to streamline data processing, ensure robust data governance, and facilitate seamless integration across diverse data sources makes it an invaluable asset for organizations aiming to leverage their data for insightful analytics and informed decision-making. Together, Apache Iceberg and Dremio not only offer a robust foundation for data management but also embody the future of data analytics, where accessibility, efficiency, and collaboration are key. Whether you're a data engineer looking to optimize data storage and retrieval or a data analyst seeking intuitive and powerful data exploration tools, this combination presents a compelling solution in the ever-evolving landscape of data technology. Also published . here

Data Potential: 10 Reasons Apache Iceberg and Dremio Should Be Part of Your Data Lakehouse Strategy

designed for data lakehouse architectures, enabling the organization of data on data lakes like tables found in traditional databases and data warehouses. This innovative table format provides a crucial abstraction layer, allowing users to leverage database-like features on their data lakes. Its key features are ACID transactions, which ensure data integrity and consistency, time travel capabilities allowing users to access historical data snapshots, and robust table evolution mechanisms for managing partitions and schema changes. By integrating these functionalities, transforms data lakes into more structured and manageable environments, facilitating advanced data analytics and management tasks. Apache Iceberg is an open-source table format Apache Iceberg The Role of Catalogs in Apache Iceberg The catalog mechanism is a cornerstone in the functionality of Apache Iceberg tables, providing a crucial layer of organization and accessibility, even though the specifics of its implementation are beyond the Iceberg specification. A catalog in Iceberg serves as a registry for tables, tracking Iceberg tables to ensure they are discoverable by compatible tools, thereby facilitating seamless integration and usage. Moreover, catalogs are instrumental in maintaining a consistent view of the data, which is essential for the integrity and reliability of ACID transactions. By acting as the source of truth, catalogs enable concurrent transactions to reference them and . This mechanism is vital for maintaining data consistency and ensuring that the system provides ACID guarantees, thereby enhancing the robustness and reliability of data operations within the Iceberg ecosystem. ascertain the status of other ongoing transactions, effectively determining if another transaction was committed while theirs was in progress Challenges with Catalogs and Choosing the Right Catalog For you When selecting a catalog for Apache Iceberg, several key factors should guide your decision: compatibility with your current tools, the additional functionalities or integrations the catalog offers, and its maintainability. It’s crucial to choose only one catalog for managing your tables because, upon the completion of a transaction, only the active catalog is updated. Utilizing multiple catalogs could result in them referencing outdated table states, leading to consistency issues. If there is an absolute need to employ multiple catalogs, a feasible approach is to designate a single catalog for write operations while others are used solely for reading. However, this setup demands the implementation of custom systems to synchronize the read-only catalogs with the primary to ensure they reflect the most current table state, maintaining consistency to the greatest extent possible. Service and File-System Catalogs The distinction between a catalog in Apache Iceberg is fundamental to understanding their operational dynamics and use cases. Service catalogs, which constitute the majority, involve a running service that can be either self-managed or cloud-managed. These catalogs utilize a backing store to maintain all references to Iceberg tables, with locking mechanisms in place to enforce ACID guarantees. This setup ensures that when a table is modified, the catalog updates references accurately, preventing conflicting changes from being committed. service catalog and a file-system On the other hand, the “Hadoop Catalog” represents a file-system catalog that is compatible with any storage system. Unlike service catalogs, the Hadoop Catalog does not rely on a backing store but instead uses a file named “version-hint.text” on the file system to track the latest version of the table. This file must be updated whenever the table changes. However, since not all storage systems provide the same level of atomicity and consistency in file replacement, this method can lead to potential inconsistencies, especially in environments with high concurrency. Therefore, while the Hadoop Catalog might be suitable for evaluating Iceberg’s capabilities, it is generally not recommended for production use due to these potential consistency issues. A tour of Apache Iceberg Catalogs In the following section, we will delve into the diverse range of currently existing catalogs available within the Apache Iceberg ecosystem. Each catalog offers unique features, integrations, and compatibility with different data storage systems and processing engines. Understanding the nuances of these catalogs is crucial for architects and developers to make informed decisions that align with their specific data infrastructure needs and operational goals. We’ll explore a variety of catalogs, including those that are widely adopted in the industry as well as emerging options, highlighting their respective advantages, use cases, and Nessie that extends beyond the traditional catalog capabilities in the Apache Iceberg ecosystem, introducing . This catalog not only tracks table metadata but also allows users to capture commits at a holistic level, enabling advanced operations such as multi-table transactions, rollbacks, branching, and tagging. These features provide a new layer of flexibility and control over data changes, resembling version control systems in software development. Nessie is an innovative open-source catalog git-like features to data management Nessie can be either self-managed or cloud-managed, with the latter option available through the Dremio Lakehouse Platform. Dremio integrates Nessie into its , offering a seamless experience that includes automated table optimization alongside Nessie’s robust cataloging capabilities. This integration underscores Nessie’s versatility and its potential to enhance data governance and management in modern data architectures. Dremio Cloud product Hive The Hive catalog offers a seamless integration pathway for organizations already utilizing Hive in their data architectures, allowing them to leverage their existing Hive metastore as an Apache Iceberg catalog. This integration facilitates a smooth transition to Iceberg’s advanced features while maintaining compatibility with the existing Hive ecosystem. By using the Hive catalog, users can avoid the redundancy of maintaining separate metadata stores, streamlining their data management processes. However, for those not currently using Hive, adopting the Hive catalog would necessitate setting up and running a Hive metastore service. This requirement introduces an additional layer of infrastructure that might be better served using an option with additional features. REST The REST catalog represents a unique approach in the Apache Iceberg ecosystem, serving not as a standalone catalog but as a universal interface that can be adapted for any catalog type. It is based on an OPEN API specification that can be implemented in any programming language, provided that the specified endpoints are adhered to. This flexibility allows for the creation of custom catalogs tailored to specific use cases, enabling developers and organizations to contribute new catalog types to the community without the need to develop bespoke support for a myriad of engines and tools. Among the catalogs utilizing this specification are Tabular, which offers a “headless warehouse” solution; Unity Catalog from Databricks, which primarily manages Delta Lake tables but also provides Iceberg table access through its UniFormat feature; and Gravitino, an emerging open-source catalog project. The community’s vision is that all catalogs will eventually interface through this REST specification, simplifying tool integration by requiring only a single interface. However, it’s important to note that the specification is still evolving, with version 3 under discussion and development, which is anticipated to introduce additional endpoints for greater extensibility and the incorporation of custom behaviors. AWS Glue The AWS Glue catalog is an integral component of the AWS ecosystem, providing a fully managed catalog service that integrates seamlessly with other AWS tools such as Redshift and AWS Athena. As a native AWS service, it offers a streamlined experience for users deeply embedded in the AWS infrastructure, ensuring compatibility and optimized performance across AWS services. A notable feature of the AWS Glue catalog is its support for auto-compaction of tables through the AWS Lake Formation service, enhancing data management and optimization. While the AWS Glue catalog is an excellent choice for those committed to the AWS platform, organizations operating in on-premises environments or across multiple cloud providers might benefit from considering a cloud-agnostic catalog to ensure flexibility and avoid vendor lock-in. Snowflake Catalog The Snowflake Iceberg catalog offers a unique integration, allowing Snowflake to manage Apache Iceberg tables that are stored externally on your storage system. This integration aligns with Snowflake’s robust data management capabilities while offering the cost benefits of utilizing external storage. However, there are limitations to consider: all table creation, insertions, updates, and deletions must be conducted within Snowflake, as the Snowflake SDK currently only supports reading operations from Spark. While this setup allows users to leverage some cost savings by storing tables outside of Snowflake, it comes with a trade-off in terms of flexibility. Users do not have access to the full range of open-ecosystem tools for managing Snowflake-managed Iceberg tables, which could be a significant consideration for organizations that rely on a diverse set of data tools and platforms. LakeFS Catalog Initially, there was a compatibility issue between LakeFS, a file-versioning solution, and Apache Iceberg due to a fundamental difference in their design: Iceberg relies on absolute paths in its metadata to reference files, whereas LakeFS uses relative paths to manage different file versions. To bridge this gap, LakeFS introduced its own custom catalog, allowing it to integrate with Apache Iceberg. However, as with any custom catalog, there’s a dependency on engine support, which, at the time of writing, appears to be limited to Apache Spark. While versioning is a powerful feature for data management, users looking to leverage versioning for their Iceberg tables might find the built-in table versioning features of Iceberg or the catalog-level versioning offered by Nessie to be more universally compatible and supported options, especially when considering broader ecosystem integration beyond LakeFS’s Iceberg catalog. Other As the Apache Iceberg community evolves, there’s a noticeable shift away from certain catalogs, primarily due to concerns like maintenance challenges or inconsistent implementations. Two such catalogs are the JDBC catalog and the DynamoDB catalog. The JDBC catalog, which enabled any JDBC-compatible database to function as an Iceberg catalog, is seeing reduced usage. This decline is likely due to the complexities and variances in how different databases implement JDBC, potentially leading to inconsistencies in catalog behavior. Similarly, the DynamoDB catalog, initially used in the early stages of AWS support for Iceberg, is also falling out of favor. The community’s pivot away from these catalogs underscores a broader trend towards more robust, consistently supported, and feature-rich catalog options that align with the evolving needs and standards of Iceberg users and developers. Conclusion Apache Iceberg stands as a transformative force in the data lakehouse landscape, offering a structured and efficient way to manage data on data lakes with features traditionally reserved for databases and data warehouses. The journey through the diverse world of Apache Iceberg catalogs highlights the importance of these components in ensuring data accessibility, consistency, and robust transactional support. From the integration-friendly Hive catalog to the innovative Nessie catalog that brings git-like versioning to data, each catalog serves a unique purpose and caters to different architectural needs and preferences. As we’ve explored, choosing the right catalog is crucial, balancing factors like compatibility, functionality, and the specific context of your data ecosystem. Whether you’re deeply embedded in the AWS infrastructure, leveraging the AWS Glue catalog, or exploring the versioning capabilities of LakeFS or Nessie, the decision should align with your strategic objectives and operational requirements. The evolution away from certain catalogs, like the JDBC and DynamoDB catalogs, underscores the community’s drive towards more reliable, feature-rich, and consistent catalog implementations. This shift is a testament to the ongoing maturation of the Iceberg ecosystem and its users’ commitment to adopting practices and tools that enhance data reliability, scalability, and manageability. As Apache Iceberg continues to evolve, so too will its ecosystem of catalogs, each adapting to the emerging needs of data professionals seeking to harness the full potential of their data lakehouses. Embracing these tools and understanding their nuances will empower organizations to build more resilient, flexible, and efficient data architectures, paving the way for advanced analytics and data-driven decision-making. Also published . here

Navigating Apache Iceberg: A Deep Dive into Catalogs & Their Role in Data Lakehouse Architectures

Many of us have made changes to data based on requirements that later evolve. By the time we realize it, it becomes impossible to roll back. Change isn't just a constant; it's an essential part of managing data that demands a sophisticated approach. Enter the Git-like functionality of Dremio's data catalog Nessie. Just as Git has become fundamental to software development, data engineers need similar tools to work in parallel, compare data versions, promote changes to production and roll back data when needed. Nessie provides data engineers with a Git-like version control system for managing data versions, branches, merges, and commits. This can be very helpful when multiple data engineers are working with and transforming data at the same time. Nessie allows each engineer to work in separate branches while maintaining a single source of truth in the form of the main branch. This functionality empowers data engineering teams to maintain data quality collaboratively in the face of unrelenting change. This article provides a step-by-step guide demonstrating how Nessie, Dremio and MinIO work together to enhance data quality and collaboration in your data engineering workflows. Whether you're a data engineer, ML engineer, or just a modern data lake enthusiast, this blog equips you with the knowledge and tools needed to effectively enhance your data versioning practices. Understanding the Basics: Nessie Workloads Nessie allows for Git-like workloads that let you test, develop and push to production. Let's break down some key concepts: Just like in Git, branches in Nessie allow you to work on different data engineering tasks concurrently. For example, you might have a branch for feature development, data cleaning and data transformation. Each branch can have its own set of data changes. Branches: In Nessie, a commit represents a snapshot of the data at a specific point in time. When you make changes to your data, you create a new commit, which records those changes. Commits are linked to a specific branch. When you need to revert your data to a stable or known-good state, Nessie makes it easy to select a specific commit and roll back to that data version. This ensures data quality and consistency. Commits: : Nessie allows you to merge the changes from one branch into another. This is similar to merging code changes in Git. When you merge a branch, the data changes made in that branch become part of the target branch. Merges The Components is a distributed analytics engine that operates as an open-source platform, offering an intuitive self-service interface for data exploration, transformation and collaborative work. Its design is grounded in a high-speed columnar memory format. Dremio Apache Arrow, We’ve already explored how to deploy Dremio with Kubernetes, and also how to . For additional information about Dremio, consult . query Iceberg tables on MinIO with Dremio Dremio Resources is high-performance object storage. Renowned for its exceptional and , MinIO serves as a pivotal component in building and maintaining modern data lake infrastructures. MinIO empowers architects to efficiently manage and store massive volumes of data on-prem, on bare metal, on the edge, or on any of the public clouds. MinIO speed scalability is an open table format suited for managing large volumes of data in data lakes. Unique features like time travel, dynamic schema evolution, and Partition Evolution make it a game-changer, allowing query engines to work concurrently on the same data safely and efficiently. See for more information about Iceberg features. Apache Iceberg MinIO The Definitive Guide to Lakehouse Architecture with Iceberg and Getting Started You’ll need the and for this tutorial. The easiest way to get both if you don’t already have them is to install Docker Desktop. Docker Engine Docker Compose This part of the tutorial is based on Dremio’s . You can get the repo for this project . blog post here To begin, open a terminal and navigate to the folder where you cloned/downloaded the repo, and run the command below to start up Dremio. docker-compose up dremio Wait a few moments and then navigate to to access Dremio. Update the requested fields and then click Next. http://localhost:9047 Next, run the following command to start MinIO. docker-compose up minio The final command below will start up Nessie. docker-compose up docker-compose up nessie Navigate to to log in to MinIO with the username and password of . You’ll be prompted to create a bucket. http://localhost:9001 minioadmin:minioadmin Make one named . iceberg-datalake Then head back over to Dremio at and click on Add Source and select Nessie. http://localhost:9047 Set the Name to nessie Set the endpoint URL to http://nessie:19120/api/v2 Set the authentication to none Don’t click Save yet. Instead, in the navigation panel on the left, click Storage. MinIO is S3-API compatible object storage and can use the same connection pathways as AWS S3. For your access key, set minioadmin For your secret key, set minioadmin Set root path to /iceberg-datalake Scroll down for the next set of instructions. Click on the button under to create and configure the following properties. Add Property Connection Properties to fs.s3a.path.style.access true to fs.s3a.endpoint minio:9000 to dremio.s3.compat true Uncheck Encrypt connection Then click Save. You should now see Nessie Catalogs in your data sources. Create a Data Source In Dremio navigate to SQL Runner on the left-hand side. Make sure that in the upper right area of the text editor is set to our Nessie source. Otherwise, you will have to reference the context like instead of just to run this query. Copy and paste the SQL below and run. Context nessie.SalesData SalesData CREATE TABLE SalesData (
    id INT,
    product_name VARCHAR,
    sales_amount DECIMAL,
    transaction_date DATE
) PARTITION BY (transaction_date); Run the query below to insert data into the table you just created. INSERT INTO SalesData (id, product_name, sales_amount, transaction_date)
VALUES
    (1, 'ProductA', 1500.00, '2023-10-15'),
    (2, 'ProductB', 2000.00, '2023-10-15'),
    (3, 'ProductA', 1200.00, '2023-10-16'),
    (4, 'ProductC', 1800.00, '2023-10-16'),
    (5, 'ProductB', 2200.00, '2023-10-17'); Navigate back to MinIO to see that your data lake has been populated with the Iceberg tables. Branching and Merging with Nessie Return to Dremio at . Begin by querying a table on the main branch using the syntax: http://localhost:9047 AT BRANCH SELECT * FROM  nessie.SalesData AT BRANCH main; Create an ETL (Extract Transform and Load) branch to allow you to play around with and transform data without impacting production. CREATE BRANCH etl_06092023 in nessie Within the ETL branch, insert new data into the table: USE BRANCH etl_06092023 in nessie;
INSERT INTO nessie.SalesData (id, product_name, sales_amount, transaction_date) VALUES
(6, 'ProductC', 1400.00, '2023-10-18'); Confirm the immediate availability of the new data within the ETL branch: SELECT * FROM nessie.SalesData AT BRANCH etl_06092023; Note the isolation of changes from users on the main branch: SELECT * FROM nessie.SalesData AT BRANCH main; Merge the changes from the ETL branch back into the main branch: MERGE BRANCH etl_06092023 INTO main in nessie; Select the main branch again to see that the changes have indeed been merged. SELECT * FROM nessie.SalesData AT BRANCH main This branching strategy enables data engineers to independently handle numerous transactions across multiple tables. When they are ready, data engineers can merge these transactions into a single, comprehensive multi-table transaction within the main branch. Conclusion This blog post delved into the power of Git-like version control in data engineering, emphasizing how seamlessly manages data versions, branches and merges. This step-by-step guide demonstrates how Nessie, in collaboration with Dremio and MinIO, as the object storage foundation, enhances data quality and collaboration in data engineering workflows. Nessie Let us know what your data lakehouse looks like at or on . hello@minio.io our Slack channel Also appears . here

Data Lake Mysteries Revealed: Nessie, Dremio, and MinIO Make Waves

ডেটা লেক রহস্য প্রকাশিত হয়েছে: নেসি, ড্রেমিও এবং মিনিও মেক ওয়েভস

Data-Lake-Rätsel aufgedeckt: Nessie, Dremio und MinIO sorgen für Aufsehen

Misterios del lago de datos revelados: Nessie, Dremio y MinIO hacen olas

Les mystères du Data Lake révélés : Nessie, Dremio et MinIO font des vagues

डेटा लेक रहस्य का खुलासा: नेस्सी, ड्रेमियो और मिनिओ मेक वेव्स

データレイクの謎が明らかに: Nessie、Dremio、MinIO が波紋を起こす

데이터 레이크의 미스터리 공개: Nessie, Dremio 및 MinIO가 물결을 일으키다

Mistérios do Data Lake revelados: Nessie, Dremio e MinIO fazem ondas

Тайны озера данных раскрыты: Несси, Дремио и MinIO вызывают фурор

Data Lake'in Gizemleri Ortaya Çıktı: Nessie, Dremio ve MinIO Dalga Yaratıyor

Những bí ẩn về hồ dữ liệu được tiết lộ: Nessie, Dremio và MinIO tạo nên làn sóng

数据湖谜团揭开：Nessie、Dremio 和 MinIO 掀起波澜

Amid the fervor to adopt AI is a critical and often overlooked truth -  the success of any AI initiative is intrinsically tied to the quality, reliability and performance of the underlying data infrastructure. If you don't have the proper foundation, you are limited in what you can build and therefore what you can achieve. Your data infrastructure is the bedrock upon which your entire AI infrastructure is built. It's where data is collected, stored, processed, and transformed. Training models using supervised, unsupervised, and reinforcement learning require storage solutions that can handle structured data - like a Data Warehouse. On the other hand, if you are training Large Language Models (LLMs), you must manage unstructured data - documents in their raw and processed form. A Modern Datalake, or Lakehouse, is the foundation for both these different flavors of AI. A Modern Datalake is one-half Data Warehouse and one-half Data Lake and uses object storage for everything. More recently, we have seen the rise of open table formats. Open Table Formats (OTFs) like Apache Iceberg, Apache Hudi, and Delta Lake make it seamless for object storage to be used within a data warehouse. The rest of this article will examine how to leverage the characteristics of the Modern Datalake that differentiate it from conventional solutions like proprietary Data Warehouses and appliances. To build a foundation for AI infrastructure you need the following: Disaggregation of Compute and Storage Scale Out (Not Up) Software Defined Cloud Native Commodity Hardware If we agree on the above, a series of best practices emerge that focus on two areas of performance. If incorporated, the modern Datalake will be both fast and scalable. These best practices include: Optimizing Drive Price and Performance Incorporate a High-Speed Network Disaggregating Compute and Storage Disaggregating compute and storage within your data infrastructure means that separate resources are used for compute and storage. This contrasts conventional storage solutions, where everything comes packaged in a single server or, worse yet, an appliance. However, Modern Datalakes take disaggregation to another level. If the Data Lake and the Data Warehouse had completely different storage requirements, we could use two separate instances of an object store, as shown below. Additionally, If the Data Warehouse needs to support workloads that require conflicting configurations, then you could use more than one processing engine. This is shown below. A composable infrastructure allows you to scale your compute and storage resources independently. This means you can allocate more resources to the part of your infrastructure that needs it most rather than upgrading both compute and storage together. It results in cost-effective scaling as you only invest in the required resources. Scale-Out Not Up AI workloads are data intensive, are often distributed across multiple CPUs or GPUs, use copious compute power for training, and necessitate real-time inference. Scaling out, not up, helps optimize performance and accommodate high-speed networks. Scaling out and scaling up are two different approaches to increasing the capacity and performance of your data infrastructure. However, scaling out is proving to be the more viable approach as advances are made in clustering platforms like Kubernetes and more and more solutions strive to be cloud native. Scaling out in a disaggregated infrastructure provides: - If one node is busy, another node can take on a new request, reducing wait times and increasing throughput. If one node fails, the workload can be shifted to other nodes, reducing downtime and ensuring continuity. High Availability and Fault Tolerance - Scaling out can provide better performance by distributing workloads across multiple nodes or servers to handle larger volumes of data and more concurrent requests. Scaling out is also more flexible because you can add or remove nodes as needed, making it easier to adjust to fluctuating workloads or to accommodate seasonal variations. Performance and Flexibility - Maintenance and upgrades are simplified when you scale out. Instead of taking a critical system offline for upgrades, you can perform maintenance on individual storage or compute nodes without disrupting the entire infrastructure. Operationally and Resource Efficient Cloud Native + Software Defined The last component of leveraging the Modern Datalake to build a strong foundation for AI is taking a cloud-native, software-defined approach. Containers like Docker and container orchestration tools, like Kubernetes, make cloud-native architectures possible. All the components of a Modern Datalake run in containers that run in Kubernetes. Therefore, a Modern Datalake is cloud native. "Software-defined" refers to an approach in which software controls and manages the configuration, functionality, and behavior of hardware components, often in the context of computer systems and networking. This is the building block of the infrastructure as code movement where the emphasis is on smart software and dumb fast hardware. Software-defined storage abstracts and manages the storage resources through software, making allocating and managing storage capacity across different devices and storage media easier. Built for Speed: NVMe and 100GbE To take full advantage of your commodity hardware and software-defined architecture - you need two more key pieces. The first is NVMe drives. Modern, performance oriented workloads, the random nature of read/write, the rise of small objects and falling SSD pricing all favor a . Do that math, the upfront may be higher, the TCO will be lower. NVMe centric architecture The second component is 100GbE networking. In a software-defined world, the network turns out to be the bottleneck in many setups even at 100GbE. Here are some of those scenarios: - AI workloads often process massive datasets, such as images, videos, natural language text, and sensor data. High-speed networks can quickly transfer these large datasets between storage and processing units, reducing data transfer bottlenecks. Data Intensive - Many AI tasks involve distributed computing across multiple CPUs or GPUs. High-speed networks enable efficient communication and data exchange between these devices, ensuring the computing clusters work in parallel effectively. Distributed Computing - Training deep learning models, especially LLMs like transformers or convolutional neural networks, requires a lot of data and computational power. A high-speed network allows faster data loading and synchronization between distributed GPUs, which can significantly speed up training times. Model Training - Low-latency and high-throughput networks are essential for responsive applications that incorporate AI. A high-speed network ensures minimal delay between a user request and a response from a model. Real-time Inference Foundational Concepts By adhering to these principles: the disaggregation of compute and storage, scale-out, not up, dumb, fast hardware and smart cloud native software the enterprise can construct a Modern Datalake that has the right foundation for meeting these requirements and pushing your AI initiatives forward. You cannot construct a building on a poor foundation, just ask the ancient Egyptians. The AI game is about performance at scale, and this requires the right foundation. To skimp on the foundation is to accumulate technical debt that will topple your Jenga tower after a few minutes. Build smart, put the foundation in place.

The Principles to Keep In Mind When Building a Modern Datalake for Your AI Infrastructure

আপনার AI পরিকাঠামোর জন্য একটি আধুনিক ডেটালেক তৈরি করার সময় যে নীতিগুলি মাথায় রাখতে হবে

Die Grundsätze, die Sie beim Aufbau eines modernen Datalake für Ihre KI-Infrastruktur beachten sollten

Los principios a tener en cuenta al construir un lago de datos moderno para su infraestructura de IA

Les principes à garder à l'esprit lors de la création d'un datalake moderne pour votre infrastructure d'IA

आपके एआई इन्फ्रास्ट्रक्चर के लिए आधुनिक डेटालेक का निर्माण करते समय ध्यान में रखने योग्य सिद्धांत

AI インフラストラクチャ用の最新のデータレイクを構築する際に留意すべき原則

AI 인프라를 위한 최신 데이터레이크를 구축할 때 명심해야 할 원칙

Os princípios a serem considerados ao construir um Datalake moderno para sua infraestrutura de IA

Принципы, которые следует учитывать при создании современного озера данных для вашей инфраструктуры искусственного интеллекта

Yapay Zeka Altyapınız için Modern Bir Datalake Oluştururken Akılda Tutulması Gereken İlkeler

Những nguyên tắc cần ghi nhớ khi xây dựng Datalake hiện đại cho cơ sở hạ tầng AI của bạn

为人工智能基础设施构建现代数据湖时要牢记的原则

Context This article is meant to be a sober reflection on the data lakehouse table format conversation I have had as a participant over the last two years. I've written the following articles over the years on the subject: Table Format Comparison Table Format Comparison: Partitioning Table Format Comparison: Governance Architecture of the Lakehouse Table Formats Apache Iceberg 101 I am writing this article in response to these two pieces: Jacques Nadeau: Overview of state of table formats Kyle Weller: Don’t Worry About Data Lakehouse Features, Trust in Google Search… I work for , who is a contributor to Apache Iceberg and has robust support for Apache Iceberg Data Lakehouses (DDL, DML, Cataloging, Table Optimization) and also can read Delta Lake tables. DISCLOSURE: Dremio, the Data Lakehouse Platform Introduction Choosing the right open-source table format for your data lakehouse is more top of mind than ever as people explore implementing data lakehouses and data lakehouse platforms. With three prominent options in Apache Iceberg, Apache Hudi, and Delta Lake, all vying to be the heart of your lakehouse, it's easy to get lost in the noise of feature sets and performance benchmarks. This blog aims to cut through the fear, uncertainty, and doubt (FUD) surrounding table formats, providing a clear path to understanding what really matters in your selection process. Short Version for Quick Readers All major table formats are mature projects that are constantly improving performance and feature sets. The existence of multiple formats is a boon to the industry, as it fosters a competitive environment that drives improvements across the board. However, when it comes down to choosing a format, it's not just about features and performance. The critical decision should be based on the ecosystem of tools you plan to use and how well they integrate with the format. By focusing on your use cases and testing with your actual workloads, you'll find the table format that best meets your needs. My Journey with Table Formats and Lakehouses Over the last two years, I've immersed myself in the world of table formats and lakehouses, a journey that I plan to continue for the foreseeable future. My entry into this conversation began with an article comparing different table formats. That piece, which became one of my most-viewed writings, initially focused on the various features of these formats. However, as I've updated the article over time, I've noticed the narrowing differences between them, leading to a significant realization. Ecosystem Over Features What truly matters is not the array of features a format possesses but the ecosystem that supports these features and their usability. After all, what's the point of advanced functionality if it's not utilized effectively or is too complex to integrate? The real value lies in how these features are implemented within the ecosystem and how seamlessly they can be adopted into your workflows. Key Takeaways The first major takeaway from my experience is the paramount importance of the ecosystem and usability over the raw feature set. A table format might boast an impressive list of capabilities, but if those features are not accessible or practical for your team's use, their value diminishes. Thus, when evaluating table formats, consider the supporting tools, documentation, community, and ease of integration into your existing systems. Ecosystem and Usability Over Features Performance Considerations: Beyond Benchmarks When delving into the world of table formats, it's easy to get caught up in the numbers game of performance benchmarks. While these metrics offer a snapshot of efficiency under ideal conditions, they seldom provide the full picture needed for informed decision-making. The Limitations of Benchmarks Benchmarks, particularly those in Apache Spark environments, have their place in performance evaluation. However, they often reflect a momentary advantage, influenced by specific configurations and data sets that may not align with everyday usage. The reality is that the performance of a table format is not static; it evolves with each update to its libraries, the processing too, and as users configure the table and tools to suit their unique workloads. True performance evaluation requires a broader lens, one that considers how a table format behaves with the suite of tools you rely on and the nature of your data. Making Informed Choices Choosing a table format based on performance means looking beyond Spark-centric benchmarks. It involves testing how each format manages your specific data types and use cases across the tools and technologies that form the backbone of your data infrastructure. Only by assessing performance in this holistic manner can you ensure that your chosen table format aligns with your operational realities and performance expectations. Understanding the Metric of Open Source Contributors A critical aspect I'd like to clarify from my original blog post is the rationale behind using the number of open-source contributors as a key metric. This choice might have raised some eyebrows, especially considering the complex structure of repository management across many table formats. These projects often distribute their development efforts across multiple repositories, separating the core project—typically written in Java—from implementations in other programming languages like Python, Go, and Rust. Why Focus on Core Contributors? The decision to concentrate on contributors to the core format was deliberate. The core repository is where the most significant decisions regarding the format's evolution are made. It's the heart of the project, determining not just current functionalities but also setting the direction for future developments. The distribution of influence over these decisions is crucial. It reveals whether the evolution of the format is being guided by a diverse group of stakeholders or if it's under the control of a few, potentially leading to a bias in development priorities. The Impact on the Ecosystem While there are undoubtedly many talented contributors working across the various language-specific repositories, their work often focuses on mirroring the functionality established in the core. Although valuable, this does not typically drive the project's overall direction. The core repository's contributor diversity is, therefore, a more accurate indicator of the project's health and its alignment with the broader community's needs. The Importance of Competitive Diversity From a consumer's perspective, the diversity of contributors to the core repository is more than a technicality; it's a safeguard against vendor lock-in, budget inflation, and the potential stagnation of innovation in a controlled market. A table format steered by a community with varied interests ensures that no single entity can dominate the project's direction to the detriment of the community. This competitive diversity fosters an environment where vendors can innovate on equal footing, driving technological advancements and more favorable pricing for end-users. The Value of Contributor Diversity In aiming for a competitive market, the significance of an open format governed by a broad coalition cannot be overstated. The diversity of core repo contributors matters because it ensures that the format remains open, adaptable, and free from monopolistic control. By prioritizing formats with a wide array of contributors, we champion the principles of open innovation and collective progress that are foundational to the open source ethos. Conclusion It's clear that the table format conversation transcends mere technical specifications and performance metrics. The past two years have illuminated the paramount importance of ecosystem compatibility, the tangible benefits of a broad, diverse community of open-source contributors, and the nuanced understanding required to evaluate performance in real-world scenarios. As we navigate the choices between Apache Iceberg, Apache Hudi, and Delta Lake, it becomes evident that the true value lies in fostering an open, competitive market that drives innovation, ensures flexibility, and avoids vendor lock-in. By prioritizing formats that align with our technological ecosystems, supported by a vibrant community, and delivering performance on our workloads with our desired tooling, we can build data infrastructures that are not only robust and efficient but also adaptable and forward-looking. Try the Dremio Lakehouse Platform Today Build a Prototype Data Lakehouse on Your Laptop with This Tutoria Also published . here

Data Lakehouse Table Formats: Considerations, Reflection, and Perspective

As much as we say is an all-in-one data platform that is capable of various analytics workloads, it is always compelling to demonstrate that by real use cases. That's why I would like to share this user story with you. It is about how they leverage the capabilities of Apache Doris in reporting, customer tagging, and data lake analytics and achieve high performance. Apache Doris This fintech service provider is a long-term user of Apache Doris. They have almost 10 clusters for production, hundreds of Doris backend nodes, and thousands of CPU Cores. The total data size is near 1 PB. Every day, they have hundreds of workflows running simultaneously, receive almost 10 billion new data records, and respond to millions of data queries. Before migrating to Apache Doris, they used ClickHouse, MySQL, and Elasticsearch. Then, frictions arise from their ever-enlarging data size. They found it hard to scale out the ClickHouse clusters because there were too many dependencies. As for MySQL, they had to switch between various MySQL instances because one MySQL instance had its limits, and cross-instance queries were not supported. Reporting From ClickHouse + MySQL to Apache Doris Data reporting is one of the major services they provide to their customers and they are bound by an SLA. They used to support such service with a combination of ClickHouse and MySQL, but they found significant fluctuations in their data synchronization duration, making it hard for them to meet the service levels outlined in their SLA. Diagnosis showed that it was because the multiple components add to the complexity and instability of data synchronization tasks. To fix that, they have used Apache Doris as a unified analytic engine to support data reporting. Performance improvements With Apache Doris, they ingest data via the method and reach an SLA compliance rate of over 99% in terms of data synchronization performance. Broker Load As for data queries, the Doris-based architecture maintains an of less than and a of less than . This is a 50% speedup compared to the old architecture. average query response time 10s P90 response time 30s Tagging Tagging is a common operation in customer analytics. You assign labels to customers based on their behaviors and characteristics so that you can divide them into groups and figure out targeted marketing strategies for each group of them. In the old processing architecture, where Elasticsearch was the processing engine, raw data was ingested and tagged properly. Then, it will be merged into JSON files and imported into Elasticsearch, which provides data services for analysts and marketers. In this process, the merging step was to reduce updates and relieve load for Elasticsearch, but it turned out to be a troublemaker: Any problematic data in any of the tags could spoil the entire merging operation and thus interrupt the data services. The merging operation was implemented based on Spark and MapReduce and took up to 4 hours. Such a long time frame could encroach on marketing opportunities and lead to unseen losses. Then Apache Doris takes this over. Apache Doris arranges tag data with its data models, which process data fast and smoothly. The aforementioned merging step can be done by the , which aggregates tag data based on the specified Aggregate Key upon data ingestion. The is handy for partial column updates. Again, all you need is to specify the Unique Key. This enables swift and flexible data updating and saves you from the trouble of replacing the entire flat table. You can also put your detailed data into a to speed up certain queries. . Aggregate Key model Unique Key model Duplicate model In practice, it took the user 1 hour to finish the data ingestion, compared to 4 hours with the old architecture In terms of query performance, Doris is equipped with well-developed bitmap indexes and techniques tailored to high-concurrency queries, so in this case, it can finish and reach over . customer segmentation within seconds 700 QPS in user-facing queries Data lake analytics In data lake scenarios, the data size you need to handle tends to be huge, but the data processing volume in each query tends to vary. To ensure fast data ingestion and high query performance of huge data sets, you need more resources. On the other hand, during non-peak time, you want to scale down your cluster for more efficient resource management. How do you handle this dilemma? Apache Doris has a few features that are designed for data lake analytics, including Multi-Catalog and Compute Node. The former shields you from the headache of data ingestion in data lake analytics while the latter enables elastic cluster scaling. The mechanism allows you to connect Doris to a variety of external data sources so you can use Doris as a unified query gateway without worrying about bulky data ingestion into Doris. Multi-Catalog The of Apache Doris is a backend role that is designed for remote federated query workloads, like those in data lake analytics. Normal Doris backend nodes are responsible for both SQL query execution and data management, while the Compute Nodes in Doris, as the name implies, only perform computation. Compute Nodes are stateless, making them elastic enough for cluster scaling. Compute Node The user introduces Compute Nodes into their cluster and deploys them with other components in a hybrid configuration. As a result, the cluster automatically scales down during the night, when there are fewer query requests, and scales out during the daytime to handle the massive query workload. This is more resource-efficient. For easier deployment, they have also optimized their Deploy on Yarn process via Skein. As is shown below, they define the number of Compute nodes and the required resources in the YAML file, and then pack the installation file, configuration file, and startup script into the distributed file system. In this way, they can start or stop the entire cluster of over 100 nodes within minutes using one simple line of code. Conclusion For data reporting and customer tagging, Apache Doris smoothens data ingestion and merging steps and delivers high query performance based on its own design and functionality. For data lake analytics, the user improves resource efficiency by elastic scaling of clusters using the Compute Node. Along their journey with Apache Doris, they have also developed a data ingestion task prioritizing mechanism and contributed it to the Doris project. A gesture to facilitate their use case ends up benefiting the whole . This is a great example of open-source products thriving on user involvement. open source community Also published . here

Data Analytics: Apache Doris' Impact in Reporting, Tagging, and Data Lake Operations

In a previous post, I provided an introduction to Apache Iceberg and showed how it uses MinIO for storage. I also showed how to set up a development machine. To do this, I used Docker Compose to install an Apache Spark container as the processing engine, a REST catalog, and MinIO for storage. I concluded with a very simple example that used Apache Spark to ingest data and PyIceberg to query the data. If you are new to Apache Iceberg or if you need to get Apache Iceberg set up on your development machine, then give this a read. introductory post In this post, I’ll pick up where my previous post left off and investigate a common big data problem – the need for a single solution to provide storage for raw data, unstructured data, and structured data (data that has been curated from raw data). Additionally, the same solution should provide a processing engine that allows for efficient reporting against the curated data. This is the promise of Data Lakehouses - the capabilities of Data Warehouses for structured data and the capabilities of Data Lakes for unstructured data - all in a centralized solution. Let’s look at our big data scenario in more detail. A Common Problem The diagram below depicts a common problem and a hypothetical solution. Data is coming into a data center from multiple locations and in multiple formats. What is needed is a centralized solution that allows raw data to be transformed such that a processing engine can efficiently support business intelligence, data analytics, and machine learning. At the same time, this solution must also be capable of storing unstructured data (text, images, audio, and video) for data exploration and machine learning. It should also keep any data that was transformed in its original format in case a transformation needs to be replayed or a data integrity issue needs to be investigated. As a concrete example, imagine a global custodial bank that is managing mutual funds for its clients. Data representing the accounting book of records and the investment book of records for each fund for each client is streaming into the Data Lakehouse constantly from geographies around the world. From there, safe passage checks need to occur (was everything sent received), and data quality checks need to run. Finally, the data can be partitioned and loaded into another store that will support start-of-day and end-of-day reporting. Alternately, maybe this diagram represents an IOT scenario where weather stations are sending temperature and other weather-related data. Regardless of the scenario, what is needed is a way to store the data safely in its original format and then transform and process any data that needs to be stored in a more structured fashion - all in one centralized solution. This is the promise of a Data Lakehouse - the best of a Data Warehouse and a Data Lake combined into one centralized solution. Let’s make the hypothetical solution described above real. This is depicted in the diagram below. There are two logical components to our Data Lakehouse. The first is an implementation of Apache Iceberg for structured data - the equivalent of a Data Warehouse. (This is what I built out in my - so I will not go into detail here.) The second logical component is MinIO for unstructured data - the Data Lake side of our Data Lakehouse. All data coming into the Lakehouse is delivered to this logical instance of MinIO. In reality, the two logical instances of MinIO shown above could be the same instance of MinIO in your data center. If the cluster you are running MinIO in can handle the ingestion of all incoming data and the processing requirements of Apache Iceberg, then such a deployment will save money. This is, in fact, what I will do in this post. I’ll use a bucket within Apache Iceberg’s instance of MinIO to hold all unstructured and raw data. previous post Let’s start playing with data by introducing the dataset I will use for this exercise and ingesting it into MinIO. The Global Summary of the Day Dataset The dataset we will experiment with in this post is a public dataset known as the Global Surface Summary of the Day (GSOD), which is managed by the National Oceanic and Atmospheric Administration (NOAA). NOAA currently maintains data from over 9000 stations around the world and the GSOD dataset contains summary information per day from these stations. You can download the data . There is one gzip file per year. It starts in 1929 and ends in 2022 (at the time of this writing). To build our Data Lakehouse, I downloaded every year’s file and put it in the MinIO instance being used for our Data Lakehouse. I put all files in a bucket named `lake.` The two buckets within our instance of MinIO are shown below. The `warehouse` bucket was created when we installed Apache Iceberg. here I used the MinIO console to ingest the raw data manually. In a professional pipeline, you will want to do this in an automated fashion. Check out to see how to use Kafka and Kubernetes to get data into MinIO. How to Set up Kafka and Stream Data to MinIO in Kubernetes These files are packaged for downloading convenience –  if you try to use them directly to create a report or a graph, then that would be a very IO-intensive operation (and potentially CPU-intensive). Imagine that you want to chart the average temperature per year from a specified station. To do this, you have to open every file and search through every row, looking for the entries that match your station on the day of interest. A better option is to use our Data Lakehouses capabilities for curating the data and reporting on the curated data. The first step is to set up a new Jupyter notebook. Set up a Jupyter Notebook First, navigate to the Jupyter Notebook server that is installed in the Apache Spark processing engine. It can be found at . Create a new notebook and in the first cell, add the imports shown below. (All the completed notebooks created in this post can be found .) http://localhost:8888 here from collections import namedtuple
import csv
import json
import logging
import tarfile
from time import time
from typing import List

from minio import Minio
from minio.error import S3Error
import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq

pd.options.mode.chained_assignment = None
bucket_name = 'lake' Notice that we are importing the MinIO library. The notebook we are building is an ETL pipeline from unstructured storage (MinIO Data Lake) to structured storage (Apache Iceberg, which uses MinIO under the hood.) The start of your notebook should look like this. Now, we can create an Iceberg database and table for our data. Creating an Iceberg Database and Table Creating the database and table for the GSOD dataset is straightforward. The script below will create the database which we will name `noaa`. Add this in a cell after the imports. %%sql
CREATE DATABASE IF NOT EXISTS noaa; The script below will create the table. gsod %%sql
CREATE TABLE IF NOT EXISTS noaa.gsod (
  station string,
  date timestamp,
  latitude double,
  longitude double,
  name string,
  temp double
)
USING iceberg
PARTITIONED BY (station) As you play around with Apache Iceberg, you will often want to drop a table so you can start an experiment over. The script below will drop the table should you wish to change anything about its setup. gsod %%sql
DROP TABLE IF EXISTS noaa.gsod; Ingesting Data from MinIO to Iceberg Now that we have the raw year-based zip files in our Lakehouse, we can extract, transform and load them into our Data Lakehouse. Let’s introduce some helper functions first. The function below will return a list of MinIO objects in a specified bucket that matches a prefix. def get_object_list(bucket_name: str, prefix: str) -> List[str]:
'''
Gets a list of objects from a bucket.
'''
  logger = logging.getLogger('gsod_logger')
  logger.setLevel(logging.INFO)

  # Load the credentials and connection information.
  with open('credentials.json') as f:
    credentials = json.load(f)

  # Get data of an object.
  try:
    # Create client with access and secret key
    client = Minio(credentials['url'], # host.docker.internal
      credentials['accessKey'],
      credentials['secretKey'],
      secure=False)

    object_list = []
    objects = client.list_objects(bucket_name, prefix=prefix, recursive=True)
    for obj in objects:
      object_list.append(obj.object_name)

    except S3Error as s3_err:
      logger.error(f'S3 Error occurred: {s3_err}.')
      raise s3_err
    except Exception as err:
      logger.error(f'Error occurred: {err}.')
      raise err

  return object_list Note that in the code above, a MinIO credentials file is needed. This can be obtained from the MinIO console. If you do not know how to get MinIO credentials, then there is a section of this that shows how to generate and download them. post Next, we need a function to get an object from MinIO. Since the objects are tar files we also need this function to extract data out of the tar archive and transform it into a Pandas DataFrame. This is done using the function below. def tar_to_df(bucket_name: str, object_name: str) -> pd.DataFrame:
  '''
  This function will take a tarfile reference in MinIO and do the following:
  - unzip the tarfile
  - turn the data into a single DataFrame object
  '''
  logger = logging.getLogger('gsod_logger')
  logger.setLevel(logging.INFO)

  # Temp file to use for processing the tar files.
  temp_file_name = 'temp.tar.gz'
  # Load the credentials and connection information.
  with open('credentials.json') as f:
    credentials = json.load(f)

  # Get data of an object.
  try:
    # Create client with access and secret key
    client = Minio(credentials['url'], # host.docker.internal
      credentials['accessKey'],
      credentials['secretKey'],
      secure=False)


    object_info = client.fget_object(bucket_name, object_name, temp_file_name)

    Row = namedtuple('Row', ('station', 'date', 'latitude', 'longitude', 'elevation',     

                     'name', 'temp', 'temp_attributes', 'dewp',
                     'dewp_attributes', 'slp', 'SLP_attributes', 'stp', 'stp_attributes', 

                     'visib', 'visib_attributes',
                     'wdsp', 'wdsp_attributes', 'mxspd', 'gust', 'max', 'max_attributes', 

                     'min', 'min_attributes', 'prcp',
                     'prcp_attributes', 'sndp', 'frshtt'))
    # Columns of interest and their data types.
    dtypes={
      'station': 'string',
      'date': 'datetime64[ns]',
      'latitude': 'float64',
      'longitude': 'float64',
      'name': 'string',
      'temp': 'float64'
    }

    tar = tarfile.open(temp_file_name, 'r:gz')
    all_rows = []
    for member in tar.getmembers():
      member_handle = tar.extractfile(member)
      byte_data = member_handle.read()
      decoded_string = byte_data.decode()
      lines = decoded_string.splitlines()
      reader = csv.reader(lines, delimiter=',')

    # Get all the rows in the member. Skip the header.
    _ = next(reader)
    file_rows = [Row(*l) for l in reader]
    all_rows += file_rows

    df = pd.DataFrame.from_records(all_rows, columns=Row._fields)
    df = df[list(dtypes.keys())]
    for c in df.columns:
      if dtypes[c] == 'float64': df[c] = pd.to_numeric(df[c], errors='coerce')
    df = df.astype(dtype=dtypes)

  except S3Error as s3_err:
    logger.error(f'S3 Error occurred: {s3_err}.')
    raise s3_err
  except Exception as err:
    logger.error(f'Error occurred: {err}.')
    raise err

  return df Both of these functions are generic utilities that can be reused regardless of what you are doing with MinIO. Consider putting them in your personal collection of code snippets or your organization's Github Gist. Now, we are ready to send data to the warehouse side of our Lakehouse. This can be done with the code below, which starts a Spark session, loops through all the GSOD tar files, extracts, transforms and sends it to our Iceberg table. from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('Jupyter').getOrCreate()

objects = get_object_list(bucket_name, 'noaa/gsod')

for obj in reversed(objects):
  print(obj)
  df = tar_to_df(bucket_name, obj)
  table = pa.Table.from_pandas(df)
  pq.write_table(table, 'temp.parquet')
  df = spark.read.parquet('temp.parquet')
  df.write.mode('append').saveAsTable('noaa.gsod') The code in this section manually loaded data from a MinIO bucket. In a production environment, you will want to deploy this code in a service and use for automated ingestion. MinIO Bucket Events Querying the Iceberg Data Lakehouse using PyIceberg Let’s start a new notebook for reporting. The cell below imports the utilities we will need. Specifically, we will be using PyIceberg for data retrieval, Pandas for data wrangling, and Seaborn for visualizing data. from pyiceberg.catalog import load_catalog
from pyiceberg.expressions import GreaterThanOrEqual, EqualTo
import pandas as pd
import seaborn as sns

pd.options.mode.chained_assignment = None
catalog = load_catalog('default') What we want to do is calculate the average temperature per year for a given weather station. This gives us one number per year and takes into account all the seasons of the year. The first step is to query Iceberg for all data for a given station. This is done below using PyIceberg. tbl = catalog.load_table('noaa.gsod')

sc = tbl.scan(row_filter="station == '72502014734'")
df = sc.to_arrow().to_pandas()
df.head(10) The station id used in the code above is for a station located at Newark Liberty International Airport, NJ, US. It has been operational since 1973 (almost 50 years of data). When the code runs, you will get the output below. (I am using the DataFrame head() function to get a sample.) Next, we need to group by year and calculate the mean. Using Pandas, this is a few lines of code. No looping is needed. df['year'] = df['date'].dt.year
df = df[['year','temp']]
grouped_by_year = df.groupby('year')
average_by_year = grouped_by_year.mean()
average_by_year Once this cell runs, you will see a single value for each year. The top few years are shown below. Finally, we can visualize our yearly averages. We will use Seaborn to create a line plot. This takes just one line of code. sns.lineplot(data=df, x="year", y="temp", errorbar=None) The line plot is shown below. Another command you should always run after running a report for the first time is below. [task.file.file_path for task in sc.plan_files()] This is a list comprehension that will give you a list of all the data files in Apache Iceberg that have data matching your query. There will be a lot, even though Iceberg’s metadata can filter out many. Seeing all the files involved drives home the fact that high-speed object storage is an important part of a Lakehouse. Summary In this post, we built a Data Lakehouse using MinIO and Apache Iceberg. We did this using the GSOD dataset. First, raw data was uploaded to the Lake side of our Data Lakehouse (MinIO). From there, we created a database and a table in Apache Iceberg (the Data Warehouse side of our Data Lakehouse). Then we built a simple ETL pipeline to move data from the Lake to the Warehouse within the Data Lakehouse. Once we had Apache Iceberg fully populated with data, we were able to create an average yearly temperature report and visualize it. Keep in mind that if you want to build a Data Lakehouse in production, then you will need MinIO’s enterprise features. Consider looking at , , and . Object Lifecycle Management Security Best Practices Kafka Streaming Bucket Events Also published . here

Creating a Data Lakehouse using Apache Iceberg and MinIO

Creando un Data Lakehouse usando Apache Iceberg y MinIO

Création d'un Data Lakehouse à l'aide d'Apache Iceberg et MinIO

अपाचे आइसबर्ग और मिनिओ का उपयोग करके डेटा लेकहाउस बनाना

Apache Iceberg と MinIO を使用したデータ レイクハウスの作成

Criando um Data Lakehouse usando Apache Iceberg e MinIO

Tạo Data Lakehouse bằng Apache Iceberg và MinIO

使用 Apache Iceberg 和 MinIO 创建 Data Lakehouse

Developing Streaming Data Lakes with Hudi and MinIO

Desarrollo de streaming de lagos de datos con Hudi y MinIO

Développer des lacs de données de streaming avec Hudi et MinIO

हुडी और मिनिओ के साथ स्ट्रीमिंग डेटा लेक विकसित करना

Hudi と MinIO を使用したストリーミング データ レイクの開発

Desenvolvendo Streaming Data Lakes com Hudi e MinIO

Phát triển các hồ dữ liệu phát trực tuyến với Hudi và MinIO

使用 Hudi 和 MinIO 开发流数据湖

Data Lake

data-lake.co

-- employees

Since n.d.

COMPANY RANKING

DATA LAKE

EVERGREEN INDEX #2727

Data Lake WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

Data Lake

data-lake.co

-- employees

Since n.d.

COMPANY RANKING

DATA LAKE

EVERGREEN INDEX #2727

RELATED COMPANIESRANK

STORIES ABOUT DATA LAKE

Data Lake WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!