This article was original published on the New Stack. Compare Apache Iceberg, Delta Lake and Apache Hudi and learn how to choose the right open table format for your data lakehouse. Open table formats and object storage are redefining how organizations architect their data systems, providing the foundation for scalable, efficient and future-proof data lakehouses. By leveraging the unique strengths of object storage — its scalability, flexibility and cost-effectiveness — alongside the advanced metadata management capabilities of open table formats like Apache Iceberg, Delta Lake and Apache Hudi, organizations can create modular architectures that meet the demands of modern data workloads. At the core of this architectural shift is the disaggregation of compute and storage. Object storage serves as the foundation, offering seamless management of structured, semi-structured and unstructured data, while open table formats act as a metadata abstraction layer, enabling database-like features such as schemas, partitions and atomicity, consistency, isolation and durability (ACID) transactions. Compute engines like Spark, Presto, Trino and Dremio interact with these table formats, delivering the flexibility to process and analyze data at scale without vendor lock-in. This guide will delve into the role of open table formats and object storage in building modern data lakehouses. I’ll explore their evolution, compare leading table formats and highlight performance considerations that optimize your architecture for advanced analytics and AI workloads. By understanding these components, you’ll be equipped to design data systems that are not only efficient and scalable but also adaptable to the rapidly changing demands of the data-driven era. Where Open Table Formats Fit In Where Open Table Formats Fit In Where Open Table Formats Fit In The modern data lakehouse architecture builds upon three critical components: the storage layer, the open table format and the compute engines. This modular design is optimized to take full advantage of object storage's scalability and cost-efficiency while leveraging open table formats for seamless metadata management and interoperability across diverse compute engines. At its foundation lies the storage layer of object storage, which provides scalable and flexible storage for structured, semi-structured and unstructured data. In the storage layer sit the open table formats, which could be Apache Iceberg, Delta Lake or Apache Hudi. These open table formats act as a metadata abstraction layer, providing database-like features including schemas, partitions and versioning as well as advanced features like ACID transactions, schema evolution and time travel. Finally, compute engines like Spark, Presto, Trino and Dremio interact with the open table formats to process and analyze data at scale, offering users the flexibility to choose the best tool for their workload. Evolution of Data Architectures Evolution of Data Architectures The rise of data lakehouses can be understood as part of the broader evolution of data architectures. Early systems like online transaction processing (OTLP) databases prioritized transactional integrity but lacked analytical capabilities. The advent of online analytical processing (OLAP) systems introduced data warehouses, optimized for querying structured data, but at the cost of handling semi-structured and unstructured data efficiently. Data lakes emerged to address these limitations, offering scalable storage for varied data types and schema-on-read capabilities. However, the lack of transactional guarantees in data lakes spurred the development of data lakehouses, which integrate the strengths of data lakes and data warehouses into a unified architecture. Lakehouses are built on open table formats and object storage and are fully decoupled, meaning they are constructed of modular components. This disaggregated architecture provides both the transactional consistency of databases and the scalability of object storage. Why Open Table Formats Are Ideal for Object Storage Why Open Table Formats Are Ideal for Object Storage Data lakehouse architectures are purposefully designed to leverage the scalability and cost-effectiveness of object storage systems, such as Amazon Web Services (AWS) S3, Google Cloud Storage and Azure Blob Storage. This integration enables the seamless management of diverse data types — structured, semi-structured and unstructured — within a unified platform. Key features of data lakehouse architectures on object storage include: Unified storage layer: By utilizing object storage, data lakehouses can store vast amounts of data in its native format, eliminating the need for complex data transformations before storage. This approach simplifies data ingestion and enables compatibility with various data sources.Scalability: Object storage systems are inherently scalable, allowing data lakehouses to accommodate growing data volumes without significant infrastructure changes. This scalability enables organizations to efficiently manage expanding data sets and evolving analytics requirements.Flexibility: Best-in-class object storage can be deployed anywhere — on premises, in private clouds, in public clouds, colocation facilities, data centers and at the edge. This flexibility allows organizations to tailor their data infrastructure to specific operational and geographic needs. Unified storage layer: By utilizing object storage, data lakehouses can store vast amounts of data in its native format, eliminating the need for complex data transformations before storage. This approach simplifies data ingestion and enables compatibility with various data sources. Unified storage layer Scalability: Object storage systems are inherently scalable, allowing data lakehouses to accommodate growing data volumes without significant infrastructure changes. This scalability enables organizations to efficiently manage expanding data sets and evolving analytics requirements. Scalability Flexibility: Best-in-class object storage can be deployed anywhere — on premises, in private clouds, in public clouds, colocation facilities, data centers and at the edge. This flexibility allows organizations to tailor their data infrastructure to specific operational and geographic needs. Flexibility: By integrating these elements, data lakehouse architectures offer a comprehensive solution that combines the strengths of data lakes and data warehouses. This design facilitates efficient data storage, management and analysis, all built upon the foundation of scalable and flexible object storage systems. Open Table Formats Defined Open Table Formats Defined An open table format is a standardized, open source framework designed to manage large-scale analytic datasets efficiently. It operates as a metadata layer atop data files, facilitating seamless data management and access across various processing engines. Here is an overview of the three open table formats: Iceberg Delta Lake and Hudi: Apache Iceberg Apache Iceberg Apache Iceberg is a high-performance table format designed for massive data sets. Its architecture prioritizes efficient read operations and scalability, making it a cornerstone for modern analytics workloads. One of its defining features is the separation of metadata from data, allowing efficient snapshot-based isolation and planning. This design eliminates costly metadata operations, enabling parallel query planning across large datasets. Recent advancements in the Iceberg ecosystem highlight its growing adoption across the industry. S3 Tables simplify data management by enabling query engines to directly access table metadata and data files stored in S3-compatible systems, reducing latency and improving interoperability. Meanwhile, Databricks’ acquisition of Tabular underscores Iceberg’s primacy role in open lakehouse platforms and emphasizes its focus on performance and governance. Additionally, Snowflake’s decision to make Polaris open source demonstrates the industry’s commitment to openness and interoperability, further solidifying Iceberg’s position as a leading table format. Databricks’ acquisition of Tabular Databricks’ acquisition of Tabular Polaris open source Polaris open source Delta Lake Delta Lake Originally developed by Databricks, Delta Lake was closely tied to Apache Spark. It is fully compatible with Spark APIs and integrates with Spark’s Structured Streaming, allowing for both batch and streaming operations. One key feature of Delta Lake is that it employs a transaction log to record all changes made to data, ensuring consistent views and write isolation. This design supports concurrent data operations, making it suitable for high-throughput environments. Apache Hudi Apache Hudi Apache Hudi is designed to address the challenges of real-time data ingestion and analytics, particularly in environments requiring frequent updates. Its architecture supports write-optimized storage (WOS) for efficient data ingestion and read-optimized storage (ROS) for querying, enabling up-to-date views of data sets. By processing changes in data streams incrementally, Hudi facilitates real-time analytics at scale. Features like Bloom filters and global indexing optimize I/O operations, improving query and write performance. Additionally, Hudi includes tools for clustering, compaction and cleaning, which aid in maintaining table organization and performance. Its capability to handle record-level updates and deletes makes it a practical choice for high-velocity data streams and scenarios requiring compliance and strict data governance. Comparing Open Table Formats Comparing Open Table Formats Apache Iceberg, Delta Lake and Apache Hudi each bring unique strengths to data lakehouse architectures. Here's a comparative overview of these formats based on key features: ACID transactions: All three formats provide ACID compliance, ensuring reliable data operations. Iceberg employs snapshot isolation for transactional integrity, Delta Lake utilizes a transaction log for consistent views and write isolation, and Hudi offers file-level concurrency control for high-concurrency scenarios. Schema evolution: Each format supports schema changes, allowing the addition, deletion or modification of columns. Iceberg offers flexible schema evolution without rewriting existing data, Delta Lake enforces schema at runtime to maintain data quality, and Hudi provides pre-commit transformations for additional flexibility. Partition evolution: Iceberg supports partition evolution, enabling seamless updates to partitioning schemes without rewriting existing data. Delta Lake allows partition changes but may require manual intervention for optimal performance, while Hudi offers fine-grained clustering as an alternative to traditional partitioning. Time travel: All three formats offer time-travel capabilities, allowing users to query historical data states. This feature is invaluable for auditing and debugging purposes. Widespread adoption: Iceberg is the most widely adopted open table format by the data community. From Databricks to Snowflake to AWS, many large platforms have invested in Iceberg. If you’re already part of these ecosystems or thinking about joining them, Iceberg might naturally stand out.Indexing: Hudi provides multimodal indexing capabilities, including Bloom filters and record-level indexing, which can enhance query performance. Delta Lake and Iceberg rely on metadata optimizations but do not offer the same level of indexing flexibility.Concurrency and streaming: Hudi is designed for real-time analytics with advanced concurrency control and built-in tools like DeltaStreamer for incremental ingestion. Delta Lake supports streaming through change data feed, and Iceberg provides basic incremental read capabilities. ACID transactions: All three formats provide ACID compliance, ensuring reliable data operations. Iceberg employs snapshot isolation for transactional integrity, Delta Lake utilizes a transaction log for consistent views and write isolation, and Hudi offers file-level concurrency control for high-concurrency scenarios. ACID transactions: Schema evolution: Each format supports schema changes, allowing the addition, deletion or modification of columns. Iceberg offers flexible schema evolution without rewriting existing data, Delta Lake enforces schema at runtime to maintain data quality, and Hudi provides pre-commit transformations for additional flexibility. Schema evolution: Partition evolution: Iceberg supports partition evolution, enabling seamless updates to partitioning schemes without rewriting existing data. Delta Lake allows partition changes but may require manual intervention for optimal performance, while Hudi offers fine-grained clustering as an alternative to traditional partitioning. Partition evolution: Time travel: All three formats offer time-travel capabilities, allowing users to query historical data states. This feature is invaluable for auditing and debugging purposes. Time travel: Widespread adoption: Iceberg is the most widely adopted open table format by the data community. From Databricks to Snowflake to AWS, many large platforms have invested in Iceberg. If you’re already part of these ecosystems or thinking about joining them, Iceberg might naturally stand out. Widespread adoption: Indexing: Hudi provides multimodal indexing capabilities, including Bloom filters and record-level indexing, which can enhance query performance. Delta Lake and Iceberg rely on metadata optimizations but do not offer the same level of indexing flexibility. Indexing: Concurrency and streaming: Hudi is designed for real-time analytics with advanced concurrency control and built-in tools like DeltaStreamer for incremental ingestion. Delta Lake supports streaming through change data feed, and Iceberg provides basic incremental read capabilities. Concurrency and streaming: These distinctions highlight that while all three formats provide a robust foundation for modern data architectures, the optimal choice depends on specific workload requirements and organizational needs. Performance Expectations Performance Expectations Achieving optimal performance in data lakehouse architectures is essential to fully leverage the capabilities of open table formats. This performance hinges on the efficiency of both the storage and compute layers. The storage layer must provide low latency and high throughput to accommodate large-scale analytics demands. Object storage solutions should facilitate rapid data access and support high-speed transfers, ensuring smooth operations even under high workloads. Additionally, efficient input/output operations per second (IOPS) are crucial for handling numerous concurrent data requests, enabling responsive data interactions without bottlenecks. Equally important is compute layer performance, which directly influences data processing and query execution speeds. Compute engines must be scalable to manage growing data volumes and user queries without compromising performance. Employing optimized query execution plans and resource management strategies can further enhance processing efficiency. Additionally, compute engines need to integrate seamlessly with open table formats to fully utilize advanced features like ACID transactions, schema evolution and time travel. The open table formats also incorporate features designed to boost performance. These also need to be configured properly and leveraged for a fully optimized stack. One such feature is efficient metadata handling, where metadata is managed separately from the data, which enables faster query planning and execution. Data partitioning organizes data into subsets, improving query performance by reducing the amount of data scanned during operations. Support for schema evolution allows table formats to adapt to changes in data structure without extensive data rewrites, ensuring flexibility while minimizing processing overhead. By focusing on these performance aspects across the storage and compute layers, organizations can ensure that their data lakehouse environments are efficient, scalable and capable of meeting the demands of modern analytics and AI workloads. These considerations enable open table formats to reach their full potential, delivering the high performance needed for real-time insights and decision-making. Open Data Lakehouses and Interoperability Open Data Lakehouses and Interoperability The data lakehouse architecture builds on open table formats to deliver a unified approach to data management. However, achieving true openness requires more than just adopting open table formats. Open data lakehouses must integrate modular, interoperable and open source components such as storage engines, catalogs  and compute engines to enable seamless operation across diverse platforms. The open table formats are open standards and, by their very design, support interoperability and openness throughout the stack. Yet, practical challenges remain, such as ensuring catalog interoperability and avoiding dependencies on proprietary services for table management. The recent introduction of tools like Apache XTable demonstrates progress toward universal compatibility, providing a path to write-once, query-anywhere systems. It’s important to note that XTable doesn’t allow you to write in multiple open table formats, just read. Hopefully future innovations in interoperability will continue to build on these and other projects surrounding the open table formats. Apache XTable Apache XTable The Future of Open Table Formats The Future of Open Table Formats As the landscape of data lakehouses continues to evolve, certain trends and advancements are likely to shape its future. A significant area of growth will likely be the integration of AI and machine learning (ML) workloads directly into the lakehouse architecture. For the storage layer, this could look like platforms with direct integrations to key AI platforms like Hugging Face and OpenAI. For the compute layer, AI integration could lead to creating specialized compute engines optimized for ML algorithms, enhancing the efficiency of training and inference processes within the lakehouse ecosystem. ML ML workloads workloads Another area of significant growth will likely be in the open source community. When major private companies like Databricks, Snowflake and AWS start to throw their weight around, it’s easy to forget that the open table formats are true open standards. Iceberg, Hudi and Delta Lake are available for any contributors, collaboration or integration into open source tools and platforms. In other words, they are part of a vibrant and growing open-standard data ecosystem. It’s important to remember that open source begets open source. We will see the continued proliferation of open source applications, add-ons, catalogs and innovations in this space. Finally, adoption of open table formats will continue to rise as enterprises build large-scale, high-performance data lakehouses for AI and other advanced use cases. Some industry professionals equate the popularity of open table formats with the rise and supremacy of Hadoop in the early 2000s. Big data is dead; long live big data. Some Some Build for Today and Tomorrow Build for Today and Tomorrow Combining open table formats with high-performance object storage allows architects to build data systems that are open, interoperable and capable of meeting the demands of AI, ML and advanced analytics. By embracing these technologies, organizations can create scalable and flexible architectures that drive innovation and efficiency in the data-driven era.

Walkthroughs, tutorials, guides, and tips. This story will teach you how to do something new or how to do something better.

Google

The Architect’s Handbook to Open Table Formats and Object Storage

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

A Closer Look Into the MinIO Enterprise Object Store Firewall

Branches & Tags: Comparing Iceberg, Hudi, and Delta Lake Tables

Building High-Performance Data Lake Using Apache Hudi and Alluxio

Developing Streaming Data Lakes with Hudi and MinIO

The Data Table Format Wars

Data Lakehouse Table Formats: Considerations, Reflection, and Perspective

A Closer Look Into the MinIO Enterprise Object Store Firewall

Branches & Tags: Comparing Iceberg, Hudi, and Delta Lake Tables

Building High-Performance Data Lake Using Apache Hudi and Alluxio

Developing Streaming Data Lakes with Hudi and MinIO

The Data Table Format Wars

Data Lakehouse Table Formats: Considerations, Reflection, and Perspective

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps