We offer a cloud data platform which supports a multi-cloud strategy, including a cross-cloud approach to mix and match clouds as you see fit

Snowflake Computing

The Drive to Migrate Data to the Cloud With data being valued in recent years, many organizations feel the pressure to become innovative and cost-effective when it comes to consolidating, storing, and using data. Although most enterprises are aware of big data opportunities, their existing infrastructure isn’t always capable of handling massive amounts of data. more than oil By migrating to modern cloud data warehouses, organizations can benefit from improved scalability, better price elasticity, and enhanced security. But even with all these benefits, many businesses are still reluctant to make a move. This is because these benefits are great . However, before companies can get there, they must go through . once you get there the dreaded data migration process, and that process can be painful— especially if you don’t have the right tools To create an effective cloud data migration strategy, companies often struggle to find answers to key areas like: Gaps or anomalies in the source data Full or partial data migration Data owners and stakeholders Data lineage and dependencies Data consolidation process Automated regression testing for accuracy With so many daunting concerns, it’s no wonder why many businesses delay their migration. And as a result, they miss out on the benefits of cloud data warehousing. In this article, . While our focus in this article will be on Snowflake, a nearly identical approach can be used for migrating to Redshift, BigQuery, Databricks, or any other cloud data warehouse. we will show you how—with a tool like Datafold—you can avoid a lot of the stress and complexities of data migration to Snowflake Snowflake is a Software-as-a-Service (SaaS) data warehouse hosted entirely on cloud infrastructure. It has one of the and is available on all three major cloud platforms, offering unlimited scalability in both storage and computing. Snowflake biggest software IPOs in history Another reason for its popularity is its simple maintenance. Unlike most legacy data solutions, Snowflake takes care of the following: management upgrades tuning backups high availability indexing It keeps its storage, and computing layers separate, allowing significant cost-saving when not using computing power. Other attractive features include time travel, end-to-end encryption, and integration with popular identity providers. Snowflake also natively integrates with , creating a rich ecosystem of analytical capabilities. , a modern data reliability platform, can make data migration to Snowflake a breeze. For example, Datafold can: third-party applications Datafold Intelligently profile data sources to show exactly where a piece of data is coming from and what it depends on Show before and after migration differences between source and target data sets. Use built-in machine learning models to automatically find data anomalies and alert users. Integrate with all major data warehouses—including Snowflake Be automated with tools like or or integrated into a continuous integration (CI) process. dbt Apache Airflow The important question is: Let’s see. Can we prevent potentially costly mistakes when migrating to a Snowflake data warehouse? Typical Data Migration Mistakes and How to Avoid Them There are a few common mistakes DataOps teams make when migrating to a data warehouse. Ignoring Key Stakeholders Data engineering teams often rush through a migration project without consulting business stakeholders or including their feedback about data validation rules. This results in a flood of post-migration support tickets for erroneous or missing data. What’s worse is that some errors can go unnoticed. For example, let’s say the team is migrating billing data, but the migration team hasn’t properly consulted the finance team. The currency conversion rates in the legacy system might use a four decimal point rounding while the new system uses only three. The engineers may deem this acceptable, but for the finance team, a change like this can mean skewed numbers across all their reports and subsequent business impact. To address this issue, in Datafold can list all your data sources in one place. This allows you to search and filter to find a particular table, database, or column from thousands of databases across your environment. Data Catalog Once you find a data asset, you can easily add descriptions and tags and also set owners for the assets. This makes it easy to know who to contact if you need to make a change or what types of changes are acceptable. Taking a “Lift and Shift” Approach Many businesses take a “lift and shift” approach to their data migration. This is where all the data from old systems is migrated to a new one with little to no validation, cleansing, or transformation. One of the reasons businesses take this approach is because they deem it too time-consuming to comb through huge datasets to find out what’s relevant and what’s not. Lift and shift can only work when you’re 100% sure that the old system’s data is correct and portable to the new system—and this is seldom the case. Consequently, you risk migrating incorrect or duplicate data and paying for its storage. When working with TBs—or even PBs—of data, these storage costs quickly add up. as-is The Data Catalog summarizes the quality of the source data in terms of volume, completeness, and distribution of values. Using this, you can identify if important table columns like birthdate, email_address, or last_name are complete and if they need special business rules. In the image below, we can see a table’s data profile: The table has more than 19 million rows and 21 columns. There are no upstream dependencies. Part of the column’s data distribution looks like the following: As you can see, some fields—like pickup_latitude and pickup_longitude—have lower completeness ratios. The graph also shows data distribution in different fields. Data engineers can use such information to decide whether (or how) to cleanse and preprocess the data before migration. The feature can also help by finding the most frequently used (and, therefore, the most critical) tables and columns in the source. column-level lineage One of the major drawbacks of lift-and-shift is that you’re never sure what fields or tables are unnecessary and don’t need to be ported. Column-level lineage makes this task much easier as it can discover all the dependencies of a table or column. Using the Data Catalog and its lineage feature, let’s say you find a large source table that hasn’t been updated in the last five years, and this table has no dependencies. You can feel confident that it can be archived and doesn’t need to be migrated. As an example, in the image below, we can see that doesn’t have any dependencies: taxi_trips Migrating Old and Inefficient Queries Engineers should also be looking for inefficient queries powering existing analytic reports. Not surprisingly, this is also often overlooked in fear of breaking applications. For example, maybe you have a query with 10 joins, four of which are joining to obsolete data tables. The migration project can be the perfect time to optimize it by cutting out irrelevant joins. Showing side-by-side improvements can further showcase the value of planned migration to stakeholders. Once again, the column-level lineage feature can help refactor or remove inefficient queries. Column-level lineage can show how a data field relates to its upstream and downstream objects and what the effect will be when modifying queries containing this field. In the image below, we can see that a data field is related to two tables. This should tell the data engineer which tables may be affected when fine-tuning a query containing this field. Datafold’s feature can also show the potential impact of changing your old queries. Data Diff compares two datasets and can show if their schemas, primary keys, row counts, or column values are mismatched. Data Diff You can use such results for quality assurance of your migrated data. If the target Snowflake tables show no discrepancies, stakeholders can have confidence that the new queries are safe and effective. If the comparison shows differences, then the queries can be further fine-tuned. In the images below, we see one such Data Diff operation. Here, the target table shows non-matching columns with non-matching values: Manual or No Regression Testing For organizations dealing with terabytes of data, testing and validation is a guaranteed recipe for failure. Manually cross-checking millions of rows of query results or data is simply not an option. The result is that many engineers will simply skip validation altogether. manual According to the data engineers in , their manual validation process used to take up to two hours per pull request. In many cases, this validation just wasn’t realistic to complete, and mistakes can be costly. There are two ways to eliminate—or largely automate—manual data validations when migrating data to Snowflake. Thumbtack The first method is to use Datafold’s feature. Datafold uses machine learning to find the normal trend or seasonality in your data. You can configure alerts to send proactive messages to DataOps when anomalies are detected in that data. You can also adjust the alert thresholds. Alerting You can create alerts by writing your own SQL statements for data completeness checks or comparisons, as shown below: In both cases, proactive alerts can help data engineers fix any issues in the source data before migration. The other option is to use the Data Diff feature within the CI process. With this approach, after creating a data migration script, it’s committed to a Git repository. Once the commit is made to the development branch, a developer creates a pull request to merge the change to the main branch. At this stage, a senior team member can review the code to approve or reject it. Now, when integrated with the CI process, Datafold’s Data Diff feature can show the reviewer the effect of the ETL code—how data will change between the source and the target as the code cleanses, aggregates, and performs its operation. This is automatic regression testing for the data pipeline. If reviewers see the target table will have unwanted changes when the code runs, they can reject the change, preventing garbage data from entering the Snowflake database. When Thumbtack started using Data Diff, they reported savings of over 200 hours per month, increased productivity by at least 20%, and over 100 pull requests automatically tested every month. Conclusion There are many advantages to moving to a cloud data warehouse like Snowflake. However, migrating even a single database to a new system can have its challenges, especially when the source data hasn’t been profiled, or the ETL code hasn’t been tested. The key to successful data migration is leveraging tools that can automate most of the hard work. Datafold can help dramatically reduce the number of errors, failures, and silent issues by automating most of the testing. Although we’ve focused on Snowflake in this article, the same features of Datafold can be used for other cloud products like or . Redshift BigQuery Also Published Here

Migrating to Snowflake, Redshift, or BigQuery? Avoid these Common Pitfalls

Why Embedded Analytics Matters: Unleashing Data Insights within Applications is becoming an indispensable capability for modern SaaS applications across industries. By embedding analytics directly into applications, insights can guide internal application users and external customers to enable better and faster decision-making. A strong embedded analytics solution that SaaS companies can benefit from starts with the data layer. Many SaaS companies try to determine the best database for their SaaS solution and quite often it becomes an AWS Redshift vs Snowflake comparison. Embedded analytics Exporting data to external business intelligence tools for analysis is becoming less common. Leading organizations are realizing the competitive advantage and of using live data within their apps, so choosing the right database matters. monetization opportunities Data Warehousing: The Engine Powering Embedded Analytics To enable real-time and/or multi-tenant embedded analytics, applications need a high-performance data warehousing layer that can efficiently process queries and serve up data analysis. The data warehouse organizes and stores data from various sources specifically for use cases that span reporting, data visualization, dashboards, and analytics applications. Choosing the right data warehouse is therefore critical. Choosing the Right Tool: Redshift vs Snowflake Two leading cloud data warehouse contenders that show great promise for embedded use cases are AWS Redshift and Snowflake. Both platforms offer advantages such as scalability and flexibility which suit them well for embedded analytics. We compare the two options across crucial criteria to determine which choice best meets embedded needs. Redshift vs Snowflake: Comparing Strengths and Weaknesses AWS Redshift AWS Redshift is a fully managed, petabyte-scale data warehousing service provided by (AWS). It is a cloud-based, massively parallel processing (MPP) database optimized for analytical and reporting workloads. This makes it useful for powering dashboards, ad-hoc queries, and data warehousing. Amazon Web Services Redshift provides fast query performance by using columnar storage and parallel processing to quickly analyze large datasets using multiple nodes. Many enterprises rely on Redshift given its ability to handle heavy analytics workloads. To manage those larger workloads, Redshift can scale storage and compute capacity independently. This offers you the flexibility to pay only for what you need. Scalability and Performance: Brute Force Meets Efficiency with Redshift A pioneer in cloud data warehousing, Redshift offers fast query performance leveraging a massively parallel processing (MPP) architecture optimized for high throughput analytics workloads. Redshift allows scaling compute and storage separately on demand, automatically distributing data across nodes. Performance remains high even with ultra-large datasets and complex queries. Users have reported 50-100x faster queries near the petabyte scale. Cost-Effectiveness: Pay-as-You-Go vs Predictability As part of AWS, Redshift offers pay-as-you-go pricing allowing optimization of costs based on current needs. However, costs can vary significantly based on changing query volumes, underlying data sizes, and other factors – making longer-term budgets and forecasts difficult. Cost optimization requires continual fine-tuning of Redshift clusters and workload monitoring. For embedded analytics specifically, this cost model requires careful management as SaaS usage is meant to grow over time. Deployment and Management: The AWS Ecosystem Advantage Being natively part of AWS, Redshift enables deployment leveraging other AWS services for storage, ETL, monitoring, and more. Companies already using AWS experience less management overhead as a result. But reliance on AWS also leads to vendor lock-in – migrating to other platforms would require significant re-architecture. User-friendliness: Is Redshift Beginner-Friendly? Redshift exposes a standard SQL interface for executing queries. However optimal configuration and cost management require deeper expertise in areas like cluster sizing, workload management, and query optimization. The platform may present a learning curve for beginners. Snowflake Snowflake is a cloud-based data warehousing service that offers a unique architecture optimized for scalability, flexibility, and performance in the cloud. It utilizes a multi-cluster, shared data architecture to efficiently separate storage and computing. This allows independent scaling of resources to match workload demands. Snowflake also has native support for public clouds AWS, Azure, and GCP cloud platforms. The decoupled storage/compute architecture can auto-scale clusters and warehouse capacity based on query volumes and data sizes. This provides high concurrency and performance, similar to Redshift. Snowflake uses a SQL database engine optimized for data warehousing workloads such as analytics, dashboards, reporting, etc. Elastic Power: Scale on Demand, Pay for What You Use with Snowflake Snowflake pioneered a unique cloud-native architecture optimized for flexibility and scalability. The decoupled storage and compute allow auto-scaling to handle extreme workloads without overload. Snowflake also offers per-second pricing – pay only for the capacity used per query without paying for idle clusters. This has similar concerns to Redshift for embedded analytics use cases. As SaaS usage increases, companies realize that usage remains consistent throughout the day, contrary to their initial expectations. These cost increases present challenges for using Snowflake with embedded analytics. Cloud Agnostic Freedom: Beyond the AWS Walls A multi-cloud and hybrid cloud option, Snowflake avoids vendor lock-in by deploying across AWS, Azure, and GCP. Snowflake offers easy migration between clouds with push-button cloud failover capabilities. Snowflake also offers flexibility to query data in external stores without copying across the warehouse. Rich Data Ecosystem: Seamless Integration and Collaboration Snowflake is a strong hub for sharing and exchanging data. It helps teams, partners, and other stakeholders access and collaborate on data easily. Snowflake also offers extensive compatibility with third-party tools. Future-Proof Innovation: Embracing the Evolution of Analytics With rapid innovation across query processing, security, compliance, and machine learning capabilities, Snowflake is leading the way in cutting-edge features for modern internal analytics. Their unique architecture choices make it easy to evolve the platform over time. Organizations can benefit from new capabilities without migrations. Embedded Analytics: Where Redshift and Snowflake Shine (and Stumble) Real-Time Insights: Delivering Data at the Speed of Thought to SaaS Users Embedded analytics requires querying and aggregating live, real-time data with minimal latency to drive contextual insights and guided action within apps. Both Redshift and Snowflake leverage MPP architectures to enable speedy analysis across large datasets. Slight advantages go to Snowflake for its adaptive elastic scaling and per-second pricing which optimizes costs for spiky query workloads common in real-time dashboards and applications. Simplicity and Integration: Seamless Embedding for User Delight For delightful embedded experiences, analytics components need easy integration and simple configuration within applications built using various programming languages, frameworks and platforms. Both data warehouses offer standard JDBC/ODBC connectivity for executing SQL queries from within apps. Redshift may have quicker learning curves for current AWS application teams. But Snowflake offers SDKs for more turnkey embedding across diverse tech stacks. Security and Compliance: Building Trust with Embedded Data Embedded analytics puts live data directly into apps, so security and controls are paramount. Both Snowflake and Redshift enable enterprise-grade user access controls, encryption and data governance capabilities leveraging the underlying cloud infrastructures. For highly regulated industries, Snowflake offers additional native capabilities to track data usage, mask sensitive data and implement fine-grained access policies. Big Data Challenges of Redshift vs Snowflake: When Volume and Variety Demand More As use cases expand to big data sources like , clickstreams or genomics data, the volume, velocity and variety of data can push conventional systems over the edge. Ingesting semi-structured data like JSON events get tricky. (Although ) IoT analytics Qrvey handles all data natively Serverless options on Snowflake like Snowpark handle varied data with less friction. Handling data volumes above 100s of TB can stretch Redshift capabilities. At massive scales, Snowflake better absorbs extreme spikes in storage and concurrent users. Picking the Champion for Your Use Case in This Redshift vs Snowflake Decision Cost Considerations: Balancing Budget and Performance AWS Redshift follows typical cloud pay-as-you-go pricing with node-based commitments. Cost efficiencies kick in at higher scales above a few TB. Snowflake’s per-second pricing and adaptive scaling remove overhead for idle clusters. But per-second billing can also lead to unexpected spikes on shared systems with uneven workloads. Cross-cloud deployment, data sharing and BYOL options on Snowflake provide more levers for optimization. Read more about or try our . Snowflake cost optimization Snowflake Cost Optimization Calculator Technical Requirements: Matching Capabilities to Needs Redshift provides a tightly coupled solution with quick time-to-value for simpler analytics integrated into AWS-centric application environments. More complex use cases like large-scale machine learning, and hybrid transactional/analytical processing may benefit from Snowflake’s more advanced architecture. Snowflake better fulfills needs for multi-cloud flexibility or rich data-sharing ecosystems. Choosing a Platform to Grow With: Redshift vs Snowflake Snowflake’s platform is cloud-based offering fast innovation in security, compliance, data science, and governance. This makes it an ideal solution for the long term…assuming costs are kept in check. The underlying separation of storage and computing eases future migrations. Future-proofing for unforeseen changes favors Snowflake, but Redshift is still likely a good option. Redshift vs Snowflake: Collaboration and Hybrid Solutions The data warehousing landscape continues to evolve rapidly, with the boundaries between Redshift, Snowflake and other platforms becoming more porous over time. Rather than a winner-take-all dynamic, we see increasing convergence and collaboration between platforms. Many organizations leverage hybrid solutions with Redshift for high-intensity operational workloads integrated with Snowflake for larger-scale data science experiments. Connectors like the recently launched make interoperation easier. AWS Redshift integration for Snowflake As analytics use cases grow more sophisticated, matching the ideal platform to each specific embedded scenario will unlock more value than a one-size-fits-all choice. Takeaway: Embracing the Right Data Warehouse for Your Embedded Analytics Journey The data warehousing engine powering embedded analytics should align with technical requirements, cost constraints, and future ambitions. Both AWS Redshift and Snowflake bring unique strengths as the foundation for real-time data applications. How Qrvey is Different At Qrvey, we know that a strong data layer is the foundation that makes any embedded analytics solution successful. We are the only solution with a built-in data warehouse layer made for multi-tenant, security-first embedded analytics. However, did you know that while we connect with Redshift, Snowflake, PostGres and more, we know don’t use any of these for our native data warehouse? Discover to power our embedded analytics for SaaS applications solution. why we chose AWS OpenSearch Also published here.

AWS Redshift vs Snowflake: A Comprehensive Guide to Embedded Analytics Solutions

AWS রেডশিফ্ট বনাম স্নোফ্লেক: এম্বেডেড অ্যানালিটিক্স সলিউশনের জন্য একটি ব্যাপক গাইড

AWS Redshift vs. Snowflake: Ein umfassender Leitfaden für eingebettete Analyselösungen

AWS Redshift vs Snowflake: una guía completa para soluciones de análisis integradas

AWS Redshift vs Snowflake : un guide complet des solutions d'analyse embarquées

एडब्ल्यूएस रेडशिफ्ट बनाम स्नोफ्लेक: एंबेडेड एनालिटिक्स सॉल्यूशंस के लिए एक व्यापक गाइड

AWS Redshift と Snowflake: 組み込み分析ソリューションの包括的なガイド

AWS Redshift와 Snowflake 비교: 내장형 분석 솔루션에 대한 종합 가이드

AWS Redshift vs Snowflake: um guia abrangente para soluções analíticas incorporadas

AWS Redshift против Snowflake: комплексное руководство по решениям для встроенной аналитики

AWS Redshift ve Snowflake Karşılaştırması: Yerleşik Analitik Çözümlerine Yönelik Kapsamlı Bir Kılavuz

AWS Redshift vs Snowflake: Hướng dẫn toàn diện về giải pháp phân tích nhúng

AWS Redshift 与 Snowflake：嵌入式分析解决方案综合指南

In the dynamic landscape of data analytics, choosing an analytics platform can significantly impact your business's bottom line. In this educational article, we're on a quest to explore three heavyweight analytics contenders: Snowflake, BigQuery, and ClickHouse. We will delve into their cost models and reveal valuable strategies to help you master the art of cost-efficient analytics – it's a journey exploring insights that can transform your data game! Understanding Analytic DBMS Cost Models To make cost-effective decisions in data analytics, it's crucial to comprehend the cost models behind analytical database management systems (DBMS). A common starting point is to examine the cost structure of cloud-based businesses. So, first, let's talk about cloud-based analytic databases and how they handle costs when running in the cloud. Picture this: It’s before 2013, and setting up a data warehouse could take up to several months of dealing with multiple vendors, hardware bottlenecks, and configurations. Then came , allowing you to start your data warehouse in under 20 minutes with just a credit card—quite the leap forward. This paved the way for other cloud-based data warehouses like , , and cloud services that operate Snowflake. Amazon Redshift in 2013 BigQuery Snowflake Most developers learn about the importance of cost-efficiency as soon as they start getting their cloud bills. For example, you could run a long-running query on BigQuery that ends up costing you hundreds, often thousands of dollars. Take it from the following tweet/X post; this is not an isolated occurrence. https://twitter.com/FredKSchott/status/1631449132358598656?embedable=true We recently ran a webinar discussing this topic in further detail where Robert Hodges (CEO at Altinity Inc.) goes into the in-depth cost analysis of each option and shows you an alternative DIY solution at the end. Feel free to watch the on-demand once you’re done reading this article! recording of that webinar So, let's dig into how these databases work and, more importantly, how they price their services. We'll start with Snowflake, but before we get into its architecture, we will talk business. Snowflake, being a publicly traded company, gives us a chance to poke around its financials. In , they raked in a whopping $2 billion in total revenue. What's intriguing is the cost of delivering their service, roughly around $717 million. This cost, also known as the cost of goods sold, is what it takes Snowflake to run the show. a recent report Understanding the cost of running Snowflake's cloud services is like a hidden treasure hunt. If you crunch the numbers, it's roughly about one-third, maybe a tad more, of their total revenue. So, in plain English, if all that cost went straight into covering their cloud expenses and nothing else, they'd be marking things up by roughly three times when they charge you. But of course, it’s not that simple! The actual costs of running a powerhouse like Snowflake go beyond just running virtual machines and stashing data in Amazon S3. Now, if we do the math again, that markup on their costs? It's more like 5x. That is a nifty benchmark to carry in your back pocket. If something's got a markup greater than 5x, well, it's living large in the world of Snowflake's pricing. On the flip side, if it's less than 5x, you're looking at a more budget-friendly option. It's like having a secret decoder ring for deciphering the cost puzzle. Unpacking the Cost Models of Snowflake, BigQuery, and ClickHouse To master cost-efficient analytics, let's dissect the cost models of Snowflake, BigQuery, and ClickHouse. Snowflake Virtual Data Warehouse Model Now, let's break down Snowflake's costs. They use a "virtual data warehouse" model, where your data resides in S3 object storage. When you run SQL queries, you create virtual data warehouses powered by credits, essentially hosted processing units pulling data from storage. The pricing for these virtual machines ranges from about $2-4 /hour as listed in their pricing. Keep in mind: it's not the thriftiest choice. This would wrap up our Snowflake cost analysis if not for this intriguing twist: a recent bug in Snowflake unveiled that the credits for virtual data warehouses often translate to c5d2x large instances, costing about 38 cents per hour. Remarkably, Snowflake doesn't significantly mark up object storage, pricing it at $23-40 per terabyte per month, similar to Amazon's S3 costs. Instead, the real markup occurs in computing, which can be 5 to 10 times more expensive than storage. BigQuery Serverless Query Model Let's take a closer look at BigQuery, another heavyweight in cloud database analytics. BigQuery offers a distinct pricing model known as "serverless" or "on-demand." In this setup, you store your data on a unique distributed storage system, not the typical object storage used by most cloud services. However, the pricing is quite competitive, similar to object storage rates, at least for uncompressed data. Prices start at $0.016-0.023 per GB per month for storage—now that's more budget-friendly. Here's the catch: when you run a query, BigQuery dynamically allocates compute resources as needed. It charges you $6.25 for every terabyte of data scanned during your query processing. This pricing structure means that even seemingly innocent queries can rack up costs if they involve scanning a large amount of data from distributed storage. Now, comparing BigQuery costs to standard cloud resources isn't straightforward. Your actual expenses depend on various factors, like how often you run queries and your data storage methods. For instance, if you run queries sporadically, BigQuery might be cost-effective. But if you're running queries 24/7, it could get expensive. So, it's crucial to understand your workload and assess your real costs carefully. AWS RedShift "Buy the Box" Model The previous models we’ve discussed typically rely on object storage, or something quite similar, for data handling. However, there's another cloud database operating approach, one initially introduced by a decade ago. Let’s call it the "Buy-the-Box" model. Redshift Here's the deal: you rent a ), such as a dc28x large, which comes with attached SSD block storage. For instance, in Amazon us-west-2, this VM would cost you around 4.80 cents per hour. Now, let's break down the actual costs. This VM likely corresponds to an i38x large instance, an older Amazon instance type, which comes with local SSD storage. This type of instance costs approximately $2.50 per hour, offering nearly the same amount of RAM. virtual machine (VM In terms of markup, Redshift is 92% more expensive while providing 66% less storage than if you were to set up a similar VM yourself. Interestingly, Redshift's markup is relatively lower compared to some other services, like Snowflake, which tends to charge more for computing resources. ClickHouse Database Architecture To improve on the previous models for cost efficiency, let us first introduce you to ClickHouse.  It is a free, renowned, open-source, real-time analytics database. ClickHouse’s architecture revolves around interconnected servers capable of seamless data replication, especially with replicated tables. This system employs efficient columnar storage, where data resides in compressed arrays, not only reducing storage costs but also boosting query performance. Initially, ClickHouse was confined to block storage, but it has evolved to support S3-compatible object storage, making it versatile and open to integration with any S3 API-equipped object storage solution. To manage replication consensus efficiently, ClickHouse relies on either ClickHouseKeeper or ZooKeeper. You can read more about Clickhouse . here The Modernized "Buy-the-Box" Approach Now, let's envision this as a cloud service paradigm, which we term "Modernized Buy-the-Box." On the left side of the image below is the traditional Redshift architecture, while on the right, we embrace innovation. We replace the old i3 instances with faster Intel-based m6is, providing a significant speed boost. The game-changer is the use of EBS (Elastic Block Storage) gp3 storage, granting control over bandwidth and throughput. This, paired with efficient VMs akin to Redshift, results in an approximate cost of 2.64 cents per hour. The real magic happens with the separation of storage and computing. This flexible approach lets you easily adjust CPU and VM types while keeping the same storage, allowing cost scaling up or down. Our experience at Altinity.Cloud managing ClickHouse applications reflects this efficiency. In the graph below, you'll see the all-in on-demand cost for when running ClickHouse. For instance, opting for m6i 12x large might cost six dollars per hour, slightly more than Redshift. However, ClickHouse is so fast that smaller instance sizes get better performance than Redshift, which means lower cost without compromising performance. This showcases the tremendous benefits of separating storage and computing. Altinity.Cloud To summarize, we've explored three cloud-hosted analytic database models and their cost implications. In our comparison, we've created a wallet-sized table to make it clear how these models stack up against each other. The "Buy-the-Box" approach offers cost-effective computing but pricier storage due to block storage usage. Snowflake and BigQuery, on the other hand, offer economical storage but can be costly in different ways. Snowflake tends to be relatively expensive in terms of computing, while BigQuery's on-demand query model can become pricey when handling extensive data scans. Each model has its strengths and weaknesses, making it essential to align them with your specific analytics needs. "Buy-the-Box" suits customer-facing analytics with unpredictable workloads, while the virtual data warehouse model and BigQuery excel in specific scenarios but require careful cost management to avoid surprises. This overview helps you navigate the landscape effectively. In Conclusion When it comes to analytics, cost-efficiency is paramount. Understanding the cost models of popular platforms like Snowflake, BigQuery, and ClickHouse is essential for making informed decisions. By evaluating each platform's strengths and weaknesses and considering their cost structures, organizations can craft cost-efficient analytic solutions tailored to their specific needs. Leveraging open-source solutions and educational resources can further optimize cost, ensuring organizations achieve their analytics goals while being mindful of their budgets. This article was derived from an webinar. Altinity.com

Snowflake vs. BigQuery vs. ClickHouse: Mastering Cost-Effective Business Analytics

স্নোফ্লেক বনাম BigQuery বনাম ক্লিকহাউস: ব্যয়-কার্যকর ব্যবসায়িক বিশ্লেষণ আয়ত্ত করা

Snowflake vs. BigQuery vs. ClickHouse: Kosteneffiziente Geschäftsanalysen meistern

Snowflake, BigQuery y ClickHouse: dominar el análisis empresarial rentable

Snowflake vs BigQuery vs ClickHouse : maîtriser l'analyse commerciale rentable

स्नोफ्लेक बनाम बिगक्वेरी बनाम क्लिकहाउस: लागत प्रभावी बिजनेस एनालिटिक्स में महारत हासिल करना

Snowflake vs. BigQuery vs. ClickHouse: 費用対効果の高いビジネス分析をマスターする

Snowflake vs. BigQuery vs. ClickHouse: 비용 효율적인 비즈니스 분석 마스터하기

Snowflake x BigQuery x ClickHouse: dominando análises de negócios econômicas

Snowflake против BigQuery против ClickHouse: освоение экономичной бизнес-аналитики

Snowflake, BigQuery ve ClickHouse: Uygun Maliyetli İş Analitiğinde Uzmanlaşma

Snowflake so với BigQuery so với ClickHouse: Làm chủ phân tích kinh doanh hiệu quả về chi phí

Snowflake、BigQuery 与 ClickHouse：掌握经济高效的业务分析

By now you must have read quite a few articles about Snowflake’s absolutely mind-blowing and record-setting IPO. This article is not intended to speculate on whether the valuation makes sense or not, but rather help you understand the technological concepts that make Snowflake so unique, and why it has proven to be so disruptful for the data space in general and the data warehousing space in particular. Background: data warehouses start their journey to the cloud To understand the market Snowflake operates in, it’s important to first spend a few minutes on the history of data warehouses in the cloud. Data warehouses have been around forever on-premises, but their appearance in the cloud is still relatively recent, with the release of Amazon Redshift in 2013. It was the first cloud data warehouse that seemed to be headed towards becoming mainstream. It was only natural that AWS, as the leader of the cloud revolution, would release one of the first successful pure cloud data warehouses. Redshift quickly gained popularity and became the warehouse of choice both for companies that were transitioning from on-prem to the cloud (the name “Redshift” was a poke at Oracle — shifting from “Red” which is Oracle’s color), and for new cloud-first companies. The biggest benefit that ca9`me with Redshift was the fact that it is a managed cloud service. This meant that the ease of use of the cloud and SaaS finally became available to data warehousing. Provisioning a new warehouse suddenly was not a lengthy and complex project involving hardware and software, but rather a few mouse clicks with Redshift taking care of the heavy lifting behind the scenes. But beyond reducing complexities around installations, provisioning and maintenance, Redshift does the same things in the same manner that traditional on-premises warehouses do. It also shares the same limitations. Google and Microsoft released cloud data warehouse offerings of their own (Google’s BigQuery actually preceded Redshift), but neither was as successful as Redshift in grabbing market share and becoming the warehouse choice for on-prem to cloud transitions. ‘Decoupled storage and compute’ and simplicity as the foundation for Snowflake’s popularity In the last few years you’ve heard the term ‘elasticity’ a million times in the context of the cloud revolution. But the story of elasticity made a significant leap specifically for data warehousing thanks to the ‘decoupled storage and compute’ concept. Although there are many product capabilities behind Snowflake’s success, we will focus on decoupled storage and compute because it’s the most important of them all. Quick reminder — what is elasticity? A simple explanation of elasticity in the cloud can be found in : “the degree to which a system is able to adapt to workload changes by provisioning and de-provisioning resources in an autonomic manner…”. However, the fact that something is in the cloud doesn’t mean it automatically enjoys elasticity. This is especially true for data warehouses. Wikipedia What is ‘decoupled storage and compute’ and what does it have to do with elasticity? Decoupling storage and compute enables users to quickly change the type and size of compute (clusters of CPUs) that is assigned to be the processing unit over the storage layer (hard drives, SSDs, etc). The cloud made this approach easily accessible to software, but it took time for software products to leverage this architectural concept. Snowflake was the first cloud data warehouse that made it super simple for users to enjoy the benefits of decoupled storage and compute. No matter how much data you import into Snowflake, at any point in time you can decide which compute resource you want to use for your next workload with a few mouse clicks. For example — you might decide to use a small (and therefore relatively cheap) compute cluster to conduct some research over a sample of your data. Then you might decide to use a bigger (and therefore more expensive) cluster for production dashboards that are viewed by many people throughout the day. Since you’re typically billed per usage in the cloud, this approach provides granular control over your spend. You can literally choose on a per-query level which hardware to use. This is completely different from the traditional on-premise world, where the same hardware configuration you implemented for your warehouse always runs all queries, no matter what. Now back to elasticity — Snowflake’s approach to decoupled storage and compute, and making it easily accessible for users through no more than a few clicks, brought a new level of elasticity to data warehouses. From a product perspective, it is the key reason for Snowflake’s popularity among data professionals. What’s next for data warehouses in the cloud The benefit of decoupled storage and compute for data warehouses and cloud data technologies in general is undeniable. This is one of the key reasons users like Snowflake. Decoupled storage and compute . Amazon Redshift recently added both Amazon Spectrum and RA3 nodes for this reason. has become the architecture of choice for new data technologies As we built Firebolt on a decoupled storage and compute architecture, we also came to realize what was needed to change in storage and compute to solve many of the other problems with data warehouses, and there are plenty of them remaining. One of the bigger ones is being able to . Achieving fast query response times for big data sets is still something that modern warehouses struggle with, and when it’s possible, it typically comes with a price tag that only big enterprises can afford. Looking at public benchmarks comparing Snowflake, Redshift and BigQuery it’s clear that already in the 1TB scale of data, query runtime easily takes 7 seconds or more on average, sometimes even minutes ( ). As data keeps growing, and as being able to deliver great user experiences over data becomes increasingly important for businesses, ‘performance at scale’ limitations is what the market will be looking for the tech industry to solve. analyze huge data sets, quickly and easily without breaking the bank See Fivetran’s great benchmark post here A decade ago BI meant that a smaller group of analysts performed analytics against historical data using advanced BI tools to support decisions for the rest of the company. A decade from now we expect 100x or more employees and customers to be performing self-service analytics against historical and real-time data using newer tools that leverage machine and deep learning and other computing — all to help them make faster and better decisions on their own. Snowflake’s valuation reflects the future potential. But the data industry has a long way to go till it gets there. This is why from a technology perspective the future of cloud data technologies belongs to those that will be able to analyze more data at higher speeds, while relying on less hardware and compute power to do so, to reduce costs. This belief led us at Firebolt to start by concentrating on technology that can get much more analytic horsepower from existing hardware, as the foundation on top of which simplicity and the benefits of the cloud can bring powerful analytics to the masses.

Understanding the tech behind Snowflake’s IPO and what’s to come

Why PostgreSQL Is the Bedrock for the Future of Data

কেন PostgreSQL ডেটার ভবিষ্যতের ভিত্তি

Warum PostgreSQL das Fundament für die Zukunft der Daten ist

Por qué PostgreSQL es la base del futuro de los datos

Pourquoi PostgreSQL est le fondement de l'avenir des données

PostgreSQL डेटा के भविष्य का आधार क्यों है?

PostgreSQL がデータの未来の基盤となる理由

PostgreSQL이 데이터 미래의 기반인 이유

Por que o PostgreSQL é a base para o futuro dos dados

Почему PostgreSQL является основой будущего данных

PostgreSQL Neden Verinin Geleceğinin Temel Taşıdır?

Tại sao PostgreSQL là nền tảng cho tương lai của dữ liệu

为什么 PostgreSQL 是未来数据的基石

I recently spoke with Marcel Fohrmann, the Co-Founder and CFO of Subsquid, a decentralized data lake and query engine in Web3, aiming to make canonical blockchain data more accessible and affordable for developers. Our conversation delved into the intricacies of the Web3 landscape, the pressing issues facing the industry, and how decentralized solutions can provide better access to blockchain data. Welcome Marcel. Thanks for joining me. Let’s kick this off with your views on how 2023 went and 2024 will go. Thanks for having me. 2023 was better than most people think. We started on a somber note. Series of upsets, negative events, and regulatory onslaught. Markets bled often, and it felt like winter overall. But some of Web3’s most groundbreaking innovations happened in 2023. Projects with long-term vision were hard at work while the noise was low. Take Layer-2 protocols for example. Optimism, Metis, Base, etc., took giant strides towards faster, cheaper, and more secure blockchain transactions. They latched on to developments in Multi-Party Computation (MPC), Zero-Knowledge Proofs, and other such privacy/security-enhancing solutions. Even L1s like Solana achieved significant growth, with over 2500 developers building on the network and DEX trading volumes surpassing Ethereum. This improved Web3’s diversification and fostered healthy competition. Likewise, we launched Subsquid to make blockchain data more affordable and efficient for developers. The point is 2023 was about improving and expanding the core infrastructure. Scalability, interoperability, security, and a better user experience were the main focus areas. While also improving profitability and sustainability for businesses and for-profit projects. It’s a win-win situation in the making. This will continue and grow in 2024. I’m super bullish about that. We have positive figures and data to show the progress from 2023. Now, it’s about honing in on what’s working and doing it better going forward. We’re at it already. To set realistic expectations, what are the biggest challenges for Web3 right now? In general and, of course, in your sector. Security is one of the main challenges. Hackers are becoming smarter and more driven by the day. With so much money at stake, that’s pretty obvious. Web3 needs stronger security mechanisms that align with the principles of decentralization, autonomy, and privacy. Building them is a tough task, but we’re getting there fast. Decentralized, real-time threat monitoring and mitigation systems are already functional and effective. It’s about raising collective security awareness and making best practices the norm. Cost and scalability are the other main obstacles. They have been for many years now. And though most people don’t see it this way, these are essentially data availability and retrieval problems. Let me explain. Any blockchain-based app or platform must retrieve on-chain data. Be it for updating wallet balances or the latest level of in-game avatars. How fast and cheap you do this determines your solution’s scalability and cost-effectiveness. But Web3 developers have to rely heavily on RPC endpoints with severe rate limitations. This skyrockets data retrieval costs. It also hurts the performance and slows networks down. Especially when there’s high traffic—i.e., low scalability and bad UX. Building a decentralized data retrieval system for fast, responsive, and secure blockchain-based actions became a necessity. We filled that gap. Tell us a bit more about how you’re solving these problems and what that could mean for Web3’s future. Sure. We enable Web3 developers to access canonical blockchain data at 90% less costs compared to RPCs. In Web2, centralized data lakes like BigQuery, Snowflake, and others store and provide access to archival application data at scale. Subsquid is a decentralized, community-driven version of this critical system. It aggregates and filters Web3 app data, making it easily extractable at scale. Without forcing developers to go through complex query processes and whatnot. But while this decentralized data lake takes inspiration from its centralized counterparts, it’s not a replica. It supports infinite horizontal scalability, permissionless data access, credible neutrality, trust-minimized queries, and low maintenance costs. I won’t go into the details here. Anyone who’s interested can read our white paper at . subsquid.io Coming to the impact on Web3’s future. Even with simple, localized queries, Subsquid’s model provides a novel primitive for on-chain data availability. It eliminates the need to maintain expensive archival nodes for historical data access. Unrestricted access to archival data enables developers to build data-intensive blockchain applications at scale. This opens the path for highly complex solutions with deep functionalities that are feasible for providers to maintain and seamless for end-users. That’s the balance we need for Web3’s adoption. Maximalism is all the rage in Web3. But you’ve been vocal about multi-chain systems. Please tell us why they’re important. Bitcoin is among the best tech inventions in mankind’s history. It introduced the stable, functional form of blockchain, which has since enabled such cutting-edge innovations. But with Ethereum revolutionizing general-purpose computing, we’ve come a long way from Bitcoin’s single-chain, single-utility paradigm. The multi-chain landscape is the best way to utilize blockchain’s full potential. It’s not merely about the store of value or payment anymore. DeFi, RWAs, DeSoc, DePIN, GameFi—so many possibilities have emerged, and ignoring them for maximalism is shooting yourself in the foot. From a data access perspective, multi-chain systems help break silos and solve data fragmentation. They allow developers to access the entire data pool easily. Better interoperability, scalability, and innovation, and you can build more robust dApps this way. Thanks for the insight. So, you’ve been doing great work with Subsquid. What’s next on this journey? It’s more important than great. Strengthening Web3’s core infrastructure is a must, and that’s what we’re working to achieve. The next big thing for us is launching support for Solana. Ethereum maxis won’t agree, but Solana is one of the most resilient and high-performance Alt-L1s right now. It also proved that L2s aren’t the only viable scalability solution available. With a Nakamoto Coefficient of 22, Solana has a solid validator distribution while optimizing for latency and throughput. Growth is another reason why we’re excited about Solana. We’ve seen how it bounced back from the ashes after the Alameda crisis last year to become one of the top five chains by market cap. Above all, though, we’re rooting for Solana’s tech stack. Its innovative token standard supports fast, inexpensive, and confidential transactions—great for balancing privacy and UX. Compressed NFTs are exciting as well, and the potential launch of new clients will make the ecosystem more decentralized. Finally, as a progressive project, Subsquid is always at the cutting-edge of innovation. Solana is set to become the hub for futuristic developments, from DePIN to DeSoc. We’ll provide the infrastructure to make that happen—it’s good for all. Before the wrap, what’s your advice to newcomers who might be entering Web3 now that markets are rising again? Learn to filter the noise. Focus on signals. The sooner you achieve this, the smoother your Web3 journey will be. There’s so much to do, it’s overwhelming. Set your priorities, and they’ll guide you better than anything else. I’d also ask developers and project owners to build core solutions, optimized for experience and utility. Hype won’t get you far unless you deliver real, long-term value. No one’s a fool anymore, so don’t take things for granted. The future is yours to build.

The Future of Web3 Data: An Interview with Subsquid CFO & Co-Founder Marcel Fohrmann

What data management in 2024 and beyond will look like hangs on one question. Can open data formats lead to a best-of-breed data management platform? It will take Interoperability across clouds and formats, as well as on the semantics and governance layer. Sixth Platform. Atlas. Debezium. DCAT. Egeria. Nessie. Mesh. Paimon. Transmogrification. This veritable word soup sounds like something that jumped out of a role-playing game. In reality, these are all terms related to data management. These terms may tell us something about the mindscape of people involved in data management tools and nomenclature, but that’s a different story. Data management is a long-standing and ever-evolving practice. The days of “Big Data” have long faded and data has now taken a back seat in terms of hype and attention. is the new toy people are excited about, shining a light on AI and machine learning for the masses. Generative AI Data management may seem dull in comparison. However, there is something people who have been into AI before it was cool understand well, and people who are new to this eventually realize too: AI is only as good as the data it operates on. There is an evolutionary chain leading from . Some organizations were well aware of this a decade ago. Others will have to learn the hard way today. data to analytics to AI Exploring and understanding how, where, and when data is stored, managed, integrated and used as well as related aspects of data formats, interoperability, semantics, and governance may be hard and unglamorous work. But it is what generates value for organizations, along with -like word soups. RPG Peter Corless and Alex Merced understand this. Their engagement with data goes back to way before their current roles as Director of Product Marketing at and Developer Advocate at , respectively. We caught up to talk about the state of data management in 2024, and where it may be headed. StarTree Dremio https://www.youtube.com/watch?v=Meq247f31-A&embedable=true The sixth data platform For many organizations today, data management comes down to handing over their data to one of the “Big 5” data vendors: , Azure, and , plus Snowflake and Databricks. However, analysts David Vellante and George Gilbert believe that the need of modern data applications coupled with the evolution of open storage management may lead to the emergence of a “sixth data platform”. Amazon Microsoft Google The notion of the “sixth data platform” was the starting point for this conversation with Corless and Merced. is that open data formats may enable interoperability, leading the transition away from vertically integrated vendor-controlled platforms towards independent management of data storage and permissions. The sixth data platform hypothesis It’s an interesting scenario and one that would benefit users by forcing vendors to compete for every workload based on the business value delivered, irrespective of lock-in. But how close are we to realizing this? To answer this question, we need to examine open data formats and their properties. In turn, to understand data formats a brief historical overview is needed. https://www.youtube.com/watch?v=j7s8xSTxkYk&embedable=true We might call this table stakes, and it’s not just a pun. Historically, organizations would resort to using data warehouses for their analytics needs. These were databases specifically designed for analytics, as opposed to transactional applications. Traditional data warehouses worked, but scaling them was an issue as Merced pointed out. Scaling traditional data warehouses was expensive and cumbersome because it meant buying new hardware with storage and computing bundled. This was the problem Hadoop was meant to solve when introduced in the early 2010s, by separating storage and compute. Hadoop made scaling easier, but it was cumbersome to use. This is why an SQL interface was eventually created for Hadoop via , making it more accessible. Hive used metadata and introduced a protocol for mapping files, which was what Hadoop operated on, to tables, which is what SQL operates on. That was the beginning of open data formats. Apache Hive Data lakes, data lakehouses, and open data formats Hadoop also signified the emergence of the data lake. Since storage was now cheap and easy to add, that opened up the possibility of storing all data in one big Hadoop store: the data lake. Eventually, however, , which is the realm the “Big 5” operate on. Hadoop and on-premise compute and storage gave way to the cloud “The idea of a data lakehouse was, hey, you know what? We love this decoupling of compute and storage. We love the cloud. But wouldn’t it be nice if we didn’t have to duplicate the data and we could just operate over the data store you already have, your data lake, and then just take all that data warehouse functionality and start trying to move it on there?”, as Merced put it. However, as he added, that turned out to be a tall order, because there were elements of traditional data management missing. Other cloud-based data management platforms introduced notions like automated sharding, partitioning, distribution, and replication. That came as part of the move away from data centers and file systems towards the cloud and APIs to access data. But for data lakehouses, these things are not a given. This is where open data formats such as and come in. They all have some things in common, such as the use of metadata to abstract and optimize file operations and being agnostic about storage. However, Hudi and Delta Lake both continued to build on what Hive started, while Iceberg departed from that. Apache Iceberg, Apache Hudi, Delta Lake https://www.youtube.com/watch?v=043GGYTZCbk&embedable=true A detailed open data format comparison is a nuanced exercise, but perhaps the most important question is: Does interoperability exist between open data formats, and if yes, on what level? For the sixth data platform vision to become a reality, Interoperability should exist among clouds and formats, as well as on the semantics and governance level. Interoperability of the same format between different cloud storage vendors is possible, but not easy. It can be done, but it’s not an inherent feature of the data formats themselves. It’s done on the application level, either via custom integration or by using something like Dremio. And there will be egress and network costs from transferring files outside the cloud where they reside. Interoperability of different formats is also possible, but not perfect. It can be done by using a third-party application, but there are also a couple of other options like or . Neither works 100%, as Delta Uniform batches transactions and OneTable is for migration purposes. Delta Uniform OneTable As Merced noted, creating a solution that works 100% would probably incur a lot of overhead and complexity as metadata would have to be synchronized across formats. Merced thinks these solutions are motivated at least partially by Iceberg’s growth and the desire to ensure third-party tools can work with the Iceberg ecosystem. To federate or not to federate? What’s certain is that there’s always gonna be sprinkles of data in other systems or multiple clouds. That’s why federation and virtualization that can work at scale is needed, as Corless noted. Schema management is complicated enough on one system, and trying to manage changes across three different systems would not make things easier. Whether data is left where it originally resides and used via federated queries, or ingested in one storage location is a key architectural question. As such, there are tradeoffs involved that need to be understood. Even within federation, there are different ways to go about it – Dremio, GraphQL, Pinot, Trino, and more. “How do data consumers want to size up these problems? How can we give them predictive ways to plan for these trade-offs? They could use federated queries, or a hybrid table of real time and batch data. We don’t even have a grammar to describe those kinds of hybrid or complex data products these days”, Corless said. Corless has a slightly different point of view, focusing on real-time data and processing. This is what StarTree does, as it builds on t. Pinot is a real-time distributed OLAP datastore, designed to answer OLAP queries with low latency. Apache Pino Corless also noted that the choice of data format depends on several parameters. Some formats are optimized for in-memory use, like Others like are optimized for disk storage. Apache Arrow. Apache Parquet There are attempts to have common representations of data both in-memory and on storage, motivated by the need to leverage tiered storage. Tiered storage may utilize different media, ranging from in-memory to SSD to some sort of blob storage. “People want flexibility in where and how they store their data. If they have to do transmogrification in real time, that largely defeats the purpose of what they’re trying to do with tiered storage. Is there something like the best, most universal format? Whenever you’re optimizing for something, you’re optimizing away from something else. But I’m very eager to see where that kind of universal representation of data is taking us right now.”, Corless said. Semantics and governance Regardless of where or how data is stored, however, true interoperability should also include semantics and governance. This is where the current state of affairs leaves a lot to be desired. As Merced shared, the way all table formats work is that the metadata has information about the table, but not about the overall catalog of tables. That means they are not able to document the semantics of tables and how they relate to each other. A standard format of doing that doesn’t exist yet. Merced also noted that there is something that addresses this gap, but it only works for Iceberg tables. is an open-source platform that creates a catalog, thus making tracking and versioning Iceberg tables and views possible. Project Nessie Nessie also incorporates elements of governance, and Merced noted that Nessie will eventually have Delta Lake support too. Databricks on its part offers which works with Delta Lake, but it’s a proprietary product. There is no lack of data catalog products in the market, but none of that can really be considered the solution to semantics and data interoperability. Unity Catalog Corless on his part noted that there a standard called DCAT. DCAT, which stands for Data Catalog Vocabulary, is an RDF vocabulary designed to facilitate interoperability between data catalogs published on the Web. DCAT was recently updated to v.3, it’s been around for over a decade, and it’s precisely aimed at interoperability. https://www.youtube.com/watch?v=Pb53L77_nS8&embedable=true The fact that DCAT is not used widely has probably more to do with vendors reinventing the wheel and/or aiming for lock-in, as well as clients not being more proactive in requiring interoperability standards than with DCAT itself, as per Corless. Unfortunately, it seems like what data is to AI, governance is to data for most organizations: a distant second at best. Forty-two percent of data and analytics leaders do not assess, measure or monitor their data and analytics governance, according to a . Those who said they measured their governance activity mainly focused on achieving compliance-oriented goals. 2020 Gartner survey The onset of . was an effort to standardize governance for data lakes leveraging DCAT and other metadata vocabularies. Today, however, both Data Lakes and Atlas seem to have fallen out of fashion. GDPR in 2018 marked the opportunity for governance to leverage metadata and semantics to rise up Apache Atlas A new project called was spun out of Atlas, aiming to address more than data lakes. Furthermore, there is another open standard for metadata and data lineage called and several lineage platforms that support it, including Egeria. Egeria OpenLineage Conclusion So, what is the verdict? Is the sixth data platform possible for data management in 2024 or is it a pipedream? Merced thinks that we’re already starting to see it. In his view, using open formats like Apache Iceberg, Apache Arrow, and Apache Parquet can create a greater level of interoperability. This interoperability is still imperfect, he noted, but it’s a much better state of things than in the past. Dremio’s thesis is to be an open data lakehouse, not just operating on the data lake, but across tools and data sources. Corless thinks that we’re going to see a drive toward clusters of clusters, or systems of systems. There’s going to be a reinforced drive towards the automation of integration. More like Lego bricks that snap together easily, rather than Ikea furniture that comes with a hex wrench. But for that to happen, he noted, we’ll need a language and a grammar so that systems and people can both understand each other. The ingredients for a sixth data platform are either there, or close enough. As complexity and fragmentation explode, a language and grammar for interoperability sounds like a good place to start. Admittedly, interoperability and semantics are hard. Even that, however, may be more of a people and market issue than a technical one. DCAT and OpenLineage are just some of the vocabularies out there. Even things as infamously hard to define as data mesh and data products have a vocabulary of their own – the . Data Product Descriptor Specification Perhaps then the right stance here would be cautious optimism. Also published . here

Data Management in 2024: Will Open Data Formats Shape a “Sixth Platform”?

253 Stories To Learn About Data Analysis

SNOWFLAKE COMPUTING

snowflake.net

-- employees

Since n.d.

COMPANY RANKING

SNOWFLAKE COMPUTING

EVERGREEN INDEX #9544

Representing this company? Click here to claim and customize this page!

Snowflake Computing WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

SNOWFLAKE COMPUTING

snowflake.net

-- employees

Since n.d.

COMPANY RANKING

SNOWFLAKE COMPUTING

EVERGREEN INDEX #9544

RELATED COMPANIESRANK

STORIES ABOUT SNOWFLAKE COMPUTING

Representing this company? Click here to claim and customize this page!

Snowflake Computing WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!