Fivetran, the company automating data movement out of, into, and across cloud data platforms, is announcing its recent achievement of HITRUST certification for the Fivetran Automated Data Movement Platform in all of its cloud regions,

Fivetran Achieves HITRUST Certification, Maintaining its Highly Secure, Cyber-Aware Status

In a demonstration of its industry-leading security posture, Fivetran, the global leader in automated data movement, today announced it has earned certified status by HITRUST for the Fivetran Automated Data Movement Platform in all of its cloud regions worldwide serving Amazon Web Services, Google Cloud Platform and Microsoft Azure customers.

Fivetran Attains HITRUST Implemented, 1-Year (i1) Certification to Manage Data Protection and Mitigate Cybersecurity Threats

Certification validates Fivetran’s commitment to support protection of highly sensitive information, including healthcare data OAKLAND, Calif., October 26, 2023--(BUSINESS WIRE)--In a demonstration of its industry-leading security posture, Fivetran ...

Net sentiment is a measure of the net percent of customers that intend to engage with the platform. It “nets out,” for example, customers that have no intention of engaging. Among components of an emerging 6th data platform, Starburst, dbt, Fivetran and AtScale show growing mindshare and have respectable or strong sentiment.

Getting ready for the sixth data platform

Each of these companies made the list of fastest growing startups. Need a job? This is a great resource for your search.

Dozens Of East Bay Companies On Inc. List Of Best Performing Startups

OAKLAND, Calif.--(BUSINESS WIRE)--Fivetran, the global leader in automated data movement, and Crane Worldwide Logistics, a leader in global supply chain solutions, have won Ventana Research’s ...

Fivetran and Crane Worldwide Logistics Win Ventana Research’s 2023 Digital Leadership Award

Fivetran, the global leader in automated data movement, and Crane Worldwide Logistics, a leader in global supply chain solutions, have won Ventana Research’s 2023 Digital Leadership Award in the ...

Fivetran, the global leader in automated data movement, and Crane Worldwide Logistics, a leader in global supply chain solutions, have won Ventana Research’s 2023 Digital Leadership Award in the analytics category. This award recognizes the organization ...

Fivetran, the global leader in automated data movement, and Crane Worldwide Logistics, a leader in global supply chain solutions, have won Ventana Research's 2023 Digital Leadership Award in the analytics category. This award recognizes the organization ...

Fivetran and Crane Worldwide Logistics Win Ventana Research's 2023 Digital Leadership Award

Fivetran

fivetran

How we store and manage data has completely changed over the last decade. We moved from an ETL world to an ELT world, with companies like Fivetran pushing the trend. However, we don’t think it is going to stop there; ELT is a transition in our mind towards EL+T (with EL decoupled from T). And to understand this, we need to discern the underlying reasons for this trend, as they might show what’s in store for the future. This is what we will be doing in this article. I’m the co-founder of , the new upcoming open-source standard for data integrations. Airbyte What are the problems with ETL? Historically, the data pipeline process consisted of extracting, transforming, and loading data into a warehouse or a data lake. There are serious disadvantages to this sequence. Inflexibility ETL is inherently rigid. It forces data analysts to know beforehand every way they are going to use the data, every report they are going to produce. Any change they make can be costly. It can potentially affect data consumers downstream of the initial extraction. Lack of visibility Every transformation performed on the data obscures some of the underlying information. Analysts won’t see all the data in the warehouse, only the one that was kept during the transformation phase. This is risky, as conclusions might be drawn based on data that hasn’t been properly sliced. Lack of Autonomy for Analysts Last but not least, building an ETL-based data pipeline is often beyond the technical capabilities of analysts. It typically requires the close involvement of engineering talent, along with additional code to extract and transform each source of data. The alternative to a complex engineering project is to conduct analyses and build reports on an ad hoc, time-intensive, and ultimately unsustainable basis. What changed and why ELT is way better Cloud-based Computation and Storage of Data The ETL approach was once necessary because of the high costs of on-premises computation and storage. With the rapid growth of cloud-based data warehouses such as Snowflake, and the plummeting cost of cloud-based computation and storage, there is little reason to continue doing transformation before loading at the final destination. Indeed, flipping the two enables analysts to do a better job in an autonomous way. ELT Supports Agile Decision-Making for Analysts When analysts can load data before transforming it, they don’t have to determine beforehand exactly what insights they want to generate before deciding on the exact schema they need to get. Instead, the underlying source data is directly replicated to a data warehouse, comprising a Analysts can then perform transformations on the data as needed. Analysts will always be able to go back to the original data and won’t suffer from transformations that might have giving them a free hand. This makes the business intelligence process incomparably more flexible and safe. “single source of truth.” compromised the integrity of the data, ELT Promotes Data Literacy Across the Whole Company When used in combination with cloud-based business intelligence tools such as Looker, Mode, and Tableau, the ELT approach also broadens access to a common set of analytics across organizations. Business intelligence dashboards become accessible even to relatively non-technical users. We’re big fans of ELT at Airbyte, too. But ELT is and has problems of its own. We think EL needs to be completely decoupled from T. not completely solving the data integration problem What’s changing now and why EL+T is the future Merging of Data Lakes and Warehouses There was a great analysis by Andreessen Horowitz about . Here is the architecture diagram of the modern data infrastructure they came up with after a lot of interviews with industry leaders. how data infrastructures are evolving Data infrastructure serves two purposes at a high level: Helps business leaders make better decisions through the use of data — analytic use cases Builds data intelligence into customer-facing applications, including via machine learning — operational use cases Two parallel ecosystems have grown up around these broad use cases. The data warehouse forms the foundation of the analytics ecosystem. Most warehouses store data in a structured format. They are designed to generate insights from core business metrics, usually with SQL (although Python is growing in popularity). The data lake is the backbone of the operational ecosystem. By storing data in raw form, it delivers the flexibility, scale, and performance required for applications and more advanced data processing needs. Data lakes operate on a wide range of languages including Java/Scala, Python, R, and SQL. What’s really interesting is that modern data warehouses and data lakes are starting to resemble one another — both offering commodity storage, native horizontal scaling, semi-structured data types, ACID transactions, interactive SQL queries, and so on. So you might be wondering if data warehouses and data lakes are on a path toward convergence. Will they become interchangeable in a stack? Will data warehouses also be used for the operational use case? EL+T Supports Both Use Cases: Analytics and Operational ML EL, in contrast to ELT, completely decouples the Extract-Load part from any optional transformation that may occur. The operational use cases are all unique in the way incoming data is leveraged. Some might use a unique transformation process; some might not even use any transformation. In regards to the analytics case, analysts will need to get the incoming data normalized for their own needs at some point. But decoupling EL from T would let them choose whichever normalization tool they want. has been gaining a lot of traction lately among data engineering and data science teams. It has become the open-source standard for transformation. Even Fivetran integrates with them to let teams use DBT if they’re used to it. DBT EL Scales Faster and Leverages the Whole Ecosystem Transformation is where all the edge cases lie. For every specific need within any company, there is a schema normalization unique to it, for each and every one of the tools. By decoupling EL from the T, this enables the industry to start covering the long tail of connectors. At Airbyte, we’re building a “ “ so we can get to 1,000 pre-built connectors in a matter of months. connector manufacturing plant Furthermore, as mentioned above, it would help teams leverage the whole ecosystem in an easier way. You start to see an open-source standard for every need. In a sense, the future data architecture might look like this: In the end, extract and load will be decoupled from transformation. Do you agree with us? If so, you might be interested to have a look at what Airbyte does. Previously published at https://airbyte.io/articles/data-engineering-thoughts/why-the-future-of-etl-is-not-elt-but-el/

A Guide on The Future of ETL: EL(T) not ELT

The Ultimate Directory of Apache Iceberg Resources

By Maxwell Dallinga Data is a pivotal aspect of every modern manufacturing and distribution business. It helps inform decisions, indicate areas of success and failure, and create unique insights to improve and innovate. Yet the collection and use of data can quickly get messy. Without a means to extract, aggregate, and sort through your data, gleaning meaningful insight will become next to impossible. So, how can you effectively manage your data? Many manufacturers, distributors, and supply chain organizations use data integration tools to gain meaningful insights from their ERPs. What Is Data Integration and Why is it So Important? Data integration is the process of unifying data from multiple (often disparate) sources, systems, and applications. Across an organization, integration is a vital step in keeping data consistent. By unifying data from previously disconnected or disparate sources, it allows an organization’s data to be effectively assessed and analyzed. In essence, data integration keeps your ERP connected, providing your organization with a robust and holistic view of your data. Despite this, integration can pose problems for manufacturers and distributors. The most prominent issue? Manual data integration. A recent study from Intowave [1] stated that around 74% of manufacturers are still relying on manual data integration techniques. This means using employees for data extraction, transformation, and aggregation to keep outdated systems operating. While manual integration can function, it is often at the expense of wild inefficiency. It drives up costs, takes copious amounts of time, and can damage the accuracy and reliability of data. As well, it often becomes a crutch to keep legacy ERPs connected and up to date. Overall, manual data integration can become a major roadblock for any manufacturer or distributor looking to sustainably scale and improve. To combat this, many organizations turn to data integration tools to provide data automation capabilities. As an automation solution, using a data integration tool can be extremely effective in fixing the inefficiencies of manual data integration. Top 3 Reasons You Should Adopt Data Integration Tools A recent study from Fivetran concluded that companies spend an average of $520K per year on data engineers that manually build and maintain data pipelines [2]. As such, finding a way to automate the integration process can result in huge cost-benefits. With such a massive margin to save, you may consider some of the many benefits provided by data integration tools: 1. Improvement of Data Accuracy Bad data, according to Christina R. Fritsch of The National Law Review [3], costs US companies upwards of 3 trillion dollars every year. This isn’t a number to take lightly. With modern business strategies putting huge reliance on data analysis, poor data management is a massive hindrance. Yet with data integration tools, you can significantly improve your organization’s data quality. For one, integration tools remove the potential for human error. Moreover, they can provide more reliable information on inventories, customers, sales, and other important metrics. With improved accuracy, you can put far more trust into your data, and in turn, your business decisions. 2. Superior Decision Making From stronger data accuracy, intelligence, and accessibility, you can ensure more informed decisions within your organization. In addition, integration tools make the speed at which data can be accessed and analyzed significantly faster. According to Agility PR [4], around 9 out of 10 companies cannot conduct real-time analytics from their ERPs. This problem, however, is largely mitigated by effective automation with data integration tools. Automating integration tasks previously done manually, you can ensure much quicker data analysis capabilities, helping you make timely decisions. Thus, with faster, more informative, and more reliable data, you can ensure superior decision making. 3. Greater Efficiency and Productivity Without data integration tools, companies spend around 44% of their time hiring engineers to build and maintain data pipelines [2]. Integration tools prevent this by eliminating costly errors and saving time, effort, and labor costs spent manually integrating data. Using data automation to perfect integration tasks previously executed by employees, integration tools improve the efficiency and productivity of your operation. More importantly, freeing employees from these manual tasks can be greatly proactive. Instead of wasting their talent, employees will be able to focus on scaling, improving, and innovating. The Best Data Integration Tools for Your Systems When looking to implement ERP integration effectively, you should consider which tool works best for the job. This means understanding which backend ERP systems you use to run your business. Listed below are some of the best data integration tools to consider so you can gain better ERP insights: Kore Integrate Kore Integrate is a data warehousing solution that will improve your data reporting, scalability, functionality, and speed. For those who use Rocket Software’s UniVerse and UniData MultiValue databases, Kore Integrate is one of the best integration solutions. Kore’s data integration tool can also pull from ODBC-enabled data sources like SQL, Postgres, Oracle, etc. By connecting disparate sources, you can create a unified view of the important data within your organization, while reducing operational costs. Infor OS Infor OS can help manufacturers and distributors integrate their Infor LN system. Aiding in workflows, OS can help connect all your disparate systems effectively. Providing workflow automation, Infor OS is a useful tool for anyone using an Infor LN and looking for ERP integration. With capabilities that meet your needs, Infor OS might be a great solution for you. Oracle Cloud Infrastructure Oracle Cloud Infrastructure (OCI) provides a robust suite of pre-built connectors to organizations using Oracle Fusion ERP. Offering great data automation capabilities, OCI can deliver strong and up-to-speed analytics for your Oracle system. Designed to connect both cloud and on-premise applications, it can help you integrate your ERP effectively. The Bottom Line for Supply Chain Organizations, Manufacturers and Distributors As data becomes increasingly important for modern manufacturers and distributors, being able to manage your data effectively is vital. The bottom line is that data integration tools are a necessary step to modernize your organization’s data systems. Ultimately, integration tools will save you time, money, and resources to put toward proactive aspects of your business. References: [1] Intoware, I. (2022, July). Break Free From Disconnected Data — Process & Control July Issue 2022. Break free from disconnected data — Process & Control July issue 2022. https://www.intoware.com/resources/break-free-from-disconnected-data [2] Fivetran (2021) Wakefield Research Analysis of Results for Fivetran. Fivetran https://get.fivetran.com/rs/353-UTB-444/images/2021-CDL-Wakefield-Research.pdf [3] Fritsch, C. R. (2022, December 23). 12 days of CRM: Day 9 — how much does bad data actually cost?. The National Law Review. https://natlawreview.com/article/12-days-crm-day-9-how-much-does-bad-data-actually-cost [4] Carufel, R. (2022, May 19). Data Disconnect: Over 80 percent of companies rely on stale data for decision-making. Agility PR Solutions.https://www.agilitypr.com/pr-news/public-relations/data-disconnect-over-80-percent-of-companies-rely-on-stale-data-for- By Maxwell Dallinga Maxwell Dallinga Data is a pivotal aspect of every modern manufacturing and distribution business. It helps inform decisions, indicate areas of success and failure, and create unique insights to improve and innovate. Yet the collection and use of data can quickly get messy. Without a means to extract, aggregate, and sort through your data, gleaning meaningful insight will become next to impossible. So, how can you effectively manage your data? Many manufacturers, distributors, and supply chain organizations use data integration tools to gain meaningful insights from their ERPs. What Is Data Integration and Why is it So Important? Data integration is the process of unifying data from multiple (often disparate) sources, systems, and applications. Across an organization, integration is a vital step in keeping data consistent. By unifying data from previously disconnected or disparate sources, it allows an organization’s data to be effectively assessed and analyzed. In essence, data integration keeps your ERP connected, providing your organization with a robust and holistic view of your data. Despite this, integration can pose problems for manufacturers and distributors. The most prominent issue? Manual data integration. A recent study from Intowave [1] stated that around 74% of manufacturers are still relying on manual data integration techniques. This means using employees for data extraction, transformation, and aggregation to keep outdated systems operating. While manual integration can function, it is often at the expense of wild inefficiency. It drives up costs, takes copious amounts of time, and can damage the accuracy and reliability of data. As well, it often becomes a crutch to keep legacy ERPs connected and up to date. Intowave Overall, manual data integration can become a major roadblock for any manufacturer or distributor looking to sustainably scale and improve. To combat this, many organizations turn to data integration tools to provide data automation capabilities. As an automation solution, using a data integration tool can be extremely effective in fixing the inefficiencies of manual data integration. Top 3 Reasons You Should Adopt Data Integration Tools A recent study from Fivetran concluded that companies spend an average of $520K per year on data engineers that manually build and maintain data pipelines [2]. As such, finding a way to automate the integration process can result in huge cost-benefits. With such a massive margin to save, you may consider some of the many benefits provided by data integration tools: Fivetran 1. Improvement of Data Accuracy Bad data, according to Christina R. Fritsch of The National Law Review [3], costs US companies upwards of 3 trillion dollars every year. This isn’t a number to take lightly. With modern business strategies putting huge reliance on data analysis, poor data management is a massive hindrance. Yet with data integration tools, you can significantly improve your organization’s data quality. For one, integration tools remove the potential for human error. Moreover, they can provide more reliable information on inventories, customers, sales, and other important metrics. With improved accuracy, you can put far more trust into your data, and in turn, your business decisions. The National Law Review 2. Superior Decision Making From stronger data accuracy, intelligence, and accessibility, you can ensure more informed decisions within your organization. In addition, integration tools make the speed at which data can be accessed and analyzed significantly faster. According to Agility PR [4], around 9 out of 10 companies cannot conduct real-time analytics from their ERPs. This problem, however, is largely mitigated by effective automation with data integration tools. Automating integration tasks previously done manually, you can ensure much quicker data analysis capabilities, helping you make timely decisions. Thus, with faster, more informative, and more reliable data, you can ensure superior decision making. Agility PR 3. Greater Efficiency and Productivity Without data integration tools, companies spend around 44% of their time hiring engineers to build and maintain data pipelines [2]. Integration tools prevent this by eliminating costly errors and saving time, effort, and labor costs spent manually integrating data. Using data automation to perfect integration tasks previously executed by employees, integration tools improve the efficiency and productivity of your operation. More importantly, freeing employees from these manual tasks can be greatly proactive. Instead of wasting their talent, employees will be able to focus on scaling, improving, and innovating. The Best Data Integration Tools for Your Systems When looking to implement ERP integration effectively, you should consider which tool works best for the job. This means understanding which backend ERP systems you use to run your business. Listed below are some of the best data integration tools to consider so you can gain better ERP insights: Kore Integrate Kore Integrate is a data warehousing solution that will improve your data reporting, scalability, functionality, and speed. For those who use Rocket Software’s UniVerse and UniData MultiValue databases, Kore Integrate is one of the best integration solutions. Kore’s data integration tool can also pull from ODBC-enabled data sources like SQL, Postgres, Oracle, etc. By connecting disparate sources, you can create a unified view of the important data within your organization, while reducing operational costs. Kore Integrate Kore Integrate Infor OS Infor OS can help manufacturers and distributors integrate their Infor LN system. Aiding in workflows, OS can help connect all your disparate systems effectively. Providing workflow automation, Infor OS is a useful tool for anyone using an Infor LN and looking for ERP integration. With capabilities that meet your needs, Infor OS might be a great solution for you. Infor OS Infor OS Oracle Cloud Infrastructure Oracle Cloud Infrastructure (OCI) provides a robust suite of pre-built connectors to organizations using Oracle Fusion ERP. Offering great data automation capabilities, OCI can deliver strong and up-to-speed analytics for your Oracle system. Designed to connect both cloud and on-premise applications, it can help you integrate your ERP effectively. Oracle Cloud Infrastructure The Bottom Line for Supply Chain Organizations, Manufacturers and Distributors As data becomes increasingly important for modern manufacturers and distributors, being able to manage your data effectively is vital. The bottom line is that data integration tools are a necessary step to modernize your organization’s data systems. Ultimately, integration tools will save you time, money, and resources to put toward proactive aspects of your business. References: [1] Intoware, I. (2022, July). Break Free From Disconnected Data — Process & Control July Issue 2022 . Break free from disconnected data — Process & Control July issue 2022. https://www.intoware.com/resources/break-free-from-disconnected-data Break Free From Disconnected Data — Process & Control July Issue 2022 [2] Fivetran (2021) Wakefield Research Analysis of Results for Fivetran. Fivetran https://get.fivetran.com/rs/353-UTB-444/images/2021-CDL-Wakefield-Research.pdf Wakefield Research Analysis of Results for Fivetran. [3] Fritsch, C. R. (2022, December 23). 12 days of CRM: Day 9 — how much does bad data actually cost? . The National Law Review. https://natlawreview.com/article/12-days-crm-day-9-how-much-does-bad-data-actually-cost 12 days of CRM: Day 9 — how much does bad data actually cost? [4] Carufel, R. (2022, May 19). Data Disconnect: Over 80 percent of companies rely on stale data for decision-making . Agility PR Solutions.https://www.agilitypr.com/pr-news/public-relations/data-disconnect-over-80-percent-of-companies-rely-on-stale-data-for- Data Disconnect: Over 80 percent of companies rely on stale data for decision-making

The 3 Main Reasons Manufacturers and Distributors Should Use Data Integration Tools

As we approach the second quarter of 2024, Artificial Intelligence (AI) is driving significant changes in the field of data engineering. Integrating data engineering with AI has led to the popularity of modern data integration and the expertise required. I want to highlight here how the rise of AI will change data engineering trends in 2024 compared to previous years. First thing first, a quick recap on traditional data engineering Let's quickly recap how the data engineering role is born and grows with the passage of time. Almost a decade ago, businesses realised the importance of data-driven decisions. That opened roles like with expertise in Microsoft SSIS/SSRS, Talend, Informatica, and other similar tools. However, with the emergence of social media applications, these traditional ETL development tools faced limitations in processing billions of records. The idea of distributed storage and processing came into the field of play, such as Spark and NoSQL databases. As a result, the demand for Hadoop/Spark has increased. Exciting Hadoop times ahead! BI/ETL Developers Big Data Developers In 2006, AWS launched its cloud services S3 and EC2, followed by Microsoft and Google launching Azure and Google Cloud in 2008. This led to an increase in the use of cloud technologies. The cloud technologies are based on a pay-as-you-use model that avoids buying expensive servers and operating system licenses. These cloud technologies have now been called "Modern Data Warehouses" since the union of Big Data and Cloud. In early 2010, r jobs rose in the market with the release of AWS EMR, etc.  Because now, each and every company wants to migrate from on-premise to the cloud. With the passage of time, modern cloud-based data warehousing solutions have become costly and inefficient in handling unstructured data, leading to the birth of file-based data lake solutions. Cloud Develope Different Variations of Data Engineers The Data Engineering Landscape is quite complex and vast. Engineers and businesses struggle to keep up with the latest tools and technologies to build and maintain a scalable data platform. Each business creates its definition of data engineers, and some require data engineers to focus primarily on creating pipelines. Some require software engineering and reverse engineering expertise or the ability to build KPI models as an analytics engineer along with different cloud-based specializations. This is the current state of Data Engineering variation and demand, but  this is not stopping here. I want to highlight some other trends that are rising this year. AI Meets Data Engineers Recently, I saw on LinkedIn that many positions are open for . They mostly wrote that the responsibilities are to build LLM Models or prepare the batch/streaming data for these AI-based models. This includes regular data engineering tasks to build pipelines for product feature enhancement and enable data-driven business with a centralized data warehouse or data lake solution. AI Data Engineers Quality Assurance Engineer Merge with Data Engineer As a Data Engineer by profession, I realise that a data team should have at least one Quality Assurance Engineer. The QA takes all the responsibility for testing data quality, as they used to do web or mobile application testing. That person is always up-to-date with business domain knowledge. While the engineers are mostly focusing on technical aspects. With the rising demand for Gen AI solutions like GitHub Codepilot and ChatGPT, there is no guarantee that their solutions are always reliable; hence, it increases data quality problems with their use in production. Quality is one of the biggest challenges for small to enterprise companies. You are definitely going to see a lot more job openings as in the coming days.  This role is going to be the next demanding job in the coming months. Data Quality Engineer Batch Data Processing Could Obsolete Soon Stream data integration and processing keep adding value daily for the business allowing them to make faster decisions and offer better product features. The CDC (Change Data Capture) is already taking place for batch pipeline integration, primarily for the main source database. However, Due to less demand for batch processing and the high output of stream processing architecture, it may slowly become obsolete or less required in the future. Lakehouse Architecture Will Remain The First Choice Hundreds and thousands of companies have already taken advantage of building Lakehouse Architecture. They moved from the costly “Modern cloud data warehouse“ to the file-based Lakehouse Architecture either using Delta Lake, Apache Hudi or Apache Iceberg open storage format, as its ability to manage large-scale data with ACID transactions and schema enforcement will remain the first choice for next some year(s) unless we see any other new tools or architecture in the market. The popularity of No Code Data Integration Tools Modern Data Integration Tools like Airbyte, Mage.ai, Stitch, and Fivetran simplify data integration and workflow and reduce the need for custom Python-based data pipeline development. These tools speed up the process of pipeline development, but I will come up with the cost of the complexity of debugging these tools if something breaks; it will not be very straightforward to fix it. On the other hand, expert engineers will be required to kick-start the pipeline integration in one click. I would like to see how long we can go to adapt these tools, no-code-based tools.

The Emerging Data Engineering Trends You Should Check Out In 2024

89 Stories To Learn About Big Data Analytics

361 Stories To Learn About Big Data

188 Stories To Learn About Analytics

Introduction The year 2023 has seen significant developments in innovation, adoption, and competition. HashiCorp ignited a substantial controversy when it altered the licenses of its products to be less open-source-friendly. This quickly led to the emergence of Terraform forks, such as OpenTofu. We also observed increased competition among established players like Fivetran and dbt, often collaborating to provide solutions. This brings us to the topic of this blog post—the Rust-based alternative to dbt, known as SDF (Semantic Data Fabric). So, what exactly is SDF? SDF Overview The origins of SDF can be traced back to Meta. It is written in Rust, meaning a single compact binary with no packages to manage. Once downloaded, you're ready to get started. SDF is built on Datafusion, which is part of the Apache Arrow project. With SDF, you'll primarily be working with YAML and SQL—languages that should be familiar to any data engineer. At its core, SDF is a compiler and build system that applies static analysis to SQL code. It can then generate a robust dependency graph for a clearer view of your data assets. This feature enables easier detection of potential problems and optimization issues. One of SDF's unique features is the capacity to annotate your SQL sources with metadata. This can range from simple types and classifiers (like PII) to table visibility and privacy policies. SDF refers to these as Checks. Here are some examples of the types of Checks you can perform: Data Privacy Check: Ensures all personally identifiable information (PII) is suitably anonymized. Data Ownership Check: Confirms that every table has an owner—a requirement of GDPR. Data Quality Check: Prevents calculations that mix different currency types (for example, preventing £ + $ ) The resulting information schema extends beyond standard features. You can use SQL to explore the information schema and even write your own Checks to integrate into your workflows. Additional Features The SDF engine includes a multi-dialect SQL compiler, a static analyzer, a dependency manager, and a build cache. The following images illustrate the , , and commands. deploy describe lineage These features are available in their CLI, and a cloud version offers an interactive interface to explore the catalog and lineage. You can easily zoom in and out, click on a node, and delve deeper. Here is a zoomed-out image: As previously mentioned, the metadata and configurations for SDF are in YAML and form an SDF Workspace. SDF extends YAML's typical configuration usage by allowing you to specify data asset definitions simultaneously, referred to as definition blocks. These blocks can be used to define or enrich tables, functions, classifiers, etc. Borrowing an example from their documentation, here's an illustrative SDF YAML workspace. These classifiers can conveniently tag your columns and their descendants throughout a project. Coupled with available functions, you gain substantial control and visibility over your data models. The compilation step and helpful error messages contribute to a comprehensive environment. Summary This project's compact nature is impressive; you can install it locally and experiment with minimal effort. It proposes a different approach to dbt, which may cause resistance for some. However, uncertainty remains about how this would perform in extensive, complex environments where there's a desire to deploy thousands of modules. Nevertheless, the Checks feature could be the solution to this concern. SDF is undeniably a valuable project that can meet many use cases. If you're weary of dbt or considering something akin to dbt, then SDF should probably be on your shortlist. You can read more "What the heck" articles at the following links: What The Heck Is DuckDB? What the Heck Is Malloy? What the Heck is PRQL? What the Heck is GlareDB? What the Heck is SeaTunnel? What the Heck is LanceDB?

What the Heck Is SDF?

What is Apache Doris? is an open-source, real-time data warehouse. It can collect from various data sources, including relational databases (MySQL, PostgreSQL, SQL Server, Oracle, etc.), logs, and time series data from IoT devices. It is capable of reporting, ad-hoc analysis, federated queries, and log analysis, so it can be used to support dashboarding, self-service BI, A/B testing, user behavior analysis, and the like. Apache Doris data Apache Doris supports both batch import and stream writing. It can be well integrated with Apache Spark, Apache Hive, Apache Flink, Airbyte, DBT, and Fivetran. It can also connect to data lakes such as Apache Hive, Apache Hudi, Apache Iceberg, Delta Lake, and Apache Paimon. Performance As a real-time OLAP engine, Apache Doris has a competitive edge in query speed. According to the TPC-H and SSB-Flat benchmarking results, Doris can deliver much faster performance than Presto, Greenplum, and ClickHouse. As for its self-volution, it has increased its query speed by over ten times in the past two years, both in complex queries and flat table analysis. Architectural Design Behind the fast speed of Apache Doris is the architectural design, features, and mechanisms that contribute to the performance of Doris. First of all, Apache Doris has a cost-based optimizer (CBO) that can figure out the most efficient execution plan for complicated big queries. It has a fully vectorized execution engine so it can reduce virtual function calls and cache misses. It is MPP-based (Massively Parallel Processing) so it can give full play to the user's machines and cores. In Doris, query execution is data-driven, which means whether a query gets executed is determined by whether its relevant data is ready, and this enables more efficient use of CPUs. Fast Point Queries for A Column-Oriented Database Apache Doris is a column-oriented database so it can make data compression and data sharding easier and faster. But this might not be suitable for cases such as customer-facing services. In these cases, a will have to handle requests from a large number of users concurrently (these are called "high-concurrency point queries"), and having a columnar storage engine will amplify I/O operations per second, especially when data is arranged in flat tables. data platform To fix that, Apache Doris enables hybrid storage, which means having row storage and columnar storage at the same time. In addition, since point queries are all simple queries, it will be unnecessary and wasteful to call out the query planner, so Doris executes a short circuit plan for them to reduce overhead. Another big source of overheads in high-concurrency point queries is SQL parsing. For that, Doris has prepared statements. The idea is to pre-compute the SQL statements and cache them so that they can be reused for similar queries. Data Ingestion Apache Doris provides a range of methods for data ingestion. : Real-Time stream writing : You can apply this method to write local files or via HTTP. It is linearly scalable and can reach a throughput of 10 million records per second in some use cases. Stream Load data streams : With built-in Flink CDC, this Connector ingests data from OLTP databases to Doris. So far, we have realized auto-synchronization of data from MySQL and Oracle to Doris。 Flink-Doris-Connector : This is to subscribe data from Kafka message queues. Routine Load : This is especially useful when you try to do ETL in Doris internally, like writing data from one Doris table to another. Insert Into : Batch writing : With this method, you can leverage Spark resources to pre-process data from HDFS and object storage before writing to Doris. Spark Load : This supports HDFS and S3 protocol. Broker Load : This simple statement allows you to connect Doris to various storage systems, data lakes, and databases. insert into <internal table> select from <external table> Data Update For data updates, what Apache Doris has to offer is that it supports both Merge on Read and Merge on Write, the former for low-frequency batch updates and the latter for real-time writing. With Merge on Write, the latest data will be ready by the time you execute queries, and that's why it can improve your query speed by 5 to 10 times compared to Merge on Read. From an implementation perspective, these are a few common data update operations, and Doris supports them all: : to replace or update a whole row Upsert : to update just a few columns in a row Partial column update : to filter out some data by combining a few conditions in order to replace or delete it Conditional updating : to rewrite a table or partition. Insert Overwrite In some cases, data updates happen concurrently, which means there are numerous new data coming in and trying to modify the existing data record, so the updating order matters a lot. That's why Doris allows you to decide the order, either by the order of transaction commit or that of the sequence column (something that you specify in the table in advance). Doris also supports data deletion based on the specified predicate, and that's how conditional updating is done. Service Availability & Data Reliability Apart from fast performance in queries and data ingestion, Apache Doris also provides a service availability guarantee, and this is how: Architecturally, Doris has two processes: frontend and backend. Both of them are easily scalable. The frontend nodes manage the cluster’s metadata and handle user requests; the backend nodes execute the queries and are capable of auto data balancing and auto-restoration. It supports cluster upgrading and scaling to avoid interruption to services. Cross Cluster Replication Enterprise users, especially those in finance or e-commerce, will need to backup their clusters or their entire , just in case of force majeure. So Doris 2.0 provides Cross Cluster Replication (CCR). With CCR, users can do a lot: data center : for quick restoration of data services Disaster recovery : master cluster + slave cluster; one for reading, one for writing Read-write separation : For cluster scaling, CCR allows users to pre-create a backup cluster for a trial run so they can clear out the possible incompatibility issues and bugs. Isolated upgrade of clusters Tests show that Doris CCR can reach a data latency of minutes. In the best case, it can reach the upper-speed limit of the hardware environment. Multi-Tenant Management Apache Doris has sophisticated Role-Based Access Control, and it allows fine-grained privilege control on the level of , tables, rows, and columns. databases For resource isolation, Doris used to implement a hard isolation plan, which is to divide the backend nodes into resource groups and assign the Resource Groups to different workloads. This is a hard isolation plan. It was simple and neat. But sometimes, users can make the most out of their computing resources because some Resource Groups are idle. Thus, instead of Resource Groups, Doris 2.0 introduces Workload Group. A soft limit is set for a Workload Group about how many resources it can use. When that soft limit is hit, and meanwhile, there are some idle resources available. The idle resources will be shared across the workload groups. Users can also prioritize the workload groups in terms of their access to idle resources. Easy to Use As many capabilities as Apache Doris provides, it is also easy to use. It supports standard SQL and is compatible with MySQL protocol and most BI tools on the market. Another effort that we've made to improve usability is a feature called Light Schema Change. This means if users need to add or delete some columns in a table, they just need to update the metadata in the front end but don't have to modify all the data files. Light Schema Change can be done within milliseconds. It also allows changes to indexes and data types of columns. The combination of Light Schema Change and Flink-Doris-Connector means synchronization of upstream tables within milliseconds. Semi-Structured Data Analysis Common examples of semi-structured data include logs, observability data, and time series data. These cases require schema-free support, lower cost, and capabilities in multi-dimensional analysis and full-text search. In text analysis, people mostly use the LIKE operator, so we put a lot of effort into improving its performance, including pushing the LIKE operator down to the storage layer (to reduce data scanning) and introducing the NGram Bloomfilter, the Hyperscan regex matching library, and the Volnitsky algorithm (for sub-string matching). We have also introduced an inverted index for text tokenization. It is a powerful tool for fuzzy keyword search, full-text search, equivalence queries, and range queries. Data Lakehouse For users to build a and a unified query gateway, Doris can map, cache, and auto-refresh the metadata from external sources. It supports Hive Metastore and almost all open data lakehouse formats. You can connect it to relational databases, Elasticsearch, and many other sources. And it allows you to reuse your own authentication systems, like Kerberos and Apache Ranger, on the external tables. high-performing data lakehouse Benchmark results show that Apache Doris is 3~5 times faster than Trino in queries on Hive tables. It is the joint result of a few features: Efficient query engine Hot data caching mechanism Compute nodes Views in Doris The Compute Nodes is a newly introduced solution in version 2.0 for data lakehousing. Unlike normal backend nodes, Compute Nodes are stateless and do not store any data. Neither are they involved in data balancing during cluster scaling. Thus, they can join the cluster flexibly and easily during computation peak times. Also, Doris allows you to write the computation results of external tables into Doris to form a view. This is a similar thinking to Materialized Views: to trade space for speed. After a query on external tables is executed, the results can be put in Doris internally. When there are similar queries following up, the system can directly read the results of previous queries from Doris, and that speeds things up. Tiered Storage The main purpose of tiered storage is to save money. Tiered storage means to separate hot data and cold data into different storage, with hot data being the data that is frequently accessed and cold data that isn't. It allows users to put hot data in the quick but expensive disks (such as SSD and HDD), and cold data in object storage. Roughly speaking, for a data asset consisting of 80% cold data, tiered storage will reduce your storage cost by 70%. The Apache Doris Community This is an overview of Apache Doris, an open-source real-time data warehouse. It is actively evolving with an agile release schedule, and the embraces any questions, ideas, and feedback. community

Introduction to Apache Doris: A Next-Generation Real-Time Data Warehouse

Extracting data from an existing database is one of the most challenging and tedious jobs that typically fall on the plate of the Data Engineering team. And when it comes to dealing with true legacy systems, well, let’s just say it adds an extra layer of intrigue to the mix. Unraveling the secrets left behind by generations of engineers is like solving a complex puzzle. Inconsistent data modeling and potential challenges like insufficient documentation, limited knowledge of the existing team, and unclear semantics are just a glimpse of the challenges ahead. Yet, amidst these daunting obstacles, the team is expected to deliver results with lightning speed. In this article, I aim to impart insights gained through hard-earned experience, offering a lifeline that may save you precious time. So, let us explore these pivotal non-technical starting points: Even if a system/database is very old, there is usually at least one person who has some understanding of how it works and where data is coming from. This individual can be an invaluable resource to you, but keep in mind that they may be busy, have knowledge gaps, or may not like your project at all. Nevertheless, building a positive relationship with this person can be critical to your success. They can help you save significant time and effort. Do not criticize the current state of things, even if there are obvious flaws and improvement opportunities. Focus on extracting the data as efficiently as possible with minimum dependencies on the source system. Identify a knowledge holder: It’s important to educate stakeholders about the nature of data extraction projects. Many people mistakenly assume it’s a one-time task, often due to a lack of experience in this area. However, unless you’re working with a very small database, it’s likely to require multiple iterations to ensure that the data is complete and accurate. Therefore, it’s essential to communicate realistic timeframes and the need for iterative processes to stakeholders. It’s a very challenging task to estimate that effort. So try to find the most critical piece of data you can focus on initially. Manage project stakeholders’ expectations: In any domain, people tend to use different terms for the same things. Therefore, involving those who use the source system can help you explain the data behavior and identify any data anomalies that may arise. For instance, when we had to extract data from a widely-used EHR system, it was a surprise that patient information was stored in a table named ‘users.’ By collaborating with the end-users and the data source users, we were able to address such semantic inconsistencies and prevented any confusion down the line. Establish a shared understanding of semantics with both the users of the source system and the end users of the data: There are always some standardized reports generated from the legacy system. Typically you can use those reports as a set of reconciliation metrics. Don’t be surprised that you discover some inconsistencies and/or errors during your reconciliation process. But it’s always helpful to establish some baseline to evaluate the fullness and quality of the data. Establish early success criteria: There are many approaches and tools you can use to do the job. From plain all the way to tools. It’s up to you and your arsenal of skills, but here are some high-level suggestions that typically help: Ok, and what about the technical side of the project? SQL no-code It’s essential to understand how the data changes over time, including retrospective changes, in order to establish reliable Change Data Capture for the tables you’re interested in. Many legacy databases don’t have an OOTB CDC feature, so you may have to deal with a full data refresh. An alternative would be to use a data replication tool with CDC capabilities integrated. Tools like Airbyte and Fivetran may help to save time there. Change Data Capture (CDC): Table names alone can be misleading, so analyzing the database workload can help you better understand which tables are critical for your extraction process. It can help you to identify critical objects in the data model. Analyze the workload to identify the most and least used objects/tables: It’s always a good practice to extract the raw data and store it as is. Don’t perform any transformations in that step. Use your Data Lake or Staging area to store it until the iteration/project is complete. This minimizes the performance impact on the source database. This also enables you to compare the extracted data from the previous run to the current version, helping to identify changes you may have missed during implementation. Another advantage is the ability to rerun transformations without impacting the source database. Extract and store data in its raw form: When it comes to transforming the extracted data, it’s best to break it down into multiple steps rather than cramming everything into a single, complex set of instructions. This helps with the maintainability of the data transformation code and makes it easier for you and your team to understand what’s going on later on. Overcomplicated code can be a hindrance, especially when revisiting it after several iterations. Good documentation and structured code can save you and your team time. Structure transformations in multiple stages: Maintaining both the code and the documentation can be a lot of work. Having a no-code tool that can handle data lineage, transformations, and documentation can be helpful. When selecting a tool, always consider who will maintain the pipeline and how to make that process as efficient as possible. How technical that person will be. Some tools require more technical skills than others. Consider using a no-code tool: A semantic model can be extremely useful when integrating data from multiple sources into a single data model. Normalizing data and using transformations that can be demonstrated are highly beneficial features. This approach ensures that your data is well-structured and easy to understand, making it more accessible and simpler for data users to access and analyze. Create a Semantic Model: Data extraction can be challenging, but following these key takeaways can help you avoid issues, especially if you’re doing it for the first time. We intentionally skipped all parts related to security and compliance and assumed you have access to the database. It’s essential to build strong relationships with knowledge holders, manage project stakeholders’ expectations, establish a common definition of semantics with data users, set clear success criteria, and structure transformations in multiple stages. No-code automation tools like Datuum and data replication tools like Airbyte or Fivetran can simplify the process. Documenting transformations, maintaining data lineage, and creating a semantic model is crucial to establishing a sustainable and maintainable data pipeline. These steps will benefit the entire organization in the long run. Also published here.

Stories, News & Mentions about FIVETRAN

Fivetran’s Stories on HackerNoon

Fivetran's News & Mentions